NTP's REFID

The Network Time Protocol specifies a data packet that contains information needed to communicate the time. One of the key portions of this packet is NTP’s REFID – a reference identifier. The refid is a 32-bit field that identifies the source of time for the data packet. The NTP data packet also includes some timestamps, various other pieces of information, and the stratum of the time source, or the number of layers between the server and its ultimate source of time.

Most people who know that the refid exists have a pretty good idea of what it’s for, but they also tend to have some misconceptions about it, too. Most people think that the refid is either some informational data or the IPv4 address of that association’s upstream connection. This used to be a reasonable simplification back when the only IP protocol was IPv4, but with IPv6’s growing role and certain operators disseminating non-UTC time with NTP, it’s becoming increasingly important that operators properly and fully understand what the refid is and is not designed and able to do.

The interpretation of the refid field depends on the provider’s stratum.

At Stratum 0, or S0, the source is considered unspecified or invalid, and the refid in an S0 packet contains 1-4 ASCII bytes of what is called a kiss-of-death code, or KoD code. You can learn more about KoD codes in the documentation since they’re not particularly interesting for the purpose of this whitepaper.

At Stratum 1, or S1, the source is considered a reference clock, and the refid in an S1 packet contains 1-4 ASCII bytes that identify the type of the reference clock. An example of how this would be displayed for a GPS receiver is .GPS., while the value would be .WWVB. if the source was the WWVB radio signal.

At Stratum 2-15, the source is considered a secondary source of time, from an NTP server. For this whitepaper, this is the interesting case and I’ll just use S2+ to identify it.

At Stratum 16 and above, the source is considered unsynchronized.

Just to make things interesting, it’s possible to fudge the stratum of a reference clock to something other than 1. In this case the refid will still be 1-4 ASCII bytes, but nobody else will know about this and it generally won’t decode sensibly.

When ntpd selects the source of time it will believe, that source is called its system peer.

The refid for an S2+ instance is based on its system peer, in particular the IP address of the system peer. The primary reason for this is because the refid is used to detect a timing loop, which means that if machines A and B are exchanging time and B decides that it will follow A, then A can use the refid to make sure it won’t decide to follow B. Think of it as a self-deceit filter.

Years ago, NTP used to detect and prevent another case, one where if A provided time to both B and C, then if B was synced to A it (B) would not decide to follow C if C was also following A. Dave Mills says that this degree of protection turned out to actually cause problems so this test was removed.

This was all pretty clean and easy back when all we had was IPv4, and when computers had very few IP addresses assigned to them. But two of the things that have changed since then are the use of IPv6, in which the IP address is 128 bits long (remember, the refid field is only 32 bits long), and it is common now for a computer to have many IPs assigned to it. Therefore, when A gets an S2+ NTP packet from B, it might have to check against all of its possible refids to make sure that B is not using A as its system peer. ntpd today does not check every possible interface address when looking to see if B’s refid came from any one of A’s network addresses — the assumption is that if A sends its request to B, B will send its response back to the same address that A used when sending its request to B. This is a pretty safe assumption, since that’s how ntpd is written to behave.

Also note that A does not and can not tell B what B should use as its refid. Again, this could mean that A needs to be able to generate all of the possible refids that its clients may generate to identify that A is their system peer. But we’ve never had a report of this being a problem over NTP’s more than 20 year history, and to-date that likely means something like 100,000,000,000,000 operational hours of operation. Yes, that’s 100 trillion hours of operation.

Here’s another wrinkle. Since NTP was designed when IPv4 was the only game in town, using the IP address to detect time loops was a really great idea. For S2+ packets, we displayed the refid as an IPv4 address, which meant it was obvious where each machine was getting its time from. But then came IPv6, and a way needed to be found to translate the 128 bits that make up an IPv6 address into a 32-bit field. The solution that was decided on was to use the first 4 bytes (32 bits) of the MD5 hash of the IPv6 address. This was considered a good idea because it was felt that the risk of collision was low. More directly, what are the odds that a 4-byte effectively random number will turn out to match something we’d actually see in real-use? Originally, the answer to this question was “there’s little chance”. Time moves on, and more and more addresses are now in use. The chance of a collision is much higher now. We even have a reported case where the first 4 bytes of the MD5 hash of the IPv6 address of a system peer decodes to the IPv4 address in that same provider’s network space.

Additionally, for a very long time the most common way to check on the status of ntpd was to run ntpq -p, which displayed the associations that the target ntpd had. The first two columns of this output were the remote machine, and the refid. There are a number of other fields displayed as well, and they are all carefully designed to fit in an 80 column display. Displaying the refid in its full IPv4 glory takes up a significant amount of that space. But back in the day it was worth it. After all, for S2+ machines that only spoke IPv4 it was a very efficient way to know where time was coming from. But there was a double problem when IPv6 came on the scene. One is that a refid that was based on an IPv6 address still looked like it was an IPv4 address, and the other problem was there was no easy way to look up any given association to see what its full name or its IPv6 address really was. There are now two ways to get the full source name/address of the peer. 4.2.7p461 added the -w option to ntpq, which will display the full IP address or DNS name of the source address, and if that information is too wide it will show it all anyway, putting the remainder of the data on the next line. This is fine for human readers, but it adds a complication when trying to get the information into a script. 4.2.8p3 adds the apeers command, which displays the refid in hex and also displays the association ID. A script can easily parse the association ID from this output and then look up the peer’s address using the association ID.

Another problem is that systems are not all running the latest software. Indeed, there are still a good number of NTPv3 servers out there, and NTPv3 has been obsolete for nearly 20 years’ time now. There are vendors who are still shipping 4.2.4, which was EOL’d in 2009. There are also a huge number of installed instances of 4.2.6, which was EOL’d in December of 2014, when 4.2.8 was released. If we make any changes we need to be comfortable that we’re not creating more problems than we’re solving.

Changing the refid

So what sort of changes to the refid might be useful?

There are three primary changes worth considering. One has to do with the IPv6 representation, the second has to do with the leap smear, and the third has to do with protecting NTP associations from abuse. In each of these cases we want to be sure we’re using something that won’t be confused with any usable IPv4 address.

For the IPv6 representation, one idea is to use an initial octet of 255 followed by 3 bytes of the MD5 hash of the IPv6 address. There are two questions about this approach. One has to do with whether or not that first 255 can ever legitimately appear as the first byte of an IPv4 source address from an NTP time source. Another is what is the risk that an older ntpd sending from its IPv6 address will fail to detect that a newer ntpd is synchronized to it, since this second machine will be sending back a refid that the first machine will not know how to generate. While this is a potential problem, there are two ways it doesn’t seem to be that much of a problem. For one thing, a huge number of things have been fixed in ntpd between these old versions of ntpd and the latest versions. For another, if this case happens and the two machines decide to follow each other then their stratum numbers will ratchet up and eventually the servers will either find better sources of time or will become unsynchronized. If they do become unsynchronized, then their configuration was such that they had no better sources of time anyway, so this failure to detect a time loop is no additional loss – the original configuration had no other sources of time, either.

The second case has to do with the experimental change to allow ntpd to smear a leap second correction. The majority-use case here is to have old clients apply the leap second correction in a way that avoids jumping time backwards, and also keeps all of the clients reporting the same time during the leap second correction. While we believe we have a mechanism to do this, since there is no way in NTPv4 to communicate that a server is offering smeared time instead of correct time the proposal is that we use a refid of the form 254.x.y.z to indicate that the NTP packet contains smeared time. The amount of smear correction would be encoded in the x.y.z portion. Since smeared time packets are only sent to clients, not peers, and there are very few (no?) good reasons why, if B is a client of A, A would list B as a machine it would get time from. So the fact that one could not detect a time loop based on these refids during the time a leap second is being applied does not seem to be a compelling reason to avoid it. There is still the question of whether or not 254 can ever appear as the first byte of an IPv4 source address from an NTP time source.

The third case is one in which each time packet would include a field that says “If you use me as your system peer, use this value as your refid”. In NTPv4 we could do this with an extension field, and in NTPv5 we could build this in to the basic packet format. While this feature has some benefits, it also has some new costs. For example, given a set of NTP servers there needs to be a way to make sure each of them has a unique refid. While this is pretty easy to do if the set of NTP servers remains the same, that’s not how it works in reality, especially when an increasing number of clients are using the NTP Pool for their servers. If we find a case where two servers want to use the same refid then we must either live with these collisions or define a process by which any server can be told to re-negotiate its refid. This renegotiation would involve checking with all servers that are at least 1 degree of separation away. The last time I did the arithmetic on this problem it seemed clear to me that we could do this with a refid that contained at least 10 bits of information, so continuing to use 24 or 32 bits is probably enough. But it will still need a renegotiation dance, and that mechanism is another potential abuse vector.

Tricky stuff.