Particularly interested in the performance stats, how well the daemon keeps time in the face of various network problems. Chrony is very good at this. Some of the other NTP implementations (not on that chart) are so bad they shouldn't be used in production.

Given the amount of testing we (and other parties) have done, and given the strong theoretical foundation of our algorithm I’m pretty confident we’d do well in many production environments. If you do find any performance issues though, we’d love to hear about them!

For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for consideration.

(And I say this as a massive rust aficionado using it for both work and pleasure.)

Hosts should then be synced using PTP or a proper NTP local stratum (just get a proper GNSS source for each DC if you have then funds).

https://tsn.readthedocs.io/timesync.html

Deploy chrony to bare metal servers wherever possible.

But last time I said that with some pretense of authority, someone shoved me a whitepaper from VMware that said the opposite. Best practice was stated be to sync each guest individually with a completely virtual clock.

I'm not sure I agree, but at least I try to be open to be possibility that there are situations I had not considered. If anyone else knows more about this, please share.

(a) the inherent risks in implementing a protocol based on trivially spoofable UDP that can be used to do amplification and reflection

and

(b) emergent resonant behavior from your implementation that will inadvertently DDOS critical infrastructure when all 100m installed copies of your daemon decide to send a packet to NIST in the same microsecond.

I'm happy to see more ntpd implementations but always a little worried.

One thing to note about amplification: amplification has always been something that NTP developers have been especially sensitive to. I would say though that protocols like QUIC and DNS have far greater amplification risks. Meanwhile, our server implementation forces that responses can never be bigger than the requests that initiated them, meaning that no amplification is possible at all. Even if we would have allowed bigger responses, I cannot imagine NTP responses being much bigger than two or three times their related request. Meanwhile I've seen numbers for DNS all the way up to 180 times the request payload.

As for your worries: I think being a little cautious keeps you alert and can prevent mistakes, but I also feel that we've gone out of our way to not do anything crazy and hopefully we will be a net positive in the end. I hope you do give us a try and let us know if you find anything suspicious. If you have any feedback we'd love to hear it!

Only a short mention in the main article about NTP itself:

> Network Time Security (NTS) is a secure version of NTPv4 with TLS and AEAD. The main improvement over previous attempts is that a separate "key establishment" server handles the heavy asymmetric cryptography, which needs to be done only once. If the server goes down, previous users would still be able to fetch time without fear of MITM. NTS is currently supported by several time servers, including Cloudflare. It is supported by NTPSec and chrony.

As far as I know, the Debian “default” is systemd-timesyncd. That is what you get out of the box. (Though, honestly, I automate most of my Linux installs, so I don’t interact with a stock install very often.) AFAIK, systemd-timesyncd does not support NTS at all.

Doing NTS on a pool would be quite complicated. The easy way is to share the same key across the pool. That is obviously not workable when pool servers are run by different people. The other way would be to have an another out-of-band protocol where the pool NTP servers share their key with the centralized pool NTS-KE servers. Nobody has built that, and it’s non-trivial.

Each such machine mints (as often as it wants, but at least once) a document called a Certificate Signing Request. This is a signed (thus cannot be forged) document but it's public (so it needn't be confidential) and it basically says "Here's my public key, I claim I am some-shared-name.example, and I've signed this document with my private key so you can tell it was me who made it".

The centralized service collects these public documents for legitimate members of the pool and it asks a CA to issue certificates for them. The CA wants a CSR, that's literally what it asks for -- Let's Encrypt clients actually just make one for you automatically, they still need one. Then the certificates are likewise public documents and can be just provided to anybody who wants them (including the NTP pool servers they're actually for which can collect a current certificate periodically).

So you're only moving two public, signed, documents, which isn't hard to get right, you should indeed probably do this out-of-band but you aren't sharing the valuable private key anywhere, that's a terrible idea as well as being hard to do correctly it's just unnecessary.

On my initial encounter with ntpsec, I found ntpsec running, but ntp was also installed. That's an interesting construction of "replace". This would be hard to replicate, because I don't know when ntpsec turned up; otherwise I'd try to make a bug report. If ntpsec was replacing ntp, I'd expect to find no ntp after the update.

It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now.

I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start.

NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary. A quick Google for NTP vulnerabilities will show you that there are plenty of memory safety vulnerabilities lurking in C NTP implementations:

https://www.cvedetails.com/vulnerability-list/vendor_id-2153...

Some of these are severe, some aren't. It's only a matter of time though until another severe one pops up.

I don't think any critical service on a network boundary should be written in C/C++, we know too much at this point to think that's a good idea. It will take a while to change that across the board though.

If I had to pick the most important thing in the context of Let's Encrypt to move to a memory safe language it would be DNS. We have been investing heavily in Hickory DNS but it's not ready for production at Let's Encrypt yet (our usage of DNS is a bit more complex than the average use case).

https://github.com/hickory-dns/hickory-dns

Work is proceeding at a rapid pace and I expect Hickory DNS to be deployed at Let's Encrypt in 2025.

Seems no digital system is truly secure. Moving foundational code to memory safe seems like a good first step.

Translate the internet to the real world, and basically every building (IP address) is getting shot at, hit by planes, nuked, bioweapons are stuffed into the mail slot, and lock-picked all day, every day.

The reason people know this about postfix and keep bringing this one specific example up is because it's so unusual! It's an example that almost stands alone, it's extraordinary.

I don't think this is making the point about the safety of C that you think it is.

There is a mountain of evidence suggesting that C is dangerous, particularly for network services, and one possible example to the contrary doesn't change that.

Don't learn Rust if you aren't interested, it's not a one true language, but don't cheat yourself on engaging with an interest because HN struggles to discuss it amicably.

GP is obstinately failing to grasp the difference between “can” and “should”. Networked services can be securely written in C. Networked services should not be written in C.

There are people who can operate motor vehicles safely at speeds regularly in excess of 100mph. People should not do so on public roads.

“What about this one C project that hasn’t been a mess of exploitable vulnerabilities” is thoroughly unconvincing in a world where networked C programs are an unending source of severe vulnerabilities.

Censorship doesn't require content to be completely hidden or blocked; even just partially obscuring the content in some way is still censorship.

Plenty of reasonable disagreement happens here without getting downvoted into obscurity. Substantive points are generally upvoted even when they’re controversial opinions. So I don’t feel too upset when borderline-trolling or bad-faith arguments get hidden by consensus.

Ironic counterexample: It looks like you may have gotten downvoted despite having a fundamentally reasonable perspective.

There's a reason why "this is the exception that proves the rule" is a common saying. Picking out one exceptional example of something, and using that to argue against a general point, is at best lazy, and at worst actively dishonest. I'm fine with those kinds of comments being "censored". They -- and their replies that call people out for doing this -- are boring and don't further discussion.

this thread isn't about Postfix, you brought up Postfix as a counterpoint to a wider topic where such a anecdote doesn't hold up nearly as well in a broader scope. Don't purposefully narrow the topic and move goalposts just to win internet fights.

The getting traction part is the relevance part.

1. They didn't have a big, well-known, company name with good enough reputation to attract contributors.

2. They didn't have brackets.

Success was because traction, traction was because appeal, and appeal was mostly because those two things. Nothing else was new AFAIK.

A lot of them simply assumed that some amount of "overhead," vaguely described, was acceptable to the target audience. Either tracing GC or reference counting. "but we're kinda close to C performance" was a thing that was said, but wasn't really actually true. Or rather, it was true in a different sense: as computers got faster, more application domains could deal with some overhead, and so the true domain of "low level programming languages" shrunk. And so finally, Rust came along, being able to tackle the true "last mile" for memory unsafety.

Rust even almost missed the boat on this up until as late as one year before Rust 1.0! We would have gotten closer than most if RFC 230 hadn't landed, but in retrospect this decision was absolutely pivotal for Rust to rise to the degree that it has.

Their problems was a mix of not being hieroglyph languages (too much text to type!), being much strongly typed (straitjacket programming with a famous rant post, wrongly), commercial offerings being expensive (more so against free in the box alternative), and not comming with an OS to make their use unavoidable.

Note that for all the hype, Zig is basically Modula-2 features with C syntax, to certain extent.

Google it, C memory boundary issues have been a problem for security forever.

To be honest, that's a surprisingly deep question, and the answer is something I'm yet to see any developer I worked with understand. For example, did you know that SQL injection and XSS are really the same problem? And so is using template systems[0] like Mustache?

In my experience, very few people appreciate that the issue with "SQL query forming via string concatenation" isn't in the "SQL" part, but in the "string concatenation" part.

--

[0] - https://en.wikipedia.org/wiki/Template_processor

Really? To me it's pretty obvious that not escaping properly is the issue, and therefore the same issue applies wherever you need escaping. I don't think I've ever heard anyone say that SQL itself was the problem with SQL injection. (Although you certainly could argue that - SQL could be designed in such a way that prepared statements are mandatory and there's simply no syntax for inline values.)

Memory safety means that your program is free of memory corruption bugs (such as buff overflow or under flow bugs) that could be used to retrieve data the user isn't supposed to have, or can be used to inject code/commands that then get run in the programs process. These don't get shut down by the OS because the program is interacting with its own memory.

However, calloc everywhere won't save you from smashing the stack or pointer type confusion (a common source of JavaScript exploits). Very rarely is leftover data from freed memory the source of an exploit.

Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that:

- runs as root

- talks to the network

- doesn't usually authenticate its traffic

- uses a bespoke binary packet format

- almost all network security depends on (for checking cert expiration)

That looks to me like an excellent candidate for a memory-safe reimplementation.

ntpd can (and should) run as a user

> talks to the network

Makes outbound requests to the network. For it to be compromised, the network itself or a downstream server needs to be compromised. That's very different from something like hosting an http server.

> doesn't usually authenticate its traffic

Yes it does. ntp uses TLS to communicate with it's well known locations.

> uses a bespoke binary packet format

Not great but also see above where it's talking to well known locations authenticated and running as a user.

It's a service that to be compromised requires state level interference.

(But regardless: “the transport layer is secure” is not a good reason to leave memory unsafe code on the network boundary. Conventional wisdom in these settings is to do the “defense in depth” thing and assume that transport security can be circumvented.)

For servers it's definitely harder, although definitely not only state-level, but another big issue could be client-side. NTP servers can be set by DHCP, so the admin of any network you connect to could exploit such a bug against you. And once you have code execution on a desktop OS, all bets are off, even if you're not under the primary UID.

It's not the most important threat vector, but it also doesn't seem as difficult as some of the other system services to rewrite, so I'd say it was a good first step for the memory-safe-everything project.

> Yes it does. ntp uses TLS to communicate with it's well known locations.

My knee-jerk reaction is that TLS is not authentication. After skimming through the relevant spec [0], it is interesting how NTP uses TLS.

[NTS-KE] uses TLS to establish keys, to provide the client with an initial supply of cookies, and to negotiate some additional protocol options. After this, the TLS channel is closed with no per-client state remaining on the server side. [0]

0. https://datatracker.ietf.org/doc/html/rfc8915

I'll flip the question around, why not start at ntpd?

Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous.

For example, JEMalloc is used by nearly every app compiled in *nix.

Perhaps systemd which is just about everywhere running everything.

Maybe sshd, heaven knows it's been the root of many attacks.

JEmalloc is used by very very very few apps compiled for *nix. That's a conscious decision that an app developer needs to make (and few would bother without specialized needs) or a distro packager needs to shoehorn into their package builds (which most/all would not do).

Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start.

Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.

The answer is that there is observability functionality which provides its results as JSON output via a UNIX socket[0]. As far as I can see, there's no other JSON functionality anywhere else in the code, so this is just to allow for easily querying (and parsing) the daemon's internal state.

(I'm not convinced that JSON is the way to go here, but that's the answer to the question)

[0] https://docs.ntpd-rs.pendulum-project.org/development/code-s...

  print "{"
  print "\"some_state\": \"";
  print GlobalState.Something.to_text();
  print "\", ";
  print "\"count_of_frobs\": ";
  print GlobalState.FrobsCounter;
  print "}";

You're gonna have a hard time exploiting a text file output that happens to be JSON.

If you’re not escaping double quotes in strings in your hand-rolled JSON output, and some string you’re outputting happens to be something an attacker can control, then the attacker can inject arbitrary JSON. Which probably won’t compromise the program doing the outputting, but it could cause whatever reads the JSON to do something unexpected, which might be a vulnerability, depending on the design of the system.

If you are escaping double quotes, then you avoid most problems, but you also need to escape control characters to ensure the JSON isn’t invalid. And also check for invalid UTF-8, if you’re using a language where strings aren’t guaranteed to be valid UTF-8. If an attacker can make the output invalid JSON, then they can cause a denial of service, which is typically not considered a severe vulnerability but is still a problem. Realistically, this is more likely to happen by accident than because of an attacker, but then it’s still an annoying bug.

Oh, and if you happen to be using C and writing the JSON to a fixed-size buffer with snprintf (I’ve seen this specific pattern more than once), then the output can be silently truncated, which could also potentially allow JSON injection.

Handling all that correctly doesn’t require that much code, but it’s not completely trivial either. In practice, when I see code hand-roll JSON output, it usually doesn’t even bother escaping anything. Which is usually fine, because the data being written is usually not attacker-controlled at all. For now. But code has a tendency to get adapted and reused in unexpected ways.

Using C0 codes is likely safer at least in the sense that you will probably think to check for those and there is no reason whatsoever for them to be found in user data.

.. than use library, because you should not rely on the assumption that next developer adding one more piece to this code will magically remember to validate it with json spec.

OpenBSD style ntpd.conf:

    servers 0.gentoo.pool.ntp.org
    servers 1.gentoo.pool.ntp.org
    servers 2.gentoo.pool.ntp.org
    servers 3.gentoo.pool.ntp.org

    constraints from "https://www.google.com"

    listen on *

The configuration object will always and by definition be proprietary to ntp. Expressing it as plain text allows for a trivial parser, without any of the security implications of wrapping it in a general language language ("should this string be escaped?", "what should we do with invalid utf8?").

The more simple format has survived over thirty years, is trivial to parse by anyone, and does not bring any dependencies that needs maintaining. That should count for something.

The fact that it has survived tells you nothing other than it's not so completely awful that someone went through the pain of fixing it. That doesn't mean it is good. There are plenty of awful things that survive because replacing them is painful due to network effects. Bash for example.

1. Code doesn't change at the whims of others.

2. The entire parser for an INI-style config can be in about 20 lines of C

3. Attacker doesn't also get to exploit code you've never read in the third party dependency (and its dependencies! The JSON dependency now wants to pull in the ICU library... I guess you're linking to that, too)

4. Complexity of config file formats are usually format-independent, the feature-set of the format itself only adds complexity, rather than takes it away. To put it another way, is this any saner...

    {"user":"ams","host":"ALL","runas":["/bin/ls","/bin/df -h /","/bin/date \"\"","/usr/bin/","sudoedit /etc/hosts","OTHER_COMMANDS"}

    # I may be crazy mad but at least I can have comments!
    ams ALL=/bin/ls, /bin/df -h /, /bin/date "", /usr/bin/, sudoedit /etc/hosts, OTHER_COMMANDS

An example of an NTP server's config:

    # I can have comments too
    [Time]
    NTP=ntp.ubuntu.com
    RootDistanceMaxSec=5
    PollIntervalMinSec=32
    PollIntervalMaxSec=2048

2. And how many times have we seen 20 lines of C backfire with some sort of memory safety issue.

3. First off, i'd go out on a limb and say the number of attacks from a well-established (or even a naive one) rust json parsing library is dwarfed by the number of attacks from adhoc config parsers written in C with some overlooked memory safety issue.

4. Usually being the key word, tons of adhoc config formats have weird shit in them. With json (or yaml/toml) you know what you're getting into and you immediately know what you're able and unable to do.

(I do agree that JSON is a terrible configuration file format, though.)

My argument was data interchange format standards are orthogonal to config files. They don't have the same goals.

A programmer who thinks "I'll use JSON|YAML|TOML for my config file" - well, you didn't solve your first problem (what should the config look like, in a way that makes sense and is easily readable, understandable, updateable by the user) and you added a second problem before you even started solving the first - whatever your config looks like, it now also has to be 100% compliant with some data interchange format and support all its features, and that's going to require a lot of code - and then we get into whether you write the compliant parser/generator, or if someone else does and you do/don't audit every line of it. And then on top of that you add an additional pile of code to parse/generate whatever your actual config format is.

    [Alarm]
    Name=Nginx Errors
    Pattern="[error] #: "

    Pattern="[error] 

Hand-rolling config formats is hard.

In 2003 those programs would have used XML, in 1993 probably .INI files. Are you sure that despite all its shortcomings JSON is the end of history? I don't believe you.

If you want "at this point" you can, as software does today, just use a crate. Unlike the stdlib, if next week Fonzie files are huge and by 2026 "nobody" is using JSON because Fonzie is cool, the JSON config crate merely becomes less popular.

And going full-assed with one or more actually-outside antennas can also be problematic, since it's a Bad Day when (eg) grounding is done improperly and lightning comes by to fuck their shit up.

And ISPs can (and certainly should!) provide decent NTP servers. That's the logical place for shared servers, network-wise. But the one's choice of ISP is often limited by geography, and the most-viable ISP here stopped publishing what their NTP servers are -- if they still exist in a form that customers can use, they don't publish this information. (It wasn't always this way and I remember using them years ago when they were more forthcoming. They were shitty NTP servers with high jitter. Good enough for some things, I suppose, but not as good as the members of the public NTP pool tend to be.)

I mean: Many ISPs can't even manage to handle DNS queries quickly. When public servers like 8.8.8.8 and 1.1.1.1 (or the semi-public ones like 4.2.2.1) are faster than the ISP's own servers, then that's a problem. And it's a stupid problem, and it should not happen. But it does happen.

So thus, public NTP servers are useful for many -- including those who have a tinkered-together NTP server with a GPS antenna in a window somewhere, where public servers can be used as backup.

It's good to have options, and it's nice that some organizations provide some options for the greater network.

https://www.vox.com/2015/5/23/8647095/kool-aid-jonestown-fla...

They really do appear to be all in on avoiding memory leaks from C/CPP:

> Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

It seems like a really challenging endeavor, but I appreciate their desire to maintain uptime and a public service like they do.

Rust is really good for correctness. Take "Hello, World", the obvious toy program. Someone tried giving it various error states instead of (as would be usual) a normal happy terminal environment. In C or C++ the canonical "Hello, World" program terminates successfully despite any amount of errors, it just doesn't care about correctness.

The default Rust Hello World, the one you get out of the box when you make a new project, or you'd show people on a "My First Rust Program" course, will complain about the errors when they happen. Because doing so is correct.

It's the New Jersey style. The priority for these languages was simplicity of implementation. It's more important that you can cobble together a C compiler easily than that the results are useful or worthwhile. This contributed to C's survival, but we pay the price until we give it up.

You will not see the message if you get a panic either ...

Similarly in Rust, serious programs will use writeln rather than println, and will receive the standard compiler warning if the Result produced by writeln is ignored.

TFA's second sentence explains the facts of the matter along with the flavor of Kool-Aid they stock:

> The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Because it is a fact. An obvious, obvious fact to anyone who has been working in this ecosystem for any amount of time.

> Drinking the Rust kool-aid by the sound of it.

Would you say the same thing if they'd instead decided to use golang, zig, nim, ocaml, etc.? If not, maybe consider that your emotional stance around Rust is coloring your judgment.

Our modern systems are built on a house of cards where security is concerned. I agree that the "RiiR" meme is tiresome and dumb, but I'm tired of seeing report after report of new vulnerabilities found in foundational libraries and programs. The majority of those vulnerabilities are of the type that Rust won't even let you compile. Languages like C and C++ have their place, but for most applications, there are safer alternatives that don't require harsh compromises or significant trade offs.

True, there are plenty of alternatives for many of those networking services, not necessarily Rust.

Rust chose a good in-between (the "?" unwrapping syntax). In mid or low level code, anything can still fail, but I suspect the performance impact of supporting exceptions or similar for basic operations (integer overflow, div by 0, etc) wouldn't be worthwhile vs just crashing the program or doing something else. Interestingly, Rust doesn't crash in this case: https://doc.rust-lang.org/book/ch03-02-data-types.html

However then I realise, why bother, and go back into using C#, Java, D instead.

However I don't think the pace of updates really changes anything in terms of tool chain security. If Rust decided to go to a 36 week release cycle, each release would just have 6x as much stuff in it. If you can't keep up reviewing N changes in a 6 week release cycle, moving to a 6*X release cycle will not help you review N*X changes.

In addition Rust was born from Mozilla and imitates Firefox's rapid release schedule of one release per six weeks. This does not mean that Rust releases are substantial or that they are traumatizing, only that they are frequent. The contract with users is that Rust releases must be painless so as to not fatigue users and discourage them from upgrading. The success of this painless upgrade strategy is proved by the fact that library authors are so quick to upgrade, as mentioned.

This is in contrast to other languages where historically a new version of their compiler might only be released as infrequently as once per three years. It seems that these languages have begun taking queues from Rust as even Java now releases once per six months.

I suspect if you were to look at other reasonably popular languages that are of the same era, you'll see a similar level of changes.