> I like how it's very reliable and integrated: the "kitchen sink mentality" can have positive effects.
Many people don't. When something more complex and integrated works it can seem perfectly fine. When something more complex and integrated fails, it often has more cascading effects that would otherwise be desired.
> In retrospect, given how well it all works, maybe he was right to refuse to compromise.
systemd's timesyncd is less accurate than ntpd and is terrible at dealing with drift; it is mostly only suitable for desktop timesyncing. As soon as you need to deal with synchronous timing with multiple machines, it's usually the number one problem. Even worse is that it doesn't tend to use less memory or cpu than alternatives.
systemd's DHCP client continues to be a nightmare for people using it with many ISPs or corporate networks due to the creators deciding to only handle one format of optional headers (in particular option 43, but also many others) resulting in networking issues in many situations. These issues are long-historical and can even be due to ancient endian issues. Every other DHCP client implementation deals with this - except systemd's. I have to cleanup my ec2 instances because systemd's dhcp client adds 032 instead of a space to resolv.conf's search field because we have more than one domain there.
In some cases these problems are because of long-established practices that technically violate an RFC. In others it's because they refuse to deal with more than one legit way of doing things. Either way, what is a service/process manager doing in this fight?
When you replace something that's been around for decades (init), you're going to ruffle feathers for good and bad reasons. It's had decades of maturity. But now one is going to do it for dozens of services and not appreciate or handle the nuances that come with such a responsibility? It's literally pure arrogance - hence the hate.
You make very good points, but I think they come from a difference in expectations.
For time sync, what you do depends on what you need: there's a long list of option: rdate, sntp, ntp, ptp and I have used all of them.
Even if rdate is technically obsolete, for embedded systems it's sufficient that the time is not grossly wrong (and it's faster to write a rdate client and server)
However, when I'm doing high precision sampling data with multiple computers and I have to reorder the observations, sub microsecond accuracy is not enough: then I use PTP with a GPS and PPS (or chrony with special NICs if I can't get a GPS signal)
Both of these are very far from the normal needs of linux installs: there is no one size fits all, there are even interesting inbetween (chrony doing hardware timestamping with ntp is very clever) but when you design a system, you have to make choices.
> But now one is going to do it for dozens of services and not appreciate or handle the nuances that come with such a responsibility? It's literally pure arrogance - hence the hate.
I'm very happy systemd exists, because I like the well thought out design it has created: I can feel how it's lacking in the BSD.
I understand systemd success was not a given, and choices had to made. I just don't think it was arrogance, but compromises needed to achieve a vision.
> it's because they refuse to deal with more than one legit way of doing things. Either way, what is a service/process manager doing in this fight?
It's making choices.
Maybe systemd could have used ntp or better (maybe it will? or maybe there's a reason it's a bad idea?) but it does most things extremely well: if I have very specific needs (ex: ptp) far outside the norm, I can concentrate on that and let systemd do everything else far better than I would.
> these problems are because of long-established practices that technically violate an RFC
For DHCP, you should suggest an update, and if it gets adopted, a patch to make systemd follow the updated RFC.
That would show some love, and it might be better (and more productive) than the hate
> Both of these are very far from the normal needs of linux installs: there is no one size fits all, there are even interesting inbetween (chrony doing hardware time-stamping with ntp is very clever) but when you design a system, you have to make choices.
There were choices/options, some of which have existed for decades. Instead of replacing them, systemd could have provided optional or mandatory services (eg time, dhcp, etc) that could be passed into systemd. "NTP is enabled and provides time, so it gets started at this point in the dependency tree"...or "NTP with priority 1, chrony with priority 2 provide time, start NTP and if it fails, switch to chrony". This also means that systemd could lag if successor protocols for time syncing, etc take off.
Millisecond or less time syncing is not outside the norm for distributed tracing or other server-side tech stacks, especially if there's lots of traffic. I've been lucky in that AWS's modern underlying hardware seems to keep clock pretty well, meaning I've not had to personally fight with it (that and ubuntu already defaulted to chrony, doing the override work for me.), but I do know people doing on-prem kubernetes loads that have had to...
Swinging back to networking, the tight integration with systemd makes (made? I am only now starting to plan the upgrade of our ubuntu 20.04 fleet to 24.04, so maybe it is better now...) disabling components a huge pain in the ass. My initial fights with systemd's dependency management meant that disabling its internal DHCP had several cascading consequences with other things that expected networking to be "up". This is to say nothing of the tooling that expects a full-slate of systemd services.
> For DHCP, you should suggest an update, and if it gets adopted, a patch to make systemd follow the updated RFC.
It's not mainly an issue with strict adherence to RFCs. There's quite bluntly real-world usage and industry norms.
Firstly RFCs can be vague and unclear. IPSEC is a notorious example in that vendors treat the IP restriction headers differently because the spec is nebulous to how precisely it should work (some call it a proxy-id or proxy-header, others subnet range, etc). Due to a variety of reasons that can be summed up as the consequences of committee driven design, a lot of the spec was defined with great "flexibility" in mind. Some vendors will not bring up the tunnel unless both ends agree on the ranges exactly. Others will happily bring up the tunnel, but only allow traffic that overlaps within the local restrictions. Even more infuriatingly, some vendors (cisco in particular) tie it to the firewall rules, which could include other non-vpn traffic. Some (like checkpoint) dynamically supernet the configured entries. The result is every person who's setup IPSEC VPN tunnels with third parties maintains a "quirks" record to understand how best to bring up tunnels between vendors (eg juniper srx to cisco ASA, juniper srx to palo alto, etc). You just cannot be stubborn here. You need to work with the way the real world works.
Secondly, there can be competing standards or necessary migration paths between them. I'm not sure what the IPv6 status is on systemd so this is only an example, but IPv6's autoconfiguration didn't initially include DNS and other optionally useful fields, meaning DHCPv6 was a thing. This is now mostly deprecated by SLAAC, but DHCPv6 deployments will be needed for awhile as it's all that windows' IPv6 implementation supported until recently. You can't just say "RFC now supports SLAAC...don't whine to me that DHCPv6 doesn't work as it's not part of the RFC".
> That would show some love, and it might be better (and more productive) than the hate
People have tried, but the response can be TL;DR'd to "works on my machine/setup" or to go harass AWS, Microsoft (via AD), etc to make things work the way the systemd mainainers think is right. There's a reason Lennart Poettering has a "difficult" reputation.
> Instead of replacing them, systemd could have provided optional or mandatory services (eg time, dhcp, etc) that could be passed into systemd
If sntp is not sufficient for your needs, just disable systemd own sntp solution (systemctl disable...) and use chrony (systemctl start / systemctl enable to make it persistent): then `systemctl status systemd-timesyncd.service` will show you it has been disabled
> but I do know people doing on-prem kubernetes loads that have had to...
What did they have to do? `systemctl disable systemd-timesyncd` is not very complicated. If they want to create a service, they can have it disable timesyncd iff chrony has started successfully (in an ExecStartPost), or doing a logic chain.
It's extremely practical: you can create very precise logic chains with systemd (if this and that, then...)
> My initial fights with systemd's dependency management meant that disabling its internal DHCP had several cascading consequences with other things that expected networking to be "up". This is to say nothing of the tooling that expects a full-slate of systemd services.
Maybe I can help you with that - I used to run a non systemd DHCP when I needed a rapid DHCP to reduce the TTFB for a specific configuration (read https://news.ycombinator.com/item?id=2755461 if you are not familiar with rapid DHCP)
It did not require much effort, just "inserting" my own service in the dependency chain.
Create a /etc/systemd/system/[email protected] with : (put the right option for your dhcp in execstart)
```
[Unit]
Description=dhcpcd on %I
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device
[Service]
Type=forking
ExecStart=/usr/bin/dhcpcd -q -w %I
ExecStop=/usr/bin/dhcpcd -x %I
ExecStop=/usr/sbin/ip addr flush dev %I
[Install]
WantedBy=multi-user.target
```
Using the @ syntax, you can create a dhcp just for the one interface you care about: then use systemctl daemon-reload, and start your custom service as my-dhcpd@eth1 : the %i will be replaced by eth1
If you are interested in rapid dhcp, I give more details on https://www.reddit.com/r/archlinux/comments/1392wrc/dhcpcd_1...
> I am only now starting to plan the upgrade of our ubuntu 20.04 fleet to 24.04, so maybe it is better now...
It shouldn't be very hard to fix, I had a similar setup working with a 18.04 and a 20.04
> My initial fights with systemd's dependency management meant that disabling its internal DHCP had several cascading consequences with other things that expected networking to be "up". This is to say nothing of the tooling that expects a full-slate of systemd services.
You need to insert yourself in the dependencies, but the example I gave above should be sufficient to do that
> This is now mostly deprecated by SLAAC, but DHCPv6 deployments will be needed for awhile as it's all that windows' IPv6 implementation supported until recently. You can't just say "RFC now supports SLAAC...don't whine to me that DHCPv6 doesn't work as it's not part of the RFC".
I have maintained such solutions and I had both SLAAC and DHCPv6: giving multiple IPv6 to an host is fine. On linux side, you can even use preferred_lft to change the priority (ex: use 0 to have an IPv6 address that will reply to connections, but that will not be used to initiate connections, for example if you don't want to expose "easy to remember" short addresses like yourprefix:subnet::1
In case you don't know, systemd-networkd IPv6AcceptRA is a separate option (and section) from DHCPv4 and DHCPv6 - you could decide to use either or both, and also do more advanced routing with RouteMetric to prefer one over the other if you have different gateways for redundancy
If what I've explained is not clear, try to read https://wiki.archlinux.org/title/Systemd-networkd it's a good starting guide.
If you have more advanced IPv6 needs, it's usually to request a /56: use PrefixDelegationHint
You can then chunk it out with systemd: check https://major.io/p/dhcpv6-prefix-delegation-with-systemd-net... for a nice guide
> People have tried, but the response can be TL;DR'd to "works on my machine/setup" or to go harass AWS, Microsoft (via AD), etc to make things work the way the systemd mainainers think is right.
Everything we talked about doesn't seem very complicated.
If with all the pointers I've provided you still can't make it work, or if you run into more complicated problems, send an email (username at outlook), I will see what I can do.
> What did they have to do? `systemctl disable systemd-timesyncd` is not very complicated. If they want to create a service, they can have it disable timesyncd iff chrony has started successfully (in an ExecStartPost), or doing a logic chain.
This was over 5 years ago so I don't recall the details, but it did involve an issue/bug with systemd's handling of the way its internal services were disabled - it MAY have been unique to RHEL. Their fix was ultimately to upgrade RHEL that by then defaulted to chrony, but they were planning on that upgrade anyways. At the time I was making fun of my friend for using Kubernetes, which IMO is another overly complicated solution. :-P
> Maybe I can help you with that - I used to run a non systemd DHCP
Thanks, but I did solve most of the problems; it was a pain as both documentation and other issues (that again may be resolved) at the time made it a much larger hassle and we were an ubuntu shop moving from upstart to systemd that ubuntu 16.04 moved to. Now that systemd has "won out" (and I have accepted that it has), there's more clear and concise examples in documentation and stack overflow. I still dislike that it bloated out what IMO should be just a service and process management framework, though.
> RE IPv6
As I said, IPv6 (and IPSec) was used as an example of how flexibility is required when supporting services or protocols (and I'm a bit pleasantly surprised that systemd-networkd does), not an issue I was actually fighting. Also, enough consumer ISPs hand out /64's that flexibility isn't easy.
I'm also a strong believer that network configurations shouldn't be "chunked" out into multiple files, as it makes it harder to get a holistic configuration view. Fortunately that's a rare experience I need to deal with in today's world and I don't use linux as a router.
> At the time I was making fun of my friend for using Kubernetes, which IMO is another overly complicated solution. :-P
Using Kubernetes is rarely justified because most companies are NOT at google scale :)
> I'm a bit pleasantly surprised that systemd-networkd does
There are many very pleasant surprises with it. I only wish it integrated better with intel iwctl
I'd also be happy if it duplicated iwctl functions so I could get rid of it, and it would be even better if it could also replace bluez: wireless is getting more integrated, with the new WIFI protocols giving a role to BLE (ex: Wifi-Aware 4.0 Instant Communication)
Read about Wifi Aware 4.0 and you will see why having systemd-network work with iwctl and bluez would be much harder than integrating them
> I'm also a strong believer that network configurations shouldn't be "chunked" out into multiple files
Oh no it's about chunking out your /56 prefix to multiple /64 subsets within that prefix. In that specific case, I agree with the article that systemd also made that simpler compared to what needs to be done when not using systemd
> Thanks, but I did solve most of the problems
Great! Should you run into problems during your 24.04 migration, my offer still stands: I think systemd is a nice tool, and I'll be happy to show you how to use it for advanced usecases
Don't hesitate to ping me on ycombinator if I don't reply to your email (the spam filter is not optimal)