Why pro-systemd and anti-systemd people will never get along

I was inspired to write this by the recent announcements of public desire for a Debian fork, an idea that I find to be dumb and which likely will not lead to a lot of technical work.

Nonetheless, I saw the same systemd debate unfold again. I’ve seen it countless times already, and there was virtually no variation from the archetypal formula. You have two ardent and vocal sides, roughly classified into an opponent/proponent dichotomy, neither of which have anything enlightening to say and both with their own unique set of misunderstandings that have memetically mutated into independent ideas that poison virtually every debate of this nature.

I largely avoid systemd “debates” these days. They depress me due to all of the flawed reasoning and shitflinging emerging everywhere, but I felt that perhaps this little write-up could try to explain the background and causes for just why systemd inspires so much vitriol and turf warring.

common mistakes made by proponents and opponents

The fallacies that systemd opponents make have been pointed out so many times that it’s not worth giving a comprehensive overview. For the purposes of this illustration, we will narrate and respond to common arguments for and against systemd that are seen in nearly every “polite and nuanced discussion” of it. The point here is to illustrate that, contrary to popular belief, it’s not just the opponents with their annoying ranting about the “Unix philosophy” and technical incredulity who are wrong, but that just about everyone involved in arguing is clueless in one way or another.

Dumb arguments made by systemd supporters have largely been left unaddressed and left to propagate as equally as dumb arguments made by its opponents, though fewer will acknowledge the former. This was, until, on September 26 of 2014, Judec Nelson wrote a comprehensive blog post called “Systemd: The Biggest Fallacies”, which became huge on /r/linux and garnered comparatively little response on HN.

Let’s take a look at the reception on /r/linux.

The top comments include someone screaming back “fallacy fallacy”, a partial rebuttal to the points that has some key errors in its reasoning (like failing to separate init system from its rc scripts), and largely responds to the paragraph blurbs instead of the deeper elucidations hidden within, a person who makes a non sequitur by responding to a single statement in a way that is unrelated to its actual meaning, a copout argument about how people should stop yelling about software preferences, and some positivity or nuanced discussion mixed in all over. Most objectionable is the rancid vitriol written by a user who calls the author a “filthy fucking concern troll”.

In essence, we see that nearly everyone is doing the best they can to avoiding addressing anything, immediately becoming defensive.

This is not meant as an indictment on systemd proponents, but rather to show one thing: the systemd debate is rarely a technical argument for either side, instead it is an ideological and cultural war waged by two opposing demographics that inhabit the same general sphere of Linux and FOSS. This isn’t about technical merits, it’s about politics. Few would acknowledge it, but the people who argue are not really concerned with improving the state of process management systems. They are involved in an unprecedented philosophical rift in free software, which will likely have some notable implications in the future. Identities and egos are at stake here.

Moving on.

monolithic architecture

“systemd is monolithic! It implements everything inside PID1!”, says the systemd opponent, being totally wrong.

“systemd is not monolithic! Think of the 69 binaries!”, replies the systemd proponent, also being totally wrong.

Architectural arguments about systemd tend to be messes of jargon with differing interpretation, often with people assuming that something being modular automatically makes it non-monolithic. The trouble is that this is actually correct by certain definitions. On the other hand, something being both modular (at least to a degree) and monolithic simultaneously is also absolutely possible and considered the correct interpretation for various software like X.Org, Busybox and the Linux kernel.

This sort of linguistic warfare is typically a distraction from some of the more profound technical decisions made by systemd, and leads to a self-imposed lack of mutual understanding. Opponents will often refuse to drop the falsehood of a huge blob stuck inside PID1, but proponents will also refuse to admit that systemd’s modularity is not as extensive as it could be, and that its PID1 is still larger than most other systems, regardless.

that dirty devil lennart

Often initiated by opponents, they will lament on the horrors of PulseAudio and point out their scorn for Lennart Poettering. This later became a common canard for proponents to dismiss criticism as Lennart-bashing. Futile to even discuss, but it’s a staple.

Lennart’s character is actually, at times, relevant.. Trying to have a large discussion about systemd without ever invoking him is like discussing glibc in detail without ever mentioning Ulrich Drepper. Most people take it overboard, however.

just world hypothesis

A lot of systemd opponents will express their opinions regarding a supposed takeover of the Linux ecosystem by systemd, as its auxiliaries (all requiring governance by the systemd init) expose APIs, which are then used by various software in the desktop stack, creating dependency chains between it and systemd that the opponents deemed unwarranted. They will also point out the udev debacle and occasionally quote Lennart. Opponents see this as anti-competitive behavior and liken it to “embrace, extend, extinguish”. They often exaggerate and go all out with their vitriol though, as they start to contemplate shadowy backroom conspiracies at Red Hat (admittedly it is pretty fun to pretend that anyone defending a given piece of software is actually a shill who secretly works for it, but I digress), leaving many of their concerns to be ignored and deem ridiculous altogether.

Moderation is rarely seen. Most proponents usually resort to outright denying any political meddling and insist that systemd is being adopted based solely on its technical merits, and that such a wide consensus by Linux distros cannot possibly be forged, but has to have valid technical reasons that clearly show systemd to be superior behind it.

Neither of these views are strictly true. There most likely is no Red Hat conspiracy invading all of the distribution bureaucracies (as amusing as it sounds), but to say that systemd adoption is purely technical and not at all political, is equally false. This is especially evident from the fact that just about every Linux distribution (though particularly Debian) has an elaborate governmental structure. Technical and political decisions overlap very frequently (just ask Microsoft and Oracle), and free software is not necessarily immune from it.

In addition, the Linux community is known for reinventing the square wheel over and over again. Chaos is both Linux’s greatest strength and its greatest weakness. Remember HAL? Distro adoption is not an indicator of something being good, so much as something having sufficient mindshare.

That said, the phenomenon appears not so much to be that people deny that political meddling occurs in software, but rather denying that it occurs in their camp. Opponents are also guilty of this (like in Ian Jackson’s reopening the Debian systemd resolution, which was certainly fiery in nature, regardless of its intention), though they are simply not as influential.

Expecting systemd debates not to involve politics, as such, is rather self-contradictory. systemd is already the most politically charged topic in the history of free software, and it is not so arbitrarily. Nonetheless, both sides are frustratingly obtuse and we will analyze their generic backgrounds later on, to see why.

portability

Portability in init systems is rare, largely because stages 1 (immediately after the kernel starts init(8)) and stages 3 (shutdown) are inherently system-specific. This is also a major reason for much of the philosophical drift on init designs, even if infrequently stated so explicitly.

Proponents rightfully argue that systemd being unportable is typical and not a solid argument against it, but opponents typically point out the politics and the dependency creep, arguing that systemd doesn’t need to be init and should just be a process supervisor on top of an existing init. There are justifications to be made for both cases, though systemd’s architecture in general lends it to be an init. It trades greater security against critical errors in expense of a larger coverage, unlike the daemontools approach where you have a strict separation between system manager and service manager.

So while the portability argument would normally be a fallacy, systemd exposing APIs for other packages to use makes it relevant. Of course, the “just-world hypothesis” plays a role here, as due to this argument being intrinsically political, it is therefore often rejected outright and leads to more bang-your-head-against-the-wall (“What we’ve got here is... failure, to communicate. Some men, you just can’t reach.”) situations during debate.

it’s about choice

Generally, a proud Gentoo user who understands the importance of unwinding all loops when compiling a package, will rise up and proclaim “Linux is about choice! systemd is encroaching on all of my software and taking away my freedom to choose!”

They will almost immediately be rebutted by someone who links them to islinuxaboutchoice.com and sometimes even goes as far as to explain why, in fact, choice is a bad thing and leads to sadness. If only we all voted for the same political party, the world would be a better place.

Excuse the caricatures, but the gist of it is that both approaches are wrong and lead to nowhere. The proponent is correct in that Linux isn’t inherently about choice, but they also often fail to realize that the Linux tradition of building systems by mixing and matching white-box components (as a result of Linux only being a kernel) has actually been quite instrumental in shaping its culture, and no doubt a reason why so many companies base their infrastructure on Linux, despite its more restrictive licensing compared to the BSDs.

In fact, none of it is about “choice”. What is actually intended is “breaking workflows”.

cyanide-laced semantics, or the manipulation of language

As it turns out, most people don’t even know what systemd is. I develop uselessd, and I don’t either, really. At least, I cannot think of a concise explanation that properly conveys systemd’s scope.

When lamenting about all the criticism (read: unabashed hatred) about systemd, the proponent will often ask “Why do so many people care about their init system? I don’t care, as long as it just works!”

The opponent will then voice their concerns by saying something to the effect of “I don’t like how systemd handles so many things like logging, containers, locale, logins, mount and automount points...”

At which point another person will yell out “But systemd isn’t just an init system, it’s a set of daemons and services for a Linux-based operating system!”

Indeed, we will see systemd variously referred to as:

etc.

Usually, when a proponent is trying to defend systemd they’ll insist it’s “just an init system”, but when challenged they will instead mention one of the broader definitions.

All of the above definitions are technically correct. None of them are complete.

It is thus that debates are doomed to go nowhere, as no one can agree on a single definition of systemd, and the developers have been largely unhelpful in that matter. Vagueness can be a strength, in that it allows for people to divide and conquer, but it is also unproductive and leads to unnecessary bloodshed. In addition, vagueness gives carte blanche for dismissing concerns and consolidating features, while also leading to a lot of ignorance from opponents and proponents alike.

boot times of great justice

systemd opponents frequently assume that the only reason people like systemd is faster boot times. This is false, though it certainly doesn’t help that fans of systemd praise that aspect a lot, even though the systemd developers themselves discourage using it as an argument, insisting it’s so because of “good design” instead (and thus the circle of flame warring continues).

The developers acknowledge that boot times aren’t as stellar by default as they could be, and most Linux distros accumulated a lot of hacks throughout the years like insserv and startpar that sped up SysV performance by augmenting its serial execution through parallelization hacks in the form of running services as background jobs.

unix philosophy

The Unix philosophy is a gigantic flamefest of its own. It is remarkably politically charged. Many people will actually agree with a large amount of its points if you present them independently, and indeed, some of it has become synonymous with good design, but if you mention it under the umbrella of the “Unix philosophy”, bad things will ensue.

Most systemd opponents usually just blindly resort to screaming “Yoonax phoolawsaphay” like it means anything without elaborating on it, but a lot of proponents are also unusually hostile towards it and themselves misunderstand it. A common rebuttal of “Linux is not Unix” is only correct from a licensing standpoint, in that Linux is indeed, not a licensed Unix. This does not negate the fact that its architecture is intrinsically related to that of a Unix.

An email response that we received from Lennart Poettering, following an inquiry from one of our forum members, said the following:

"That all said, I am pretty sure that systemd actually brings Linux much closer to Unix than Linux ever was. All the true Unixes of today (such as FreeBSD, Solaris, ...) are maintained in central place, sharing code repository infrastructure, lifecycles and release schemes, for all their compenents, regardless if its the kernel, the libc or the rest of userspace. On real Unixes it is much easier to patch through the entire stack from kernel to userspace, because everything comes from the same source, and follows the same cycles. Real UNIXes tend to feel more uniform, since the same folks work on the whole stack, things come out of a single hand. Linux always was different: our components are independently maintained, in different repos, in different coding styles, by different people, with different release cycles. They are differently well maintained, a lot of our stack is actually traditionally very badly maintained if at all.

With systemd we kinda try to find the middle position, move to a more UNIX-like scheme, without dumping everything into the same repo like UNIX does, but at least the core userspace bits. But even beyond the procedural bits, there's a lot of areas where systemd is closer to traditional UNIX than Linux was. For example, one Unix mantra is "everything is a file" (which, btw, is a pretty broken one, because my printer is not a file, not at all), and you could say that systemd exposes one of the most core concepts of a Unix system, that of services/daemons as files via the cgroup logic. So yeah, if you claim that we are not UNIX, then I'll tell you that we are actually much closer to UNIX in many ways then we ever were.

I am pretty sure that most folks who constantly repeat that systemd wasn't Unix-like like a parrot actually have no idea what UNIX really is...”

Apparently, Lennart’s takeaway from Unix is “developed in a single repository” (though this has never been a strict requirement... it’s just that individual OS kernels never really took off before the same way Linux did) and that systemd is more Unix-y than anything before it because it uses cgroupfs... even though the credit for cgroupfs’ design belongs to the kernel developers, not systemd (that and the Solaris contract file system, which was probably a major inspiration for cgroups). In addition, he enlightens us on the fact that his printer is not a file. Evidently, Lennart hasn’t heard of 9P, which is an example of a simple protocol that was successfully applied to expose nearly all system resources and services as virtual file systems, thus being controllable through the most basic of system calls that manipulate file descriptors.

writing an alternative

After much bickering, someone will just pine in that if opponents don’t like systemd, they should write an alternative.

Of course, at least a dozen or so of them already exist. What is actually meant by “Write an alternative”?

systemd is really the userspace middleware layer that sits between GNU and Linux. I’ve also heard comparisons of systemd to servers for a microkernel, much like the Hurd. This actually makes more sense than it seems to at first glance, and with former kernel components like the Linux console being shelved out to userspace (first as kmscon, then as systemd-consoled - not implying that this is bad, by the way) does give some credence to the idea. systemd’s components often implement former standalone tools into auxiliary, programmable daemons.

As such, for one to write an alternative to systemd, a second systemd must in essence be written.

Yet this is a catch-22, as it is exactly what proponents don’t want. Many of the technically competent anti-systemd people subscribe to wholly different philosophies, such as the daemontools approach, or something else.

Further still, many ignore that writing a second systemd would be a huge risk. Doing lower level systems programming and userspace plumbing is not like doing end-user application programming. A process supervisor and a word processor are wholly different. The latter can be swapped out trivially if it is inadequate, the former requires integration and has fundamental ramifications for how one interacts with a given operating system. The task is non-trivial and has a large risk of being all for nothing. A lot of people would rather hack on personal projects than expend effort into writing systems that they largely believe have already been solved by separate and discrete tools (no doubt some of which they themselves have written).

In this regard, the “implement an alternative” argument only leads to poisoning the debate through an implicit assumption that systemd’s philosophy is inherently correct, and thus largely making things even more politicized.

sysvinit: the eternal red herring

It seems that through some sort of divine destiny, every systemd debate will devolve into a dead end argument about the advantages of systemd over sysvinit. The proponents will happily talk about all the nifty features of systemd that replaced all the ugly shell script boilerplate, whereas the opponents will proclaim sysvinit’s superiority in its minimalism and infinite flexibility, due to delegating services to a transparent and interpreted command language that is Turing complete.

Both are wrong and miss the point entirely.

The observation that sysvinit is dumb and heavily flawed with its clunky inittab and runlevel abstractions, is absolutely nothing new. Richard Gooch wrote a paper back in 2002 entitled “Linux Boot Scripts”, which criticized both the SysV and BSD approaches, based on his earlier work on simpleinit(8). That said, his solution is still firmly rooted in the SysV and BSD philosophies, but he makes it more elegant by supplying primitives for modularity and expressing dependencies.

Even before that, DJB wrote the famous daemontools suite which has had many successors influenced by its approach, including s6, perp, runit and daemontools-encore. The former two are completely independent implementations, but based on similar principles, though with significant improvements. An article dated to 2007 entitled “Init Scripts Considered Harmful” encourages this approach and criticizes initscripts.

Around 2002, Richard Lightman wrote depinit(8), which introduced parallel service start, a dependency system, named service groups rather than runlevels (similar to systemd targets), its own unmount logic on shutdown, arbitrary pipelines between daemons for logging purposes, and more. It failed to gain traction and is now a historical relic.

Other systems like initng and eINIT came afterward, which were based on highly modular plugin-based architectures, implementing large parts of their logic as plugins, for a wide variety of actions that software like systemd implements as an inseparable part of its core. Initmacs, anyone?

Even Fefe, anti-bloat activist extraordinaire, wrote his own system called minit early on, which could handle dependencies and autorestart. As is typical of Fefe’s software, it is painful to read and makes you want to contemplate seppuku with a pizza cutter.

And that’s just Linux. Partial list, obviously.

At the end of the day, all comparing to sysvinit does is show that you’ve been living under a rock for years. What’s more, it is no secret to a lot of people that the way distros have been writing initscripts has been totally anathema to basic software development practices, like modularizing and reusing common functions, for years. Among other concerns such as inadequate use of already leaky abstractions like start-stop-daemon(8). Though sysvinit does encourage poor work like this to an extent, it’s distro maintainers who do share a deal of the blame for the mess. See the BSDs for a sane example of writing initscripts. OpenRC was directly inspired by the BSDs’ example. Hint: it’s in the name - “RC”.

The rather huge scope and opinionated nature of systemd leads to people yearning for the days of sysvinit. A lot of this is ignorance about good design principles, but a good part may also be motivated from an inability to properly convey desires of simple and transparent systems. In this way, proponents and opponents get caught in feedback loops of incessantly going nowhere with flame wars over one initd implementation (that happened to be dominant), completely ignoring all the previous research on improving init, as it all gets left to bite the dust. Even further, most people fail to differentiate init from rc scripts, and sort of hold sysvinit to be equivalent to the shoddy initscripts that distros have written, and all the hacks they bolted on top like LSB headers and startpar(2). This is a huge misunderstanding that leads to a lot of wasted energy.

Don’t talk about sysvinit. Talk about systemd on its own merits and the advantages or disadvantages of how it solves problems, potentially contrasting them to other init systems. But don’t immediately go “SysV initscripts were way better and more configurable, I don’t see what systemd helps solve beyond faster boot times.”, or from the other side “systemd is way better than sysvinit, look at how clean unit files are compared to this horribly written initscript I cherrypicked! Why wouldn’t you switch?”

cultural and technical backgrounds of proponents/opponents

Now that we pointed out how most systemd debates play out in practice and why it’s usually a colossal waste of time to partake in them, let’s do a crude overview of the personalities that make this clusterfuck possible.

The technically competent sides tend to largely fall in these two broad categories:

a) Proponents are usually part of the modern Desktop Linux bandwagon. They run contemporary mainstream distributions with the latest software, use and contribute to large desktop environment initiatives and related standards like the *kits. They’re not necessarily purely focused on the Linux desktop. They’ll often work on features ostensibly meant for enterprise server management, cloud computing, embedded systems and other needs, but the rhetoric of needing a better desktop and following the example set by Windows and OS X is largely pervasive amongst their ranks. They will decry what they perceive as “integration failures”, “fragmentation” and are generally hostile towards research projects and anything they see as “toy projects”. They are hackers, but their mindset is largely geared towards reducing interface complexity, instead of implementation complexity, and will frequently argue against the alleged pitfalls of too much configurability, while seeing computers as appliances instead of tools.

b) Opponents are a bit more varied in their backgrounds, but they typically hail from more niche distributions like Slackware, Gentoo, CRUX and others. They are largely uninterested in many of the Desktop Linux “advancements”, value configuration, minimalism and care about malleability more than user friendliness. They’re often familiar with many other Unix-like environments besides Linux, though they retain a fondness for the latter. They have their own pet projects and are likely to use, contribute to or at least follow a lot of small projects in the low-level system plumbing area. They can likely name at least a dozen alternatives to the GNU coreutils (I can name about 7, I think), generally favor traditional Unix principles and see computers as tools. These are the people more likely to be sympathetic to things like the suckless philosophy.

It should really come as no surprise that the former group dominates. They’re the ones that largely shape the end user experience. The latter are pretty apathetic or even critical of it, in contrast. Additionally, the former group simply has far more manpower in the right places. Red Hat’s employees alone dominate much of the Linux kernel, the GNU base system, GNOME, NetworkManager, many projects affiliated with Freedesktop.org standards (including Polkit) and more. There’s no way to compete with a vast group of paid individuals like those.

conclusion

The “Year of the Linux Desktop” has become a meme at this point, one that is used most often sarcastically. Yet there are still a lot of people who deeply hold onto it and think that if only Linux had a good abstraction engine for package manager backends, those Windows users will be running Fedora in no time.

What we’re seeing is undoubtedly a cultural clash by two polar opposites that coexist in the Linux community. We can see it in action through the vitriol against Red Hat developers, and conversely the derision against Gentoo users on part of Lennart Poettering, Greg K-H and others. Though it appears in this case “Gentoo user” is meant as a metonym for Linux users whose needs fall outside the mainstream application set. Theo de Raadt infamously quipped that Linux is “for people who hate Microsoft”, but that quote is starting to appear outdated.

Many of the more technically competent people with views critical of systemd have been rather quiet in public, for some reason. Likely it’s a realization that the Linux desktop’s direction is inevitable, and thus trying to criticize it is a futile endeavor. There are people who still think GNOME abandoning Sawfish was a mistake, so yes.

The non-desktop people still have their own turf, but they feel threatened by systemd to one degree or another. Still, I personally do not see them dwindling down. What I believe will happen is that they will become even more segregated than they already are from mainstream Linux and that using their software will feel more otherworldly as time goes on.

There are many who are predicting a huge renaissance for BSD in the aftermath of systemd, but I’m skeptical of this. No doubt there will be increased interest, but as a whole it seems most of the anti-systemd crowd is still deeply invested in sticking to Linux.

Ultimately, the cruel irony is that in systemd’s attempt to supposedly unify the distributions, it has created a huge rift unlike any other and is exacerbating the long-present hostilities between desktop Linux and minimalist Linux sides at rates that are absolutely atypical. What will end up of systemd remains unknown. Given Linux’s tendency for chaos, it might end up the new HAL, though with a significantly more painful aftermath, or it might continue on its merry way and become a Linux standard set in stone, in which case the Linux community will see a sharp ideological divide. Or perhaps it won’t. Perhaps things will go on as usual, on an endless spiral of reinvention without climax. Perhaps we will be doomed to flame on systemd for all eternity. Perhaps we’ll eventually get sick of it and just part our own ways into different corners.

Either way, I’ve become less and less fond of politics for uselessd and see systemd debates as being metaphorically like car crashes. I likely won’t help but chime in at times, though I intend uselessd to branch off into its own direction with time.