[CentOS] What are the differences between systemd and non-systemd Linux distros?

Thu Oct 18 00:55:09 UTC 2018
Warren Young <warren at etr-usa.com>

On Oct 17, 2018, at 3:28 PM, Mark Rousell <mark.rousell at signal100.com> wrote:
> On 17/10/2018 20:03, Warren Young wrote:
>> On Oct 17, 2018, at 10:03 AM, Mark Rousell <mark.rousell at signal100.com> wrote:
>>> launchd is not being forced on them as systemd is in practice
>> Try doing without launchd on macOS.
> If launchd was on Linux and it had systemd's cultural
> issues and, in many people's views, technical issues then the opposition
> to it would be identical to the opposition to systemd.

Try this gedankenexperiment instead: what if RHEL 7 shipped based on launchd instead of systemd, with no differences relative to the version shipped in the contemporaneous version of Mac OS X?

I’m uncertain as to whether the opposition would have been as great, but I’m dead certain the opposition would have been vociferous and strident, because Linux, though less conservative than the BSDs, is much more conservative than macOS.

The systemd vs launchd vs sysvinit vs whatever-else arguments are more about human factors than about technology, even though they’re usually couched as technical battles.

> When people go to Mac they accept what it is (mostly).

I doubt most Mac people even know launchd exists, much less have an informed technical opinion about it.  And of the small minority that do have such an opinion, it almost certainly doesn’t drive buying decisions.  Maybe that’s what you mean by accepting macOS as it is.

The thing I don’t get is, why is it different in the Linux world?  Why did we in the Linux community spend so much time arguing about systemd over the past several years?  Why is it still an active topic five years after the key events?  And why is the BSD community continuing to stir the pot?

Here’s what I want to see: I want one of the BSDs to clean Linux’s clock with a thoroughly awesome modern init system that makes us Linux fans so jealous we start noisily advocating to replace systemd with it, much as ZFS is starting to replace the horrid lash-up that is ext4/xfs+md+LVM+DM.

What I *don’t* want is more of this retrenchment to SysVInit.  I liked it well enough within its limitations, but we can do better in 2018.

(It’s a related tragedy that a slightly modified ksh88 remains the most powerful general purpose scripting language mandated by POSIX three decades after it was released by AT&T.  We’ve got better alternatives here, too.)

>> For an init system to gain sufficient momentum, it must be the default, with no easy way to avoid it.
> That's an argument for authoritarianism

I call it leadership.  Working code argues best.

Benno Rice is right: Lennart Poettering gets stuff done.

In the BSD world, they call the opposite tendency bikeshedding.  You can find a thousand people willing to argue about why something shouldn’t be done, or why it shouldn’t be done *that* way for every person capable, willing, and available to write a given piece of software.

For all the complaints about systemd over the past several years, I note that there is still no fork of CentOS 6 or CentOS 5, keeping the prior init system(s) but updating their package set to recent versions.  Many would rather gripe about change than put in the work it takes to maintain the status quo.

Then you get the crowd that tries to argue that we should just stay with what works, apparently under the misapprehension that staying in place is a zero-effort choice, when in reality it is at best an accrual of technical debt; the bill will come due eventually, with interest.

I suspect these two groups overlap quite a lot, inconsistent though the combined position is.

> the fact that some people do dislike change (a) does not
> make the substantive and objective problems, in many people's views,
> with systemd any less real or important

Of course not, but I also don’t see a lot of effort going into replacing systemd with something better.  Most of the effort opposing systemd seems to be going into anti-systemd advocacy campaigns, plus a tiny slice off to the side going into retrenchment to SysVInit.  That’s conservatism, plain and simple.

> he effectively claimed that it was all to do with fear of change when,
> as you agree, there in fact are substantive, real and objective issues
> which are widely recognised.

I suspect that for many people, those are rationalizations rather than reasons.

I saw much the same sort of logic in the Unix vs Linux wars, roughly spanning the decade surrounding 1996.

The Big Iron Unix and SCO Unix fans had all kinds of myopically rational reasons why Linux wasn’t going to replace their OS of choice: journaled filesystems, better SMP, STREAMS, hot-swap hardware, real ksh93 instead of that cheesy nonstandard imitation Bash… 

The analogy I used at the time is that the Unix fans saw Linux surfing behind their big yacht and laughed at the tiny, flimsy, cheap little surfboard, not realizing that it would take an awfully big wave to allow someone to be surfing out in the middle of the ocean.  We’ve been skimming the splinters of that yacht off the surface of the sea for decades now.

Like the Linux wave, the systemd wave took years to build beneath the surface.  If you perceive that it broke fast, it’s only because you weren’t looking in the right places, where you could see and even help direct that long build-up.

You can spend your time rationalizing against systemd while it sheds enough of its weaknesses and gains enough features to sweep the planet.  You’re years late if you want to stop that from happening, but I believe it can be done, just as systemd replaced rc files, SysVInit, and Upstart.

It’ll take a lot of hard work, but it can be done.

> One of these aspects is the "do
> one job and do it well" expectation of componentisation.

That’s never been an especially hard and fast rule.  Witness Emacs, Perl, GNOME, Firefox, ZFS… 

My DVCS of choice is Fossil, which is about as far from the “small pieces loosely joined” philosophy as you can get, from which comes much of its power and simplicity.  Fossil’s primary opposition is Git, which is famously difficult to understand and use properly, even though it adheres perfectly to that philosophy.

(Incidentally, I’ve spent a fair bit of time in recent months making Fossil work better, so my calls to action in this thread are not just me proposing work for other people to do.)

I *want* my init system to build a dependency graph and maintain that graph at run time.  I *want* it to start services in parallel, consistent with that dependency graph.  I *want* it to restart services that die unexpectedly, with exponential back-offs.  I *want* it to have sensible answers to a “status” command without having to write the shell code to dig that that status out of service-specific files.  I *want* my service definition scripts to be 15 lines, rather than 150.  I *want* to be able to start services as a normal user, with all of the power of a real system service, limited only by OS permissions, without needing sudo.

That puts an awful lot of code into a single place.  It is not small pieces loosely joined.  What it *is* is powerful.

> In many
> people's views, systemd wilfully and unnecessarily tramples all over
> this cultural/technical requirement. If this is the case in many
> people's views, then it makes a lot of sense that hey are unhappy with it.

I’m not a particularly big fan of systemd.  I agree with much of what its opponents dislike it for.

The only reason I enter these arguments is that I feel the need to fight against the complainers, because I’d rather see opposition deflected toward a replacement.

Give me a better option, and I’ll happily switch to it.

Tricky bit: “better option” is evaluated on a systems basis, not on a per-feature basis.  If the combined system doesn’t provide most of CentOS’ key virtues, as I see them, it isn’t a better option.

If you ask why I don’t just write something better myself, it’s because I don’t care enough about the Linux init system to do that, so I’d rather spend time advocating for and taking advantage of systemd’s features than complaining about its weaknesses.

(Automatic service restarts saved me a lot of work just a few weeks ago!)

I’m also unwilling to stick with increasingly poorly supported technology.  If the cheese moves, I’m following the cheese.

>> Alternatives to the BSD rc init system are readily available, yet I think if you were to survey actual use, you’d find that over 99% of BSD boxes use the stock init system.
> That's a different metric. People may well stick with the stock init
> system but that doesn't mean that they like it or really want it.

That’s a large part of my point: if you are unable or unwilling to take the time to replace a thing, why bother spending time griping about its weaknesses?

Talk about a first world problem: Oh, waaah, my automated software management service isn’t ideally designed.  Woe is meeee!

It’s much like my macOS argument above: CentOS sucks in many ways, but I have no good answer to the “What’s better?” question for the applications where I use it.  I’m not going to spend a bunch of time listing all the ways CentOS sucks: either I’ll fix the problems myself, or I’ll work around them, or I’ll ignore them.

Notice that putting up with systemd is only one of your several choices.  The FOSS ecosystem advances every time someone scratches an itch.  My objection is that we’ve got all these people claiming to have itches, but very, very few are scratching.

>> Change has to be forced from the center out on this kind of thing.
> Again, an appeal to authoritarianism. Excuse me if I don't wish to join
> you on that. Authoritarianism, in all its forms, is dangerous and, in my
> view, a form of vandalism.

In what useful sense do you believe you own the parts of CentOS affected by the move to systemd?  It can’t be vandalism otherwise.

Intellectual property is no answer in the FOSS world, since that implies that you’ve relinquished the relevant rights already, so you have little to no say in how the wall gets reshaped, if in fact you built it in the first place.

I suspect it’s more that you found this free wall just sitting around online, and no one was guarding it, and in fact there were people offering you free copies of the wall, as many as you want, so you started making buildings out of these walls.  Then someone changed the wall generator to produce differently-shaped walls, and now you’re upset that the walls aren’t the same any more.

I get it: you’re annoyed.  What I don’t get is why you believe your annoyance should, of itself, effect a change in the world.

FOSS is a do-ocracy: he who does the work makes the rules.  If you want the wall to be shaped differently, and no one is making new walls with that shape any more, go make your own wall, or maintain an old wall that was already correctly-shaped, by your lights.

In this case, I think the lowest friction path is to fork CentOS 5 or 6, then backport as much of the feature set as is possible without dragging in systemd.  No one’s done this, despite having over 7 years to react, dating from systemd’s deployment as the default init system in Fedora.  Why not?

The closest thing I’m aware of in the Linux world is Devuan, which was low in popularity a year ago and has been dropping ever since:


According to the same stats, CentOS is pretty stable, which I find appropriate. :)

> Also one might ask: What centre?

In this case, the one currently headquartered in Raleigh, NC, USA.

If you want a different centre, pick one, or be your own.  That’s why we have several major Linux flavors and hundreds of Linux distros.

Centralized control is clearly not the key problem here, since so much of this anti-systemd sentiment is coming from the much more centralized BSD camps.

Centralized control didn’t do the Big Iron Unix vendors much good.  A better thing came along and wiped them out.  The thing is, a whole lot of programmers went to a whole lot of effort to make that happen.  Are you going to do more than post to mailing lists in an effort to wipe out systemd?

> As I observed above, this authoritarian centrism in in part why systemd
> is so despised. It was effectively forced on a great many users in in a
> centrist, choice-limiting manner.

You also didn’t get a choice over SysVInit vs BSD style init, short of switching from Red Hat to Slackware.  So what?

Someone has to make a decision.  We call those who do this often and well leaders.

Leadership is not central control or authoritarianism: we get to choose which leaders we wish to follow.

Don’t like where Poettering is taking his slices of the Linux world?  Fine; follow someone else who is taking equivalent slices in a direction you like better.  Or, be that someone!

> That's up to them and their users, isn't it. Just because it's a good
> idea doesn't mean it has to be done in a hurry. This was and is one of
> the problems with systemd on Linux.

The first version of systemd was released in 2010, and it wasn’t deployed “for real” until RHEL 7 came out in 2014, as far as I’m concerned.  I don’t call that “in a hurry.”

This in a world where Linux’s major competition all had systemd-like service management by around 2005: 

- Solaris 10 came out in early 2005 with the Service Management Facility (SMF), which is very similar to systemd’s initial feature set.

- Several months later, Apple shipped Mac OS X “Tiger” with launchd replacing init(8).

- Microsoft, ever late to the party, shipped the last major piece missing relative to the above about 18 months after Tiger came out, adding delayed-start services to Windows Vista.

(This is probably where the “2005” in my prior post came from, by the way.  I thought systemd got started earlier.)

So yeah, systemd finally let Linux catch up to Vista in the area of background service handling, an area where Linux is supposed to be superior, but it’s all happening too fast?