Hi,
I'm currently reading the upstream "Considerations in adopting RHEL 8" document. The chapter about networking states that traditional networking scripts (shipped with the network-scripts package) are considered obsolete.
I bluntly admit I don't see the point in this. As far as I'm concerned, I've been a happy user of NetworkManager since the early days (when folks used to call it NotworkManager :oD). It's one of those nifty pieces of software that brought the Linux desktop to the masses - or at least a bit nearer to them - since it allows managing wireless and wired interfaces transparently and easily on a laptop or any computer with a wireless card.
On servers though, one of the first post-installation steps I performed was to get rid of Network-Manager and all its components. The servers I'm working on are relatively small-scale and have from one to four network interfaces. Each interface has a corresponding configuration in /etc/sysconfig/network-scripts, and that's it. From there, I rarely - if ever - touch it. In all my setups, NetworkManager is merely a useless layer of abstraction, and I like sticking to the KISS principle and shave off useless layers.
Maybe there's a reason to make NetworkManager more or less mandatory from now on, but I don't see it. So I thought I'd rather ask on this list.
Cheers from the foggy South of France,
Niki
On Mon, 10 Feb 2020 at 02:55, Nicolas Kovacs info@microlinux.fr wrote:
Hi,
I'm currently reading the upstream "Considerations in adopting RHEL 8" document. The chapter about networking states that traditional networking scripts (shipped with the network-scripts package) are considered obsolete.
I bluntly admit I don't see the point in this. As far as I'm concerned, I've been a happy user of NetworkManager since the early days (when folks used to call it NotworkManager :oD). It's one of those nifty pieces of software that brought the Linux desktop to the masses - or at least a bit nearer to them
since it allows managing wireless and wired interfaces transparently and easily on a laptop or any computer with a wireless card.
On servers though, one of the first post-installation steps I performed was to get rid of Network-Manager and all its components. The servers I'm working on are relatively small-scale and have from one to four network interfaces. Each interface has a corresponding configuration in /etc/sysconfig/network-scripts, and that's it. From there, I rarely - if ever - touch it. In all my setups, NetworkManager is merely a useless layer of abstraction, and I like sticking to the KISS principle and shave off useless layers.
Maybe there's a reason to make NetworkManager more or less mandatory from now on, but I don't see it. So I thought I'd rather ask on this list.
The reason is that having 1 way to configure networks makes it so the developer and tech support only have to diagnose issues from 1 set of tools versus two different ones (and occasionally 2 competing ones if both are trying to do their job at the same time). Basically network-scripts has been on the backburner for 10+ years and has to be dusted off every now and then to add a new networking corner case or some other item. For the developer it usually means context swapping back from python (or whatever language they prefer) to bash and then figure out what the problem is.. cause a couple of new ones they then have to fix and then get it right. Or they could do that work in 1 language they know and get it done.
Does it makes sense to us as sysadmins who are happy with a working set of scripts and configs we have to know possibly rewrite? No it doesn't.. but unless one of us takes over the network-scripts and puts in the work to make it work in all the different layers (or pay someone to do so).. we get what the soup kitchen serves :).
Once upon a time, Stephen John Smoogen smooge@gmail.com said:
The reason is that having 1 way to configure networks makes it so the developer and tech support only have to diagnose issues from 1 set of tools versus two different ones (and occasionally 2 competing ones if both are trying to do their job at the same time).
Not only that - the hodge-podge bash network scripts are kind of a mess. It is impressive that they do what they do so reliably after so long, but every new feature appears to have been hacked in by a different developer, leaving parts of them almost indecipherable.
That's not intended as a criticism of the scripts or the people who wrote that code - it's just that IMHO they managed to go beyond what is reasonable in bash scripting, which makes for a difficult to read (and I'm sure fix/extend) set of scripts.
And even on servers now, there are often dynamic network changes that work much better with NetworkManager than the old-style static scripts. Containers, VMs, and VPNs all come and go, and work better with a single system configuring their networks (rather than each layer implementing their own setup).
On 09/02/2020 23:55, Nicolas Kovacs wrote:
Hi Nicolas,
[snip]
Maybe there's a reason to make NetworkManager more or less mandatory from now on, but I don't see it. So I thought I'd rather ask on this list.
Like you, I read about NetworkManager becoming the default tool for CentOS 8. So I sat down with a colleague to figure out how we could use NetworkManager, and convert our existing network configs (on CentOS 6 and 7) to work with NetworkManager.
I'm sad to report that we ran into at least 3 issues (listed below). We found solutions to the first two, but the last one was a show-stopper, and we came to the conclusion that for servers, NetworkManager is still overkill, and for us, actually unusable. So even on CentOS 8, we will keep using the legacy scripts.
1. When NetworkManager activates interfaces, it does not wait for IPv6 DAD to complete. This makes systemd reach the "network-online" target before IPv6 is fully initialised, and some daemons fail to start. We eventually found a work-around, but not before I'd lost some of my hair.
2. NetworkManager doesn't know how to activate dummy interfaces from ifcfg-dummy* files. You have to create dummy interfaces directly in NetworkManager. This is not a problem on CentOS 8, but on CentOS 7, there is a subtle issue with loading the dummy module that makes things fail at boot. We again found the solution, but it's annoying that none of it was documented.
3. Some of our servers run full routing daemons (BIRD), and have multiple route tables. On these, when we start NetworkManager, it attempts to read the entire route tables into memory using the netlink API. This makes it log lots of errors. Then, NetworkManager's RAM usage goes up and up, going to over 3 GB!! Finally, it barfs and dies. And then systemd starts it again, and it goes and does the same.
We have NOT been able to find any solution to this stupidity of NetworkManager. And so we have made the choice to abandon it, and remain with legacy network scripts.
Regards, Anand
On 09/02/2020 23:55, Nicolas Kovacs wrote:
Hi Nicolas,
[snip]
Maybe there's a reason to make NetworkManager more or less mandatory from now on, but I don't see it. So I thought I'd rather ask on this list.
Like you, I read about NetworkManager becoming the default tool for CentOS 8. So I sat down with a colleague to figure out how we could use NetworkManager, and convert our existing network configs (on CentOS 6 and 7) to work with NetworkManager.
I'm sad to report that we ran into at least 3 issues (listed below). We found solutions to the first two, but the last one was a show-stopper, and we came to the conclusion that for servers, NetworkManager is still overkill, and for us, actually unusable. So even on CentOS 8, we will keep using the legacy scripts.
- When NetworkManager activates interfaces, it does not wait for IPv6
DAD to complete. This makes systemd reach the "network-online" target before IPv6 is fully initialised, and some daemons fail to start. We eventually found a work-around, but not before I'd lost some of my hair.
- NetworkManager doesn't know how to activate dummy interfaces from
ifcfg-dummy* files. You have to create dummy interfaces directly in NetworkManager. This is not a problem on CentOS 8, but on CentOS 7, there is a subtle issue with loading the dummy module that makes things fail at boot. We again found the solution, but it's annoying that none of it was documented.
- Some of our servers run full routing daemons (BIRD), and have
multiple route tables. On these, when we start NetworkManager, it attempts to read the entire route tables into memory using the netlink API. This makes it log lots of errors. Then, NetworkManager's RAM usage goes up and up, going to over 3 GB!! Finally, it barfs and dies. And then systemd starts it again, and it goes and does the same.
We have NOT been able to find any solution to this stupidity of NetworkManager. And so we have made the choice to abandon it, and remain with legacy network scripts.
Thanks for confirming that NetworkManager is not the solution for everyone. To me it seems that NetworkManager was developed by laptop users for laptop users and that's why it is what it is today. Useful for laptops/desktops and simple server setups.
Unfortunately, instead of fixing/refactoring the whole bash networking script mess, another new project was started instead, called systemd-networkd :-)
Regards, Simon
On Tue, Feb 11, 2020 at 06:11:04AM +0100, Simon Matter via CentOS wrote:
Thanks for confirming that NetworkManager is not the solution for everyone. To me it seems that NetworkManager was developed by laptop users for laptop users and that's why it is what it is today. Useful for laptops/desktops and simple server setups.
I've mentioned on this list countless times about how NetworkManager is actually pretty good for a general server. Automatic link detection and activation/deactivation, a dispatch service on link activation/deactivation, support for bringing up secondary interfaces after a primary goes up, a dbus interface for automation, etc.
Unfortunately, instead of fixing/refactoring the whole bash networking script mess, another new project was started instead, called systemd-networkd :-)
Actually, I'm sad that RHEL/CentOS 8 doesn't support systemd-networkd. It's really nice, especially for really pared down systems that don't need a lot of extra services like NetworkManager. But I understand that Red Hat needs to focus its support efforts.
On Tue, Feb 11, 2020 at 8:12 AM Jonathan Billings billings@negate.org wrote:
On Tue, Feb 11, 2020 at 06:11:04AM +0100, Simon Matter via CentOS wrote:
Unfortunately, instead of fixing/refactoring the whole bash networking script mess, another new project was started instead, called systemd-networkd :-)
Actually, I'm sad that RHEL/CentOS 8 doesn't support systemd-networkd. It's really nice, especially for really pared down systems that don't need a lot of extra services like NetworkManager. But I understand that Red Hat needs to focus its support efforts.
I thought that systemd was under redhat, so I am confused why they would not be pushing it instead of networkmanager. Am I missing something?
On Tue, Feb 11, 2020 at 08:17:18AM -0500, Mauricio Tavares wrote:
I thought that systemd was under redhat, so I am confused whythey would not be pushing it instead of networkmanager. Am I missing something?
systemd has several Red Hat employees working on systemd, I believe, but it's not a solely Red Hat product. And according to their documentation and tickets opened about systemd-networkd, they're focusing on a single network infrastructure, NetworkManager.
It was available as part of the optional channel in RHEL7 but I guess it caused too much confusion in the market or something. It's still kinda new and I guess having that much churn in your network infrastructure is too much for a enterprise-level OS (given that they'd have to backport fixes rather than bump the systemd version).
On Tue, 11 Feb 2020 at 08:17, Mauricio Tavares raubvogel@gmail.com wrote:
On Tue, Feb 11, 2020 at 8:12 AM Jonathan Billings billings@negate.org wrote:
On Tue, Feb 11, 2020 at 06:11:04AM +0100, Simon Matter via CentOS wrote:
Unfortunately, instead of fixing/refactoring the whole bash networking script mess, another new project was started instead, called systemd-networkd :-)
Actually, I'm sad that RHEL/CentOS 8 doesn't support systemd-networkd. It's really nice, especially for really pared down systems that don't need a lot of extra services like NetworkManager. But I understand that Red Hat needs to focus its support efforts.
I thought that systemd was under redhat, so I am confused whythey would not be pushing it instead of networkmanager. Am I missing something?
So there are two items of complexity which people have a hard time understanding (both inside and outside of Red Hat).
1. Red Hat is a company of 14,000 people many of which have diverging views on how things should be run and why. This means that you may see 4-5 different tools to fix a problem all of which solve the part that they were originally developed for but not for everyone (mainly because the tool that is solving it for everyone is still not out of design yet.)
2. systemd is maintained by multiple companies with divergent interests in how and where to solve things. It is also not a monolithic tool but a 'hurd' of services which all do some vital plumbing. Some of that plumbing works for some things but not all things any more than you put the same pipe under your kitchen sink as your bathroom as the industrial cleaner.. [well you can but it will blow up somewhere.]
This leads to a lot of 'but I thought Red Hat was doing X' which is true but 'Red Hat is also doing Y' or Z and the same for systemd and related groups. Any time you have more than 4 of anything you will start getting factorial number of solutions. (4 sysadmins, 4 developers, 4 managers etc.. at 5 you end up with 120 different solutions for some reason.)
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Le 11/02/2020 à 16:27, Stephen John Smoogen a écrit :
- Red Hat is a company of 14,000 people many of which have diverging views
on how things should be run and why. This means that you may see 4-5 different tools to fix a problem all of which solve the part that they were originally developed for but not for everyone (mainly because the tool that is solving it for everyone is still not out of design yet.)
- systemd is maintained by multiple companies with divergent interests in
how and where to solve things. It is also not a monolithic tool but a 'hurd' of services which all do some vital plumbing. Some of that plumbing works for some things but not all things any more than you put the same pipe under your kitchen sink as your bathroom as the industrial cleaner.. [well you can but it will blow up somewhere.]
This leads to a lot of 'but I thought Red Hat was doing X' which is true but 'Red Hat is also doing Y' or Z and the same for systemd and related groups. Any time you have more than 4 of anything you will start getting factorial number of solutions. (4 sysadmins, 4 developers, 4 managers etc.. at 5 you end up with 120 different solutions for some reason.)
As much as I love CentOS (been using it since 4.x), some days I just miss the bone-headed approach of Slackware and FreeBSD. Just edit /etc/rc.d/rc.inet1.conf or /etc/rc.conf and you're done.
:o)
On Tue, Feb 11, 2020 at 06:29:29PM +0100, Nicolas Kovacs wrote:
As much as I love CentOS (been using it since 4.x), some days I just miss the bone-headed approach of Slackware and FreeBSD. Just edit /etc/rc.d/rc.inet1.conf or /etc/rc.conf and you're done.
Nothing is stopping you from creating and editing a service in /etc/rc.d/init.d/ and enabling it. systemd supports launching SysV service files.
If you edited init files that were owned by packages, though, then you probably had to deal with package replacing them and losing your changes.
Le 11/02/2020 à 14:11, Jonathan Billings a écrit :
I've mentioned on this list countless times about how NetworkManager is actually pretty good for a general server. Automatic link detection and activation/deactivation, a dispatch service on link activation/deactivation, support for bringing up secondary interfaces after a primary goes up, a dbus interface for automation, etc.
I just prepared myself to catch up and learn more about NetworkManager. So I opened my big fat "Unix and Linux System Administration Handbook 5th edition", with a text file open on the computer to take extensive notes...
... only to find out that there is only half a page on NetworkManager in this book. Allow me to quote it:
"NetworkManager is primarily of use on laptops, since their network enviromment may change frequently. For servers and desktop systems, NetworkManager isn't necessary and may in fact complicate administration. In these environments, it should be ignored or configured out."
Hmmmm.
On Thu, 13 Feb 2020 at 11:40, Nicolas Kovacs info@microlinux.fr wrote:
Le 11/02/2020 à 14:11, Jonathan Billings a écrit :
I've mentioned on this list countless times about how NetworkManager is actually pretty good for a general server. Automatic link detection and activation/deactivation, a dispatch service on link activation/deactivation, support for bringing up secondary interfaces after a primary goes up, a dbus interface for automation, etc.
I just prepared myself to catch up and learn more about NetworkManager. So I opened my big fat "Unix and Linux System Administration Handbook 5th edition", with a text file open on the computer to take extensive notes...
... only to find out that there is only half a page on NetworkManager in this book. Allow me to quote it:
"NetworkManager is primarily of use on laptops, since their network enviromment may change frequently. For servers and desktop systems, NetworkManager isn't necessary and may in fact complicate administration. In these environments, it should be ignored or configured out."
The book was published in 2017 which means it was written in late 2016. As much as I love that series of books (I have read them from 1st edition), I do not expect that its comments on parts of Linux in the 3rd edition would be useful now.
In the end, the problem is that NetworkManager, FirewallD, and other 'automatic' helpers are 'part' of the OS.. and while it was easy to tear them out in earlier versions.. as time goes on it is not.
For a car analogy, it was much easier to convert any 1970 car from automatic back to manual as many parts were left over. Now in this era, you can do so if you pick the right car but for a lot of them it is not going to be easy in any form. I see the same trends in computer OS's with certain tools which were easy to pull out now requiring you to build the whole os from scratch as the part is assumed to be in so many other areas.
Le 13/02/2020 à 17:50, Stephen John Smoogen a écrit :
In the end, the problem is that NetworkManager, FirewallD, and other 'automatic' helpers are 'part' of the OS.. and while it was easy to tear them out in earlier versions.. as time goes on it is not.
For a car analogy, it was much easier to convert any 1970 car from automatic back to manual as many parts were left over. Now in this era, you can do so if you pick the right car but for a lot of them it is not going to be easy in any form. I see the same trends in computer OS's with certain tools which were easy to pull out now requiring you to build the whole os from scratch as the part is assumed to be in so many other areas.
I just came to the same conclusion. So it looks like I'll have to catch up and do some RTFM on NetworkManager, FirewallD (which I've replaced by a handcrafted iptables script) and Chrony (replaced by ntpd).
Cheers,
Niki
On Thu, Feb 13, 2020 at 05:53:41PM +0100, Nicolas Kovacs wrote:
I just came to the same conclusion. So it looks like I'll have to catch up and do some RTFM on NetworkManager, FirewallD (which I've replaced by a handcrafted iptables script) and Chrony (replaced by ntpd).
Whatever your views on the first two, I strongly discourage the latter unless you have very specific functionality beyond Chrony's capability. The original ntpd has a very large attack surface. Plus Chrony has some nice additional features. Read more about Chrony here: https://opensource.com/article/18/12/manage-ntp-chrony
On 2020-02-13 10:50, Stephen John Smoogen wrote:
On Thu, 13 Feb 2020 at 11:40, Nicolas Kovacs info@microlinux.fr wrote:
Le 11/02/2020 à 14:11, Jonathan Billings a écrit :
I've mentioned on this list countless times about how NetworkManager is actually pretty good for a general server. Automatic link detection and activation/deactivation, a dispatch service on link activation/deactivation, support for bringing up secondary interfaces after a primary goes up, a dbus interface for automation, etc.
I just prepared myself to catch up and learn more about NetworkManager. So I opened my big fat "Unix and Linux System Administration Handbook 5th edition", with a text file open on the computer to take extensive notes...
... only to find out that there is only half a page on NetworkManager in this book. Allow me to quote it:
"NetworkManager is primarily of use on laptops, since their network enviromment may change frequently. For servers and desktop systems, NetworkManager isn't necessary and may in fact complicate administration. In these environments, it should be ignored or configured out."
The book was published in 2017 which means it was written in late 2016. As much as I love that series of books (I have read them from 1st edition), I do not expect that its comments on parts of Linux in the 3rd edition would be useful now.
In the end, the problem is that NetworkManager, FirewallD, and other 'automatic' helpers are 'part' of the OS.. and while it was easy to tear them out in earlier versions.. as time goes on it is not.
I like the way you called the fact that these "automatic" things are part of OS: the PROBLEM (in case of servers).
Every time I see these discussions on Linux lists, I tell myself how happy I am after fleeing servers to different OS (huh, I'll break my plea to not mention it: FreeBSD).
Valeri
For a car analogy, it was much easier to convert any 1970 car from automatic back to manual as many parts were left over. Now in this era, you can do so if you pick the right car but for a lot of them it is not going to be easy in any form. I see the same trends in computer OS's with certain tools which were easy to pull out now requiring you to build the whole os from scratch as the part is assumed to be in so many other areas.
Le 13/02/2020 à 17:50, Stephen John Smoogen a écrit :
In the end, the problem is that NetworkManager, FirewallD, and other 'automatic' helpers are 'part' of the OS.. and while it was easy to tear them out in earlier versions.. as time goes on it is not.
For a car analogy, it was much easier to convert any 1970 car from automatic back to manual as many parts were left over. Now in this era, you can do so if you pick the right car but for a lot of them it is not going to be easy in any form. I see the same trends in computer OS's with certain tools which were easy to pull out now requiring you to build the whole os from scratch as the part is assumed to be in so many other areas.
I'm currently in the process of making peace with NetworkManager, FirewallD, etc. and adopting them on my servers.
Here's a start :
* https://www.microlinux.fr/networkmanager-centos-rhel-1/
Cheers,
Niki
hi,
On Mon, Feb 10, 2020 at 8:55 AM Nicolas Kovacs info@microlinux.fr wrote:
Hi,
.... On servers though, one of the first post-installation steps I performed was to get rid of Network-Manager and all its components. The servers I'm working on are relatively small-scale and have from one to four network interfaces. Each interface has a corresponding configuration in /etc/sysconfig/network-scripts, and that's it. From there, I rarely - if ever - touch it. In all my setups, NetworkManager is merely a useless layer of abstraction, and I like sticking to the KISS principle and shave off useless layers.
Interesting philosophical discussion but using centos means you need to go with whatever red hat decides, so if they say so, then you have few options.
I must admit I have long refused to use networkmanager, but since centos 7 it has been rock solid. And as we use config tools (salt right now, but it is the same with the rest of the competition) I do not really care what they use to abstract the network configuration as long as it works. And work it does, so everybody is happy.
Another huge selling point is that it is what cockpit uses to configure the network interfaces, and cockpit is really nice for less advanced users. So our more junior people can get their feet wet using cockpit, and we can automate everything using configuration management, and both tools use the same api so nobody gets left behind.
Tab completion makes it easy to use, too ;-)
In the end, my take is: whoever comes after me needs to understand whatever we were doing, so let's just sitck with what the vendor provides (regarding the operating system) and use best of breed tooling to manage it (which may or may not be what the OS vendor recommends, but can fit better the business's requirements).
-- regards from the sunny Netherlands, natxo
-
Anand Buddhdev -
Chris Adams -
Jonathan Billings -
Matthew Miller -
Mauricio Tavares -
Natxo Asenjo -
Nicolas Kovacs -
Simon Matter -
Stephen John Smoogen -
Valeri Galtsev