Hi,
I noticed that some, but not all, of my CentOS 7 machines have these kernel parameters for turning on systemd debug level logging added to the grub.cfg file.
systemd.log_level=debug systemd.log_target=kmsg
The parameters are *not* in the /etc/default/grub file, so if I rebuild grub.cfg with grub2-mkconfig, they disappear.
I am a bit puzzled over how they got introduced. Has anyone else seen this?
thanks,
Thomas
Oh wow, I was coming across this in my env last week and thought that it was something with the way they were kickstarted. You are not alone!
On Feb 27, 2017, 5:35 PM -0500, Thomas Eriksson thomas.eriksson@slac.stanford.edu, wrote:
Hi,
I noticed that some, but not all, of my CentOS 7 machines have these kernel parameters for turning on systemd debug level logging added to the grub.cfg file.
systemd.log_level=debug systemd.log_target=kmsg
The parameters are *not* in the /etc/default/grub file, so if I rebuild grub.cfg with grub2-mkconfig, they disappear.
I am a bit puzzled over how they got introduced. Has anyone else seen this?
thanks,
Thomas
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Once upon a time, Thomas Eriksson thomas.eriksson@slac.stanford.edu said:
I noticed that some, but not all, of my CentOS 7 machines have these kernel parameters for turning on systemd debug level logging added to the grub.cfg file.
Yep, for each of the installed kernels, I have two GRUB entries: one with and one without debugging. It seems that when I install a new kernel, the debugging entry gets the "default" choice.
I don't know what's adding them though.
Last time I saw it, I had just upgraded my CentOS 7 box with the 3.10.0-514 kernel and it rebooted already configured into debug mode. Not sure if this is a “feature” of the newer kernels or not but glad to see that i’m not the only one who had noticed this.
# awk -F' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with debugging 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) with debugging
On Feb 27, 2017, 8:40 PM -0500, Chris Adams linux@cmadams.net, wrote:
Once upon a time, Thomas Eriksson thomas.eriksson@slac.stanford.edu said:
I noticed that some, but not all, of my CentOS 7 machines have these kernel parameters for turning on systemd debug level logging added to the grub.cfg file.
Yep, for each of the installed kernels, I have two GRUB entries: one with and one without debugging. It seems that when I install a new kernel, the debugging entry gets the "default" choice.
I don't know what's adding them though.
Chris Adams <linux@cmadams.net _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Tue, February 28, 2017 9:22 am, Rob DeSanno wrote:
Last time I saw it, I had just upgraded my CentOS 7 box with the 3.10.0-514 kernel and it rebooted already configured into debug mode. Not sure if this is a âfeatureâ of the newer kernels or not but glad to see that iâm not the only one who had noticed this.
# awk -F' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with debugging 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) with debugging
I am not certain if there is real harm to have kernel with all debug stuff running on production machines. Probably no harm security wise, the only unpleasant stuff is: you really would prefer to run as slim kernel as possible on production systems. If I'm wrong about "no harm", somebody chime in, I then will be really eager to address it on my boxes.
Valeri
On Feb 27, 2017, 8:40 PM -0500, Chris Adams linux@cmadams.net, wrote:
Once upon a time, Thomas Eriksson thomas.eriksson@slac.stanford.edu said:
I noticed that some, but not all, of my CentOS 7 machines have these kernel parameters for turning on systemd debug level logging added to the grub.cfg file.
Yep, for each of the installed kernels, I have two GRUB entries: one with and one without debugging. It seems that when I install a new kernel, the debugging entry gets the "default" choice.
I don't know what's adding them though.
Chris Adams <linux@cmadams.net _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
The reason why I noticed was that telegraf was reporting context related issues even with selinux disabled. That’s when I noticed all of the debugging lines in dmesg as well as debug mode enabled in /proc/cmdline.
On Feb 28, 2017, 10:49 AM -0500, Valeri Galtsev galtsev@kicp.uchicago.edu, wrote:
On Tue, February 28, 2017 9:22 am, Rob DeSanno wrote:
Last time I saw it, I had just upgraded my CentOS 7 box with the 3.10.0-514 kernel and it rebooted already configured into debug mode. Not sure if this is a âfeatureâ of the newer kernels or not but glad to see that iâm not the only one who had noticed this.
# awk -F' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with debugging 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) with debugging
I am not certain if there is real harm to have kernel with all debug stuff running on production machines. Probably no harm security wise, the only unpleasant stuff is: you really would prefer to run as slim kernel as possible on production systems. If I'm wrong about "no harm", somebody chime in, I then will be really eager to address it on my boxes.
Valeri
On Feb 27, 2017, 8:40 PM -0500, Chris Adams linux@cmadams.net, wrote:
Once upon a time, Thomas Eriksson <thomas.eriksson@slac.stanford.edu said:
I noticed that some, but not all, of my CentOS 7 machines have these kernel parameters for turning on systemd debug level logging added to the grub.cfg file.
Yep, for each of the installed kernels, I have two GRUB entries: one with and one without debugging. It seems that when I install a new kernel, the debugging entry gets the "default" choice.
I don't know what's adding them though.
Chris Adams <linux@cmadams.net _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Tue, Feb 28, 2017 at 10:49 AM, Valeri Galtsev galtsev@kicp.uchicago.edu wrote:
On Tue, February 28, 2017 9:22 am, Rob DeSanno wrote:
Last time I saw it, I had just upgraded my CentOS 7 box with the 3.10.0-514 kernel and it rebooted already configured into debug mode.
Not
sure if this is a “feature†of the newer kernels or not but glad to see that i’m not the only one who had noticed this.
# awk -F' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with debugging 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core)
with
debugging
I am not certain if there is real harm to have kernel with all debug stuff running on production machines. Probably no harm security wise, the only unpleasant stuff is: you really would prefer to run as slim kernel as possible on production systems. If I'm wrong about "no harm", somebody chime in, I then will be really eager to address it on my boxes.
Valeri
Main issue I've seen is that logs grow by an order of magnitude larger than when it's off, due to systemd being systemd and now running in debug mode. Other than disk space, it would affect any central logging system you have with lots of unnecessary traffic, and would also add a lot of IO, amplified if you have many machines running on a VM host.
~ Brian Mathis @orev
On 02/28/2017 08:55 AM, Brian Mathis wrote:
On Tue, Feb 28, 2017 at 10:49 AM, Valeri Galtsev galtsev@kicp.uchicago.edu wrote:
On Tue, February 28, 2017 9:22 am, Rob DeSanno wrote:
Last time I saw it, I had just upgraded my CentOS 7 box with the 3.10.0-514 kernel and it rebooted already configured into debug mode.
Not
sure if this is a “feature†of the newer kernels or not but glad to see that i’m not the only one who had noticed this.
# awk -F' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with debugging 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core) 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core)
with
debugging
I am not certain if there is real harm to have kernel with all debug stuff running on production machines. Probably no harm security wise, the only unpleasant stuff is: you really would prefer to run as slim kernel as possible on production systems. If I'm wrong about "no harm", somebody chime in, I then will be really eager to address it on my boxes.
Valeri
Main issue I've seen is that logs grow by an order of magnitude larger than when it's off, due to systemd being systemd and now running in debug mode. Other than disk space, it would affect any central logging system you have with lots of unnecessary traffic, and would also add a lot of IO, amplified if you have many machines running on a VM host.
~ Brian Mathis @orev
Just to put the record straight; it's not related to kernel debugging being enabled or not. It's systemd debugging that is being turned on for all menu entries, kernel debug or not.
Anyway, I think I have found a pattern. Only those machines that were updated from 7.2 to 7.3 using the CR repo are showing this behaviour. New 7.3 installs are fine. I'll just clean up the machines affected and move on. There must have been some debug config left in an installation script in one of the CR rpms.
thanks to everyone responding,
Thomas
On Tue, Feb 28, 2017 at 1:44 PM, Thomas Eriksson < thomas.eriksson@slac.stanford.edu> wrote:
On 02/28/2017 08:55 AM, Brian Mathis wrote:
Main issue I've seen is that logs grow by an order of magnitude larger
than
when it's off, due to systemd being systemd and now running in debug
mode.
Other than disk space, it would affect any central logging system you
have
with lots of unnecessary traffic, and would also add a lot of IO,
amplified
if you have many machines running on a VM host.
~ Brian Mathis @orev
Just to put the record straight; it's not related to kernel debugging being enabled or not. It's systemd debugging that is being turned on for all menu entries, kernel debug or not.
Anyway, I think I have found a pattern. Only those machines that were updated from 7.2 to 7.3 using the CR repo are showing this behaviour. New 7.3 installs are fine. I'll just clean up the machines affected and move on. There must have been some debug config left in an installation script in one of the CR rpms.
thanks to everyone responding,
Thomas
Yes, true, this isn't technically the "kernel" debug mode, but systemd debugging is also enabled for the debug boot options, which seems to get carried into a non-debug boot entry somehow.
I have seen this issue on a few machines, and I don't use the CR repos. The ones I saw were during upgrades from 7.2 systems with all updates, which were then upgraded to 7.3 with all updates as of 2 weeks ago.
~ Brian Mathis @orev
-
Brian Mathis -
Chris Adams -
Rob DeSanno -
Thomas Eriksson -
Valeri Galtsev