I have several recently-installed CentOS 7 servers that keep having systemd-journald corruption (which stops ALL logging, including syslog). Interestingly, they are all spam-scanning servers running amavisd-new (so could be some particular pattern is triggering it).
Is there a "supported" way to just cut systemd-journald out of the picture and have log entries go straight to rsyslogd?
Has anyone else seen this?
On Tue, Apr 26, 2016, 2:09 PM Chris Adams linux@cmadams.net wrote:
I have several recently-installed CentOS 7 servers that keep having systemd-journald corruption
Determined with 'journalctl --verify' or another way?
(which stops ALL logging, including syslog).
Interestingly, they are all spam-scanning servers running amavisd-new (so could be some particular pattern is triggering it).
Is there a "supported" way to just cut systemd-journald out of the picture and have log entries go straight to rsyslogd?
No. Everything reports to journald and rsyslog gets what it wants from journald.
If you are referring to native journald logs corrupting, that should not affect rsyslog. If you remove /var/log/journal then systemd-journald logs will be stored volatile in /run.
Has anyone else seen this?
Sortof, but not in a way that affects rsyslog. Usually journalctl just skips over corrupt parts and systemd-journald will rotate logs when it detects corruption to isolate corrupt files.
Chris Murphy
Once upon a time, Chris Murphy lists@colorremedies.com said:
On Tue, Apr 26, 2016, 2:09 PM Chris Adams linux@cmadams.net wrote:
I have several recently-installed CentOS 7 servers that keep having systemd-journald corruption
Determined with 'journalctl --verify' or another way?
I get messages like this in dmesg:
[4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637 bytes), ignoring: Cannot assign requested address
I'll check journalctl --verify when it happens next (seems to happen every day on at least one of the servers). It does only seem to be happening only on my spam-scanning VMs. Some of them do have a relatively high log rate (40-50 messages per second sometimes).
I dug into it a little more after my original message, and it appears to be a recent issue; I have some VMs that were set up a little longer ago (still running CentOS 7.1 I believe) that have not had this problem.
If you are referring to native journald logs corrupting, that should not affect rsyslog. If you remove /var/log/journal then systemd-journald logs will be stored volatile in /run.
That appears to be where they're going (I don't have a /var/log/journal, but I didn't do anything to remove it). I have had to remove files from /run/log/journal to get systemd-journald working again.
Has anyone else seen this?
Sortof, but not in a way that affects rsyslog. Usually journalctl just skips over corrupt parts and systemd-journald will rotate logs when it detects corruption to isolate corrupt files.
When it happens, all logs just stop; rsyslogd appears to not get any more log entries (I have rsyslogd logging to central log hosts and they get nothing).
On Tue, Apr 26, 2016 at 3:01 PM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Murphy lists@colorremedies.com said:
On Tue, Apr 26, 2016, 2:09 PM Chris Adams linux@cmadams.net wrote:
I have several recently-installed CentOS 7 servers that keep having systemd-journald corruption
Determined with 'journalctl --verify' or another way?
I get messages like this in dmesg:
[4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637 bytes), ignoring: Cannot assign requested address
I haven't seen this. When I plug this text into a google search field, no quotes, there are 360 results.
systemd-journald failed to write entry cannot assign requested address
There's also this patch as a suggested fix: https://bugzilla.redhat.com/show_bug.cgi?id=1292447#c9
What version of systemd and rsyslog? systemd-219-19.el7_2.7 and rsyslog-7.4.7-12 are current.
If you're there already you could ry editing /etc/systemd/journald.conf and uncommenting Compress=yes and changing it to no.
Once upon a time, Chris Murphy lists@colorremedies.com said:
On Tue, Apr 26, 2016 at 3:01 PM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Murphy lists@colorremedies.com said:
On Tue, Apr 26, 2016, 2:09 PM Chris Adams linux@cmadams.net wrote:
I have several recently-installed CentOS 7 servers that keep having systemd-journald corruption
Determined with 'journalctl --verify' or another way?
One system did get into this state overnight, and that said:
[root@spamscan3 ~]# journalctl --verify 15bd478: invalid object File corruption detected at /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal:15bd478 (of 25165824 bytes, 90%). FAIL: /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal (Cannot assign requested address) (and then a bunch of passes on the rest of the files)
There's also this patch as a suggested fix: https://bugzilla.redhat.com/show_bug.cgi?id=1292447#c9
I'll take a look at that.
What version of systemd and rsyslog? systemd-219-19.el7_2.7 and rsyslog-7.4.7-12 are current.
Those are the versions I have.
If you're there already you could ry editing /etc/systemd/journald.conf and uncommenting Compress=yes and changing it to no.
Thanks, I'm trying that on these servers.
On Wed, Apr 27, 2016 at 7:05 AM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Murphy lists@colorremedies.com said:
On Tue, Apr 26, 2016 at 3:01 PM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Murphy lists@colorremedies.com said:
On Tue, Apr 26, 2016, 2:09 PM Chris Adams linux@cmadams.net wrote:
I have several recently-installed CentOS 7 servers that keep having systemd-journald corruption
Determined with 'journalctl --verify' or another way?
One system did get into this state overnight, and that said:
[root@spamscan3 ~]# journalctl --verify 15bd478: invalid object File corruption detected at /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal:15bd478 (of 25165824 bytes, 90%). FAIL: /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal (Cannot assign requested address) (and then a bunch of passes on the rest of the files)
There's also this patch as a suggested fix: https://bugzilla.redhat.com/show_bug.cgi?id=1292447#c9
I'll take a look at that.
What version of systemd and rsyslog? systemd-219-19.el7_2.7 and rsyslog-7.4.7-12 are current.
Those are the versions I have.
If you're there already you could ry editing /etc/systemd/journald.conf and uncommenting Compress=yes and changing it to no.
Thanks, I'm trying that on these servers.
Also I wonder if merely restarting the journal daemon solves it:
systemctl restart systemd-journald
What should happen is it realizes its own logs are corrupt and ignores them, and starts working on new copies. And journalctl should still try to read the old ones but skips the corrupt entries.
If that works you could schedule a restart of the journal periodically as a goofy hack work around until it gets fixed. Clearly Red Hat knows about this problem.
Once upon a time, Chris Murphy lists@colorremedies.com said:
Also I wonder if merely restarting the journal daemon solves it:
systemctl restart systemd-journald
What should happen is it realizes its own logs are corrupt and ignores them, and starts working on new copies. And journalctl should still try to read the old ones but skips the corrupt entries.
I tried that and it didn't work (at least not every time). I had to actually remove the journal file to get it functional again.
So far, turning off the compression appears to have worked (but I'll have to watch it for a day or two to really see).
Once upon a time, Chris Adams linux@cmadams.net said:
So far, turning off the compression appears to have worked (but I'll have to watch it for a day or two to really see).
Just to follow up: turning off journald compression does appear to have fixed the corruption problem I was seeing. I'll watch for an updated systemd package.
On Fri, Apr 29, 2016 at 6:26 AM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Adams linux@cmadams.net said:
So far, turning off the compression appears to have worked (but I'll have to watch it for a day or two to really see).
Just to follow up: turning off journald compression does appear to have fixed the corruption problem I was seeing. I'll watch for an updated systemd package.
Does journalctl --verify no longer complain of corruption? Or is it possible there's corruption but journalctl and rsyslogd now tolerate it somehow?
-
Chris Adams -
Chris Murphy