[CentOS] systemd-journald corruption

[Chris Adams]

Once upon a time, Chris Murphy <lists at colorremedies.com> said:
> On Tue, Apr 26, 2016, 2:09 PM Chris Adams <linux at cmadams.net> wrote:
> > I have several recently-installed CentOS 7 servers that keep having
> > systemd-journald corruption
> 
> Determined with 'journalctl --verify' or another way?

I get messages like this in dmesg:

[4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637 bytes), ignoring: Cannot assign requested address

I'll check journalctl --verify when it happens next (seems to happen
every day on at least one of the servers).  It does only seem to be
happening only on my spam-scanning VMs.  Some of them do have a
relatively high log rate (40-50 messages per second sometimes).

I dug into it a little more after my original message, and it appears to
be a recent issue; I have some VMs that were set up a little longer ago
(still running CentOS 7.1 I believe) that have not had this problem.

> If you are referring to native journald logs corrupting, that should not
> affect rsyslog. If you remove /var/log/journal then systemd-journald logs
> will be stored volatile in /run.

That appears to be where they're going (I don't have a /var/log/journal,
but I didn't do anything to remove it).  I have had to remove files from
/run/log/journal to get systemd-journald working again.

> > Has anyone else seen this?
> 
> Sortof, but not in a way that affects rsyslog. Usually journalctl just
> skips over corrupt parts and systemd-journald will rotate logs when it
> detects corruption to isolate corrupt files.

When it happens, all logs just stop; rsyslogd appears to not get any
more log entries (I have rsyslogd logging to central log hosts and they
get nothing).

-- 
Chris Adams <linux at cmadams.net>