I have multiple servers running stock CentOS 7 rsyslog 7.4.7-16.el7,
which are configured to log locally and over TCP to a remote logserver,
also running stock CentOS 7 rsyslog. The remote server uses imptcp to
receive, and pretty basic rules to parse and commit to disk.
I have several systems that log prolifically, but periodically, they
stop soon after the remote log server HUPs (daily logrotate). Very soon
after they stop logging (completely, even to local files), the services
on these systems block, and our monitoring system starts alerting.
Restarting rsyslog on the clients proves ineffectual.
The situation may clear itself without intervention after 90 minutes to
several hours.
However, this does not happen on all client systems in a similar
situation (CentOS 7, large volume of constant log data); nor does it
happen daily.
Any ideas as to what's going on?
Thanks in advance.