I have multiple servers running stock CentOS 7 rsyslog 7.4.7-16.el7, which are configured to log locally and over TCP to a remote logserver, also running stock CentOS 7 rsyslog. The remote server uses imptcp to receive, and pretty basic rules to parse and commit to disk.
I have several systems that log prolifically, but periodically, they stop soon after the remote log server HUPs (daily logrotate). Very soon after they stop logging (completely, even to local files), the services on these systems block, and our monitoring system starts alerting. Restarting rsyslog on the clients proves ineffectual.
The situation may clear itself without intervention after 90 minutes to several hours.
However, this does not happen on all client systems in a similar situation (CentOS 7, large volume of constant log data); nor does it happen daily.
Any ideas as to what's going on?
Thanks in advance.