From: SilverTip257 silvertip257@gmail.com
To: CentOS mailing list centos@centos.org Cc: Sent: Friday, March 21, 2014 5:40 PM Subject: Re: [CentOS] rsync triggers oomkiller
I added a subject so we can track this message on the list easier. ;)
On Fri, Mar 21, 2014 at 12:19 PM, John Doe jdmls@yahoo.com wrote:
Hey,
kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 ... kernel: Out of memory: Kill process 27974 (mysqld) score 361 or sacrifice child kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, anon-rss:2890828kB, file-rss:3324kB
rsync whines he wants more RAM and... mysql gets killed... That makes me a bit sad! And from my nagios graphs, at that time, ram usage was only 75% (4.5GB/6GB) and swap usage 0.3% (/2GB)! I cannot believe an rsync suddenly needed 1.5GB+2GB and was not satisfied... And even after the out of memory, the graphs show no swap usage change at all. Anybody would have an idea about this "fake" out of memory?
Wild. I've not encountered oomkiller being triggered when the server has free memory and hasn't swapped to disk yet.
After the fact, it will probably be almost impossible to figure out the list of files that rsync was storing in memory.
This makes me wonder if there's an option to have rsync log the list of files (to be synced) to a log file of some sort. I'll have to look into it later today when I have a moment.
Thx for the subject! After more investigation, I found: - a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel to swap to prevent an oom. - the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing between 500 and 1000 files at most...
Thx, JD