From: SilverTip257 <silvertip257 at gmail.com> > To: CentOS mailing list <centos at centos.org> > Cc: > Sent: Friday, March 21, 2014 5:40 PM > Subject: Re: [CentOS] rsync triggers oomkiller > > I added a subject so we can track this message on the list easier. ;) > > On Fri, Mar 21, 2014 at 12:19 PM, John Doe <jdmls at yahoo.com> wrote: > >> Hey, >> >> kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, >> oom_score_adj=0 >> ... >> kernel: Out of memory: Kill process 27974 (mysqld) score 361 or >> sacrifice child >> kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, >> anon-rss:2890828kB, file-rss:3324kB >> >> rsync whines he wants more RAM and... mysql gets killed... >> That makes me a bit sad! >> And from my nagios graphs, at that time, ram usage was only 75% >> (4.5GB/6GB) and swap usage 0.3% (/2GB)! >> I cannot believe an rsync suddenly needed 1.5GB+2GB and was not >> satisfied... >> And even after the out of memory, the graphs show no swap usage change at >> all. >> Anybody would have an idea about this "fake" out of memory? >> > > Wild. > I've not encountered oomkiller being triggered when the server has free > memory and hasn't swapped to disk yet. > > After the fact, it will probably be almost impossible to figure out the > list of files that rsync was storing in memory. > > This makes me wonder if there's an option to have rsync log the list of > files (to be synced) to a log file of some sort. I'll have to look into it > later today when I have a moment. Thx for the subject! After more investigation, I found: - a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel to swap to prevent an oom. - the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing between 500 and 1000 files at most... Thx, JD