On Fri, Mar 21, 2014 at 1:16 PM, Les Mikesell lesmikesell@gmail.com wrote:
On Fri, Mar 21, 2014 at 11:49 AM, John Doe jdmls@yahoo.com wrote:
kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0,
oom_score_adj=0 ... kernel: Out of memory: Kill process 27974 (mysqld) score 361 or sacrifice child kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, anon-rss:2890828kB, file-rss:3324kB
rsync whines he wants more RAM and... mysql gets killed... That makes me a bit sad!
After more investigation, I found:
- a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel
to swap to prevent an oom.
- the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing
between 500 and 1000 files at most...
In one of the rsync backup cronjobs I wrote for work, I added logic to check for running rsync processes so they don't run "on top" of each other (that's to prevent simultaneous syncs and not sequential as you noted in your scenario).
Before very recent versions of rsync (not sure exactly when it changed), it would load the entire tree listing from both sides into memory before walking them for the comparison. What's the destination side look like? Maybe you aren't doing a --delete and a lot of cruft has accumulated.
I avoid this problem by having the cronjob run more often and the script checks that syncs are running (which means on average it should sync less files and re-sync as soon as the previous sync is finished).
-- Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos