[CentOS] rsync triggers oomkiller

[SilverTip257]

On Fri, Mar 21, 2014 at 1:16 PM, Les Mikesell <lesmikesell at gmail.com> wrote:

> On Fri, Mar 21, 2014 at 11:49 AM, John Doe <jdmls at yahoo.com> wrote:
> >>>>
> >>>    kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0,
> oom_adj=0,
> >>>  oom_score_adj=0
> >>>    ...
> >>>    kernel: Out of memory: Kill process 27974 (mysqld) score 361 or
> >>>  sacrifice child
> >>>    kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB,
> >>>  anon-rss:2890828kB, file-rss:3324kB
> >>>
> >>>  rsync whines he wants more RAM and... mysql gets killed...
> >>>  That makes me a bit sad!
>
> > After more investigation, I found:
> > - a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel
> to swap to prevent an oom.
> > - the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing
> between 500 and 1000 files at most...
> >
>

In one of the rsync backup cronjobs I wrote for work, I added logic to
check for running rsync processes so they don't run "on top" of each other
(that's to prevent simultaneous syncs and not sequential as you noted in
your scenario).


>
> Before very recent versions of rsync  (not sure exactly when it
> changed), it would load the entire tree listing from both sides into
> memory before walking them for the comparison.  What's the destination
> side look like?  Maybe you aren't doing a --delete and a lot of cruft
> has accumulated.
>

I avoid this problem by having the cronjob run more often and the script
checks that syncs are running (which means on average it should sync less
files and re-sync as soon as the previous sync is finished).


>
> --
>   Les Mikesell
>       lesmikesell at gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
---~~.~~---
Mike
//  SilverTip257  //