I added a subject so we can track this message on the list easier. ;)
On Fri, Mar 21, 2014 at 12:19 PM, John Doe jdmls@yahoo.com wrote:
Hey,
kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 ... kernel: Out of memory: Kill process 27974 (mysqld) score 361 or sacrifice child kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, anon-rss:2890828kB, file-rss:3324kB
rsync whines he wants more RAM and... mysql gets killed... That makes me a bit sad! And from my nagios graphs, at that time, ram usage was only 75% (4.5GB/6GB) and swap usage 0.3% (/2GB)! I cannot believe an rsync suddenly needed 1.5GB+2GB and was not satisfied... And even after the out of memory, the graphs show no swap usage change at all. Anybody would have an idea about this "fake" out of memory?
Wild. I've not encountered oomkiller being triggered when the server has free memory and hasn't swapped to disk yet.
After the fact, it will probably be almost impossible to figure out the list of files that rsync was storing in memory.
This makes me wonder if there's an option to have rsync log the list of files (to be synced) to a log file of some sort. I'll have to look into it later today when I have a moment.
Thx, JD _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
From: SilverTip257 silvertip257@gmail.com
To: CentOS mailing list centos@centos.org Cc: Sent: Friday, March 21, 2014 5:40 PM Subject: Re: [CentOS] rsync triggers oomkiller
I added a subject so we can track this message on the list easier. ;)
On Fri, Mar 21, 2014 at 12:19 PM, John Doe jdmls@yahoo.com wrote:
Hey,
kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 ... kernel: Out of memory: Kill process 27974 (mysqld) score 361 or sacrifice child kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, anon-rss:2890828kB, file-rss:3324kB
rsync whines he wants more RAM and... mysql gets killed... That makes me a bit sad! And from my nagios graphs, at that time, ram usage was only 75% (4.5GB/6GB) and swap usage 0.3% (/2GB)! I cannot believe an rsync suddenly needed 1.5GB+2GB and was not satisfied... And even after the out of memory, the graphs show no swap usage change at all. Anybody would have an idea about this "fake" out of memory?
Wild. I've not encountered oomkiller being triggered when the server has free memory and hasn't swapped to disk yet.
After the fact, it will probably be almost impossible to figure out the list of files that rsync was storing in memory.
This makes me wonder if there's an option to have rsync log the list of files (to be synced) to a log file of some sort. I'll have to look into it later today when I have a moment.
Thx for the subject! After more investigation, I found: - a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel to swap to prevent an oom. - the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing between 500 and 1000 files at most...
Thx, JD
On Fri, Mar 21, 2014 at 11:49 AM, John Doe jdmls@yahoo.com wrote:
kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 ... kernel: Out of memory: Kill process 27974 (mysqld) score 361 or sacrifice child kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, anon-rss:2890828kB, file-rss:3324kB
rsync whines he wants more RAM and... mysql gets killed... That makes me a bit sad!
After more investigation, I found:
- a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel to swap to prevent an oom.
- the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing between 500 and 1000 files at most...
Before very recent versions of rsync (not sure exactly when it changed), it would load the entire tree listing from both sides into memory before walking them for the comparison. What's the destination side look like? Maybe you aren't doing a --delete and a lot of cruft has accumulated.
On Fri, Mar 21, 2014 at 1:16 PM, Les Mikesell lesmikesell@gmail.com wrote:
On Fri, Mar 21, 2014 at 11:49 AM, John Doe jdmls@yahoo.com wrote:
kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0,
oom_score_adj=0 ... kernel: Out of memory: Kill process 27974 (mysqld) score 361 or sacrifice child kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB, anon-rss:2890828kB, file-rss:3324kB
rsync whines he wants more RAM and... mysql gets killed... That makes me a bit sad!
After more investigation, I found:
- a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel
to swap to prevent an oom.
- the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing
between 500 and 1000 files at most...
In one of the rsync backup cronjobs I wrote for work, I added logic to check for running rsync processes so they don't run "on top" of each other (that's to prevent simultaneous syncs and not sequential as you noted in your scenario).
Before very recent versions of rsync (not sure exactly when it changed), it would load the entire tree listing from both sides into memory before walking them for the comparison. What's the destination side look like? Maybe you aren't doing a --delete and a lot of cruft has accumulated.
I avoid this problem by having the cronjob run more often and the script checks that syncs are running (which means on average it should sync less files and re-sync as soon as the previous sync is finished).
-- Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, Mar 21, 2014 at 5:31 PM, SilverTip257 silvertip257@gmail.com wrote:
Before very recent versions of rsync (not sure exactly when it changed), it would load the entire tree listing from both sides into memory before walking them for the comparison. What's the destination side look like? Maybe you aren't doing a --delete and a lot of cruft has accumulated.
I avoid this problem by having the cronjob run more often and the script checks that syncs are running (which means on average it should sync less files and re-sync as soon as the previous sync is finished).
The number of files being sync'd isn't the big memory consumer - it is the whole directory trees being traversed that are loaded into memory for the comparison. There is additional overhead it you use -H to propagate hardlinks.
-
John Doe -
Les Mikesell -
SilverTip257