Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
- aurf
On 1/21/2011 2:30 PM, aurfalien@gmail.com wrote:
Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
How long does the rsync take to complete when you run it manually?
How often does cron run the command?
On Jan 21, 2011, at 11:46 AM, Bowie Bailey wrote:
On 1/21/2011 2:30 PM, aurfalien@gmail.com wrote:
Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
How long does the rsync take to complete when you run it manually?
About 1 hour.
How often does cron run the command?
Every night at 10PM.
- aurf
On 1/21/2011 1:30 PM, aurfalien@gmail.com wrote:
Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
It sounds like it is not completing one run before the next one starts. If you have a lot of hardlinks, the -H option can make things slower than you would expect.
On Jan 21, 2011, at 11:54 AM, Les Mikesell wrote:
On 1/21/2011 1:30 PM, aurfalien@gmail.com wrote:
Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
It sounds like it is not completing one run before the next one starts. If you have a lot of hardlinks, the -H option can make things slower than you would expect.
No hard links, some sym links.
But I see what you are saying.
Here is my crontab entry via /etc/crontab
* 22 * * * root rsync --delete -avvH --progress source target
- aurf
aurfalien@gmail.com wrote:
No hard links, some sym links.
But I see what you are saying.
Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
- aurf
So you want rsync to run every minute in the 10 PM hour? I think that first "*" needs to be replaced with a number designating the minute within the hour during which you want it to start. What you have there would kick off separate jobs at 22:00, 22:01, 22:02, etc.
On Jan 21, 2011, at 12:08 PM, Jay Leafey wrote:
aurfalien@gmail.com wrote:
No hard links, some sym links. But I see what you are saying. Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
- aurf
So you want rsync to run every minute in the 10 PM hour? I think that first "*" needs to be replaced with a number designating the minute within the hour during which you want it to start. What you have there would kick off separate jobs at 22:00, 22:01, 22:02, etc.
I think I been staring at the screen too long.
Thanks for that catch, my bad.
I am a lamerz.
- aurf
On Fri, Jan 21, 2011 at 3:14 PM, aurfalien@gmail.com wrote:
On Jan 21, 2011, at 12:08 PM, Jay Leafey wrote:
aurfalien@gmail.com wrote:
No hard links, some sym links. But I see what you are saying. Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
- aurf
So you want rsync to run every minute in the 10 PM hour? I think that first "*" needs to be replaced with a number designating the minute within the hour during which you want it to start. What you have there would kick off separate jobs at 22:00, 22:01, 22:02, etc.
I think I been staring at the screen too long.
Thanks for that catch, my bad.
I am a lamerz.
May I say "no"? You looked for help when you needed it, you provided the relevant information, and you provided more data when asked to solve the issue. And you understood the answer.
I've *met* lamerz. You are merely a person who made a mistake. Glad you noticed and got it fixed.
On Sat, Jan 22, 2011 at 11:42 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
May I say "no"? You looked for help when you needed it, you provided the relevant information, and you provided more data when asked to solve the issue. And you understood the answer.
I've *met* lamerz. You are merely a person who made a mistake. Glad you noticed and got it fixed.
Here's my lamerz story from a couple nights ago...
Backing up my xen/kvm machines that I do every month or two... Copied a xen guest from one system to another and booted to make sure it was working... All looked good. Then did my normal rsync between the two. The connection kept on dropping. On one system, ls was timing out. My immediate thought was bad hard drive. Took an hour to determine that I'd rebooted two identical xen guests with the same IP, same MAC, and they were stepping on each other.
I'm very grateful for this list and for the patience that the contributors exhibit.
On Jan 21, 2011, at 12:05 PM, aurfalien@gmail.com wrote:
On Jan 21, 2011, at 11:54 AM, Les Mikesell wrote:
On 1/21/2011 1:30 PM, aurfalien@gmail.com wrote:
Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
It sounds like it is not completing one run before the next one starts. If you have a lot of hardlinks, the -H option can make things slower than you would expect.
No hard links, some sym links.
But I see what you are saying.
Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
- aurf
If that's your crontab, you do see that you are actually starting a separate copy of the command every minute in 10PM.
No wonder you're killing you machine, try
0 22 * * * .....
-- Don Krause Head Systems Geek, Waver of Deceased Chickens. Optivus Proton Therapy, Inc. P.O. Box 608 Loma Linda, California 92354 909.799.8327 Tel 909.799.8366 Fax dkrause@optivus.com www.optivus.com "This message represents the official view of the voices in my head."
On Jan 21, 2011, at 12:09 PM, Don Krause wrote:
On Jan 21, 2011, at 12:05 PM, aurfalien@gmail.com wrote:
On Jan 21, 2011, at 11:54 AM, Les Mikesell wrote:
On 1/21/2011 1:30 PM, aurfalien@gmail.com wrote:
Hi all,
I've been running rsync via cron for a while now and all is well.
However on one particular new 5.5 box, whenever its runs via crontab, the machine ends up with over 20 rsync processes and a load of ~14 and eventually the machine dies.
But when running manually, I see it spawn 3 processes with a load of 1.5.
My rsync command is simply;
rsync --delete -avvH --progress source target
Any thoughts?
It sounds like it is not completing one run before the next one starts. If you have a lot of hardlinks, the -H option can make things slower than you would expect.
No hard links, some sym links.
But I see what you are saying.
Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
- aurf
If that's your crontab, you do see that you are actually starting a separate copy of the command every minute in 10PM.
No wonder you're killing you machine, try
0 22 * * * .....
Yes my bad, a big miss.
Explains why I probably had 24 processes :)
Can you say doh!
On Fri, Jan 21, 2011 at 12:05:37PM -0800, aurfalien@gmail.com wrote:
Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
That will run your rsync at *every minute* of 10pm! Clearly not what you want. Try
0 22 * * * root rsync --delete -avvH --progress source target
That will run it at 10:00pm only.
--keith
On Jan 21, 2011, at 12:09 PM, Keith Keller wrote:
On Fri, Jan 21, 2011 at 12:05:37PM -0800, aurfalien@gmail.com wrote:
Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
That will run your rsync at *every minute* of 10pm! Clearly not what you want. Try
Actually I probably had 60! Would make more sense.
- aurf
On Fri, Jan 21, 2011 at 3:18 PM, aurfalien@gmail.com wrote:
On Jan 21, 2011, at 12:09 PM, Keith Keller wrote:
On Fri, Jan 21, 2011 at 12:05:37PM -0800, aurfalien@gmail.com wrote:
Here is my crontab entry via /etc/crontab
- 22 * * * root rsync --delete -avvH --progress source target
That will run your rsync at *every minute* of 10pm! Clearly not what you want. Try
Actually I probably had 60! Would make more sense.
- aurf
Nahh, you were crashing first.
If it ever approaches taking a day, or you worry about starting two at the same time, you might consider using rsnapshot to manage night/hourly/daily snapshots, and/or to provide locking to avoid running two at once.
-
aurfalien＠gmail.com -
Bowie Bailey -
Don Krause -
Jay Leafey -
Keith Keller -
Kwan Lowe -
Les Mikesell -
Nico Kadel-Garcia