On Sun, 2008-02-24 at 17:53 -0600, Dan Carl wrote:
----- Original Message ----- From: "William L. Maltby" CentOS4Bill@triad.rr.com
<snip list header and now irrelevant stuff>
Don't be afraid to seek/request some kind of raid/NAS/SAN resource if the data is mission-critical, growing constantly and volatile. It may not be needed now, but look down the road so you don't get into a constant cycle of scrambling to keep up with needs.
Have a LVM/raid already. Just want to have an automated offsite backup to be on the safe side. Have learned the hard way raid is not a subsitute for a backup.
<snip>
In conjunction with "lock" files mentioned in another reply, you may be able to gain something by segmenting the local and remote rsync. This allows 1) concurrent *local* compression and rsync (if CPU/memory resources are sufficient to avoid unduly slowing the user's activities - again "man nice" to reduce the effects on users) and 2) easier management of the remote rsync start/stop on directory boundaries as the window is entered/exited. This may not be needed at all or may be of limited benefit.
Lock file sound like the way I'll go. I'm going to stick with the hand-crafted stuff
Lastly, see if it's possible to run the rsync during normal hours. If your site has upload of 750KB/sec and during 90% of the normal workday only a small percentage is consumed, take advantage by doing some of the rsync (maybe in small chunks) during these hours at low priority and throttled appropriately. Presuming that most of your activity is download, not upload during the normal workday, and knowing that most of the rsync activity will be upload, not download, there is an opportunity there.
Something I'll look at down the road.
-- Bill
Thanks Bill Lots of good imformation and thanks to all Dan