hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
This is approximately 102G of data.
Thanks for any suggestions.
Jerry
Jerry Geis wrote:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
This is approximately 102G of data.
It's the number of files in the run that matters more than the amount of date. Rsync loads the entire directory listing into RAM before starting to copy so there is a certain amount of per-file overhead. It should help if you could break the run up, perhaps doing a few directories separately, then make another pass that excludes those directories.
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
IMHO, rsync is overkill here. I would:
mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
IMHO, rsync is overkill here. I would:
mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year
how about cp -a ?
On Wed, 2008-01-30 at 17:14 +0100, Nicolas Thierry-Mieg wrote:
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
IMHO, rsync is overkill here. I would:
mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year
how about cp -a ?
How about "find -newer" to just back up things that have been added or changed? A big space saver. *However*, this leaves things deleted since the previous backup(s) in the previous archives. A *good* or *bad* thing? Depends on what you want to achieve.
Rsync can handle that situation for you, and so is better if you don't want to keep deleted files around.
The cpio solution offers a lot, most beneficial here is the ability to bzip the archive (anticipating it won't be frequently used), saving a lot of space.
<snip sig stuff>
On Jan 30, 2008 8:26 AM, William L. Maltby CentOS4Bill@triad.rr.com wrote:
On Wed, 2008-01-30 at 17:14 +0100, Nicolas Thierry-Mieg wrote:
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
IMHO, rsync is overkill here. I would:
mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year
how about cp -a ?
How about "find -newer" to just back up things that have been added or changed? A big space saver. *However*, this leaves things deleted since the previous backup(s) in the previous archives. A *good* or *bad* thing? Depends on what you want to achieve.
Rsync can handle that situation for you, and so is better if you don't want to keep deleted files around.
The cpio solution offers a lot, most beneficial here is the ability to bzip the archive (anticipating it won't be frequently used), saving a lot of space.
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
I'd go with cpio if rsync causes problems. YMMV.
mhr
On Wed, Jan 30, 2008 at 1:36 PM, MHR mhullrich@gmail.com wrote:
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
I'd go with cpio if rsync causes problems. YMMV. mhr
I once knew a guy who bought a really cheap PC with an AMD CPU in it. Despite the fact that the power supply was underpowered, and everything else on the machine was just as cheap as possible, he blamed the AMD chip for all of the problems the PC had. To this day he refuses to buy AMD CPUs, "because they don't work right" -- despite the fact that millions of people use bzip2^H^H^H^H^H AMD chips every day without any problem at all.
On Jan 30, 2008 10:46 AM, Brian Mathis brian.mathis@gmail.com wrote:
I once knew a guy who bought a really cheap PC with an AMD CPU in it. Despite the fact that the power supply was underpowered, and everything else on the machine was just as cheap as possible, he blamed the AMD chip for all of the problems the PC had. To this day he refuses to buy AMD CPUs, "because they don't work right" -- despite the fact that millions of people use bzip2^H^H^H^H^H AMD chips every day without any problem at all.
Heh heh - cute.
I was going to respond in re the AMD issue when I re-read that last part.
I had this problem with several files over a period of about six months. I suppose it could have been something else (AMD CPU, bad disk, power spikes, yada, yada), but:
1) The problem never showed up anywhere else, under my P4, same or different disks, same power conditions as other processes, yada, yada, yada.
2) I never saw this problem with using zip/unzip/pkzip, which I still use.
3) I also never saw enough of a significant advantage of bzip2 over zip, but none of my files were all that large, either.
My impression was that the larger (and more binary) a file was, the better the results with bzip2.
Like I said, YMMV.
mhr
On Wed, January 30, 2008 1:36 pm, MHR wrote: <snip>
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
Why do you think that the corruption you experienced had something to do with bzip2? I have been using it on a regular basis for the last several years to compress files of all sizes (ranging from very small to several gigabytes) and have yet to experience any corruption whatsoever.
On Jan 30, 2008 10:51 AM, Marko A. Jennings markobiz@bluegargoyle.com wrote:
On Wed, January 30, 2008 1:36 pm, MHR wrote:
<snip> > As long as the majority of the files are not plain text - I have had > really bad results using bzip2 on text files - specifically, massive > file corruption. I have had to go back to pre-bzipped archives to > rebuild these files - not a fun task.
Why do you think that the corruption you experienced had something to do with bzip2? I have been using it on a regular basis for the last several years to compress files of all sizes (ranging from very small to several gigabytes) and have yet to experience any corruption whatsoever.
One of my hobbies is writing, a practice in which I have been engaged since the late 1980s. For personal reasons, until very recently, I did all of my writing in plain text files, all around 20-30k, and kept all my archives in pkzip, then zip/unzip format. From August through December, 1999, I was using bzip2 instead because it got slightly better compression. Some time in January, 2000, I found that some of the files I had not changed in a long time, and some that I had just edited, had become corrupted and I had to rebuild them.
Maybe bzip2 has improved since then, but my experience with it has been jaded ever since, and I'd rather go for reliability over a slight improvement in compression any day.
I may undertake an experiment and keep parallel bzip2 archives for a while, but now isn't a good time for it.
On the other hand, I've been using bzip2 for a few executables since that same time frame and, AFAIK, they work just fine, no corruption.
As I said, YMMV, and that's just my $0.02.
mhr
MHR wrote:
On Jan 30, 2008 8:26 AM, William L. Maltby CentOS4Bill@triad.rr.com wrote:
On Wed, 2008-01-30 at 17:14 +0100, Nicolas Thierry-Mieg wrote:
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
Isn't the kernel source stored as a tared bzip file? If so, that's a lot of plain text.
On Jan 30, 2008 11:03 AM, Milton Calnek milton@calnek.com wrote:
MHR wrote:
On Jan 30, 2008 8:26 AM, William L. Maltby CentOS4Bill@triad.rr.com wrote:
On Wed, 2008-01-30 at 17:14 +0100, Nicolas Thierry-Mieg wrote:
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
Isn't the kernel source stored as a tared bzip file? If so, that's a lot of plain text.
# file /boot/vmlinuz-2.6.18-53.1.6.el5 /boot/vmlinuz-2.6.18-53.1.6.el5: ELF 64-bit LSB shared object, AMD x86-64, version 1, stripped
Doesn't look that way - BUT, it is a self-extracting archive. AFAIK it is gzipped, and it is not tarred (why would it be - it's one file?).
mhr
MHR wrote:
On Jan 30, 2008 11:03 AM, Milton Calnek milton@calnek.com wrote:
MHR wrote:
On Jan 30, 2008 8:26 AM, William L. Maltby CentOS4Bill@triad.rr.com wrote:
On Wed, 2008-01-30 at 17:14 +0100, Nicolas Thierry-Mieg wrote:
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote: > hi all,
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
Isn't the kernel source stored as a tared bzip file? If so, that's a lot of plain text.
# file /boot/vmlinuz-2.6.18-53.1.6.el5 /boot/vmlinuz-2.6.18-53.1.6.el5: ELF 64-bit LSB shared object, AMD x86-64, version 1, stripped
Doesn't look that way - BUT, it is a self-extracting archive. AFAIK it is gzipped, and it is not tarred (why would it be - it's one file?).
Not that file... I meant linux-major.minor.blah.blah.tar.gz say from ftp://ftp.kernel.org/pub/linux/kernel/v2.6 And, my mistake... it's gzip'd.
On Jan 30, 2008 11:49 AM, Milton Calnek milton@calnek.com wrote:
MHR wrote:
# file /boot/vmlinuz-2.6.18-53.1.6.el5 /boot/vmlinuz-2.6.18-53.1.6.el5: ELF 64-bit LSB shared object, AMD x86-64, version 1, stripped
Doesn't look that way - BUT, it is a self-extracting archive. AFAIK it is gzipped, and it is not tarred (why would it be - it's one file?).
Not that file... I meant linux-major.minor.blah.blah.tar.gz say from ftp://ftp.kernel.org/pub/linux/kernel/v2.6 And, my mistake... it's gzip'd.
My mistake - I misread that (it's too late for me to claim still sleepy, so oops!).
And, well, yeah! (Kidding - ;^)
mhr
MHR wrote:
On Jan 30, 2008 8:26 AM, William L. Maltby CentOS4Bill@triad.rr.com wrote:
As long as the majority of the files are not plain text - I have had really bad results using bzip2 on text files - specifically, massive file corruption. I have had to go back to pre-bzipped archives to rebuild these files - not a fun task.
I've been using pigz for a while (Parallel gzip), to compress 100+GB tar files, it works well if you have multiple CPUs. Never encountered corruption with bzip2 myself, there is a parallel bzip but it's about 8x slower.
from my notes: -- To compile: gcc pigz17.c -lpthread -lz -o pigz
Sample command line: pigz -p 10 -v (filename)
The default 32 threads seems to be kind of high, drives load up quite a bit, while 10 threads at least in a simple test on a 2GB file kept load a lot lower but still kept the CPUs busy at 100% utilization on a dual core system. YMMV.
original source: http://zlib.net/pigz17.c.gz
if that doesn't exist there may be a new version, try pibz18.c.gz 19.c.gz ..etc ---
To be safe, since I deployed it a few months ago I've been running gzip -t afterwards on the files, and all of them have passed.
nate
Nicolas Thierry-Mieg wrote:
Marcelo Roccasalva wrote:
On Jan 30, 2008 11:24 AM, Jerry Geis geisj@pagestation.com wrote:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
IMHO, rsync is overkill here. I would:
mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year
how about cp -a ?
You may find that cp is significantly slower than cpio/tar.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Marcelo Roccasalva wrote:
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
IMHO, rsync is overkill here. I would:
mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year
Rsync will be much, much faster on the 2nd and subsequent runs when it only has to copy the changed files.
on 1/30/2008 5:24 AM Jerry Geis spake the following:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
This is approximately 102G of data.
Thanks for any suggestions.
Jerry
Rsync's main benefit is on backups of changed files. dumping to a new destination every time makes rsync less efficient than just about every other option. Now if you made the new directory, and hardlinked the old stuff to the new directory, then rsync would shine.
Yohoo!
Rsync's main benefit is on backups of changed files. dumping to a new destination every time makes rsync less efficient than just about every other option. Now if you made the new directory, and hardlinked the old stuff to the new directory, then rsync would shine.
That's what rsnapshot is designed for. It uses rsync to sync the files to a backup destination and hardlinks any existing file, so you can go back to any level you like.
Christian
Scott Silva wrote:
on 1/30/2008 5:24 AM Jerry Geis spake the following:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
This is approximately 102G of data.
Thanks for any suggestions.
Jerry
Rsync's main benefit is on backups of changed files. dumping to a new destination every time makes rsync less efficient than just about every other option. Now if you made the new directory, and hardlinked the old stuff to the new directory, then rsync would shine.
I did the rsync hard link for a while. After 30+ hardlinks to each file built up, filesystem operations slowed down - not in a killer way, but I did notice it. I think it's better to just use --backup and write the previous version to a new dir with --backup-dir=`date +%F` or some such scheme. You don't see the backups represented as a whole directory structure, but it's less messy.
Scott Silva wrote:
on 1/30/2008 5:24 AM Jerry Geis spake the following:
hi all,
I use rsync to copy/backup ALL my stuff to another disk.
When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness
I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year
This is approximately 102G of data.
Thanks for any suggestions.
Jerry
Rsync's main benefit is on backups of changed files. dumping to a new destination every time makes rsync less efficient than just about every other option. Now if you made the new directory, and hardlinked the old stuff to the new directory, then rsync would shine.
Yes. That's my experience, too. I do something like this twice weekly to 2 external USB drives. One drive contains backups from the most recent 2 Sunday mornings, the other drive contains two backups from Wednesday mornings. After mounting the backup drive, doing a space-available check and a couple other housekeeping chores, I do this. # (I've purposely left out a bunch of stuff to avoid being blamed when someone's system craters. )
$UD #-- Previously identified as the USB drive mount point # DT=`date +%F` #-- Today's date, for naming the backup directory: yyyy-mm-dd # # Shuffle directories so that oldest backup directory is renamed to today's date. # OD=`ls -1 $UD | head -1` #echo "Renaming $UD/$OD to $UD/$DT" mv $UD/$OD/ $UD/$DT touch $UD/$DT
# Now, data that is only 2 weeks old will be overwritten with current data.
for SD in <list of directories to back up> ; do rsync -ar --exclude '*.iso' --delete-excluded /$SD/* $UD/$DT/$SD done
# I've used the basic outline above since Sept 13, 2006 and have found that the backup takes slightly more than half as long as when I used "find" and "cpio". As with anything else, YMMV.
-
Brian Mathis -
Christian Volker -
Jerry Geis -
Les Mikesell -
Marcelo Roccasalva -
Marko A. Jennings -
MHR -
Milton Calnek -
nate -
Nicolas Thierry-Mieg -
Robert -
Scott Silva -
Toby Bluhm -
William L. Maltby