Hi,
I need to transfer a local raw image to another datacenter. Is there a way to achieve good compression of this image, I heard of tools overwriting unused space with zeros. Is this a good idea?
It's an W2k8 image/NTFS.
Thx Rainer
its cool as long as you use a lossless compression format like bz2
On Thu, Aug 4, 2011 at 9:35 AM, Rainer Traut tr.ml@gmx.de wrote:
Hi,
I need to transfer a local raw image to another datacenter. Is there a way to achieve good compression of this image, I heard of tools overwriting unused space with zeros. Is this a good idea?
It's an W2k8 image/NTFS.
Thx Rainer _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On Thu, 2011-08-04 at 16:35 +0200, Rainer Traut wrote:
Hi,
I need to transfer a local raw image to another datacenter. Is there a way to achieve good compression of this image, I heard of tools overwriting unused space with zeros. Is this a good idea?
It's an W2k8 image/NTFS.
Thx Rainer
I'd use dd if=<local dev> | bzip2 | ssh <remote user>@<remote server> "bunzip2 | dd of=<remote dev>" or something similar.
I'd then do ssh <remote user>@<remote server> "md5sum <remote dev>" ; md5sum <local dev> to give myself some reassurance that they were the same.
you seem to be subscribing to the 'its easier to do it all in one line' approach. don't do that. 1) dump: dd if=<device> of=imagefile.iso 2) compress: bzip2 -z imagfile.iso -c > imagfile.iso.bz2 3) send: scp your_username@remotehost.edu:imagefile.iso /some/local/directory/imagefile.iso.bz2 4) decompress : bunzip imagefile.iso.bz2
ok, enough wrinkles here that you need to iron out by reading the man pages, I'll
On Thu, Aug 04, 2011 at 04:35:20PM +0200, Rainer Traut wrote:
I need to transfer a local raw image to another datacenter. Is there a way to achieve good compression of this image, I heard of tools overwriting unused space with zeros. Is this a good idea?
You're talking about sparse files? You probably want to use rsync with --sparse to keep the file from inflating on the far side. Something like:
rsync -ctrvlzSP source.raw user@destination:source.raw
On 08/04/2011 10:59 PM, Izaac wrote:
On Thu, Aug 04, 2011 at 04:35:20PM +0200, Rainer Traut wrote:
I need to transfer a local raw image to another datacenter. Is there a way to achieve good compression of this image, I heard of tools overwriting unused space with zeros. Is this a good idea?
You're talking about sparse files? You probably want to use rsync with --sparse to keep the file from inflating on the far side. Something like:
rsync -ctrvlzSP source.raw user@destination:source.raw
No, he's talking filling with zeros unused blocks. That's a prerequisite to optimize for a sparse image _on the other side_ and/or compression during the transfer.
I know of an ext2/3/4 utility that does precisely that. I have no idea if such an utility exists for NTFS. Poor man's solution is to create a file filled with zero as big as the free space in the filesystem (or a set of files). It should not be sparse (assuming NTFS supports sparse files). Then you remove it, stop the virtual machine and now you can compress the image at best (you can use -z option to rsync, or use a compression utility and then transfer the image, but using both doesn't make sense).
.TM.
Hi
No, he's talking filling with zeros unused blocks. That's a prerequisite to optimize for a sparse image _on the other side_ and/or compression during the transfer.
I know of an ext2/3/4 utility that does precisely that. I have no idea if such an utility exists for NTFS.
It exists for NTFS, have a look at "sdelete.exe" with -c option.
Momo.
Am 08.08.2011 18:51, schrieb Momo:
Hi
No, he's talking filling with zeros unused blocks. That's a prerequisite to optimize for a sparse image _on the other side_ and/or compression during the transfer. I know of an ext2/3/4 utility that does precisely that. I have no idea if such an utility exists for NTFS.It exists for NTFS, have a look at "sdelete.exe" with -c option.
Sorry for getting back so late.
Yes, guys, exactly what I meant was filling unused space with zeros. The linux tools to compress and transfer cannot read the ntfs data structure and compress every garbage. But if unused blocks are filled with zeros one can achieve much better compression.
I looked into sdelete.exe, this really is helpful but obviously works in running instance.
My idea is to mount the raw image with ntfs3g and create a big file with zeros. This should be sufficient, like Marco said.
Regards, Rainer
Rainer Traut wrote:
Am 08.08.2011 18:51, schrieb Momo:
Hi
No, he's talking filling with zeros unused blocks. That's a prerequisite to optimize for a sparse image _on the other side_ and/or compression during the transfer. I know of an ext2/3/4 utility that does precisely that. I have no idea if such an utility exists for NTFS.It exists for NTFS, have a look at "sdelete.exe" with -c option.
Sorry for getting back so late.
Yes, guys, exactly what I meant was filling unused space with zeros. The linux tools to compress and transfer cannot read the ntfs data structure and compress every garbage. But if unused blocks are filled with zeros one can achieve much better compression.
I looked into sdelete.exe, this really is helpful but obviously works in running instance.
My idea is to mount the raw image with ntfs3g and create a big file with zeros. This should be sufficient, like Marco said.
Regards, Rainer
Don't forget rsync's parms --bwlimit=0 --compress-level=1
On 5 August 2011 00:35, Rainer Traut tr.ml@gmx.de wrote:
It's an W2k8 image/NTFS.
For what it's worth (a bit late). I'm surprised no one mentioned ntfsclone. This will save you from copying unused blocks. Just check that you can restore the image, as there was a version of ntfsclone in rpmforge/epel a while back that was broken.
-- Andrew Cutler=
-
Andrew Cutler -
Charles Polisher -
Ed Heron -
Izaac -
Marco Colombo -
Momo -
Nehemiah -
Nehemiah Dacres
-
Rainer Traut