On Dec 17, 2007 5:16 PM, Kenneth Porter shiva@sewingwitch.com wrote:
--On Monday, December 17, 2007 4:58 PM -0800 centos@911networks.com wrote:
Why use torrents? With torrents I get around 25Kb/sec.
Sounds like something is throttling your torrent connection. Start by using a non-standard torrent port to escape traffic shaping by naive throttles.
I think the EFF was accusing Comcast of doing this. http://arstechnica.com/news.ars/post/20071128-eff-study-reveals-evidence-of-...
With places such as utah.edu [I am in North America] I got 320Kb/sec steady. It took me 3hr and a bit to download the 5.1 dvd. As far as I understand it, Utah and the other mirrors donated the bandwidth to the community.
Torrents have the benefit of sharing the cost over many community contributors.
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
On Monday, December 17, 2007 7:05 PM -0800 Robert Arkiletian robark@gmail.com wrote:
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
That surprises me. Won't similar RPM's in the two images likely be at different offsets in the iso? Can rsync deal with that? I didn't realize it could handle shifted differences.
On Mon, 17 Dec 2007, Kenneth Porter wrote:
On Monday, December 17, 2007 7:05 PM -0800 Robert Arkiletian robark@gmail.com wrote:
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
That surprises me. Won't similar RPM's in the two images likely be at different offsets in the iso? Can rsync deal with that? I didn't realize it could handle shifted differences.
Rsync has no concept of the contents of files within an iso. It only knows about blocks of data. Rsync is real good at fixing/updating iso's. It will only transfer the blocks of data that have changed. Typically you get a large bandwidth savings when using a previous iso as a seed. Just make sure you get the name correct AND you do not abort the download midstream. If you abort the download midstream rsync will delete your seed iso and leave you with whatever it has succeeded in downloading and running a checksum against.
If you are really curious about all of this magic have a look here: http://rsync.samba.org/how-rsync-works.html
Most people do not realize just how cool rsync is.
Hope this helps.
Regards,
--On Tuesday, December 18, 2007 12:52 PM -0500 Tom Diehl tdiehl@rogueind.com wrote:
If you are really curious about all of this magic have a look here: http://rsync.samba.org/how-rsync-works.html
Ah, here's the critical verbiage, in the section titled "The Sender":
If a block checksum match is found it is considered a matching block and any accumulated non-matching data will be sent to the receiver followed by the offset and length in the receiver's file of the matching block and the block checksum generator will be advanced to the next byte after the matching block.
Matching blocks can be identified in this way even if the blocks are reordered or at different offsets. This process is the very heart of the rsync algorithm.
Because an ISO is actually an uncompressed filesystem and the files within it are going to be aligned on CD/DVD block boundaries, matching files from the 5.0 and 5.1 release will differ only in block-quantized offset, so there should be lots of common blocks. (This assumes that the ISO block size is some integral multiple of the rsync block size.)
On Dec 17, 2007, at 10:05 PM, Robert Arkiletian wrote:
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
I could be tired, but could you please elaborate how this is possible? A dvd iso with X number of files of all different bytes diff'd against another dvd iso with X number of entirely different bytes.
On 12/17/07, Von centos.list@eyestreet.com wrote:
On Dec 17, 2007, at 10:05 PM, Robert Arkiletian wrote:
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
I could be tired, but could you please elaborate how this is possible?
mv CentOS-5.0-i386-bin-DVD.iso CentOS-5.1-i386-bin-DVD.iso
Check out the public mirrors list. Only mirrors that offer full DVD downloads and rsync will work. Hover your mouse over the rsync link to see the rsync address.
For example, if I was to use the kernel.org mirror it's rsync://mirrors.kernel.org/centos
then navigate the http or ftp directory structure to see the full path to the iso. Then run rsync
rsync -Pv rsync://mirrors.kernel.org/centos/5.1/isos/i386/CentOS-5.1-i386-bin-DVD.iso .
You should get upto 50% savings. At least I did with the CD iso's in the past. The capital P is --partial --progress.
On Dec 18, 2007, at 4:29 PM, Robert Arkiletian wrote:
On 12/17/07, Von centos.list@eyestreet.com wrote:
On Dec 17, 2007, at 10:05 PM, Robert Arkiletian wrote:
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
I could be tired, but could you please elaborate how this is possible?
mv CentOS-5.0-i386-bin-DVD.iso CentOS-5.1-i386-bin-DVD.iso
Check out the public mirrors list. Only mirrors that offer full DVD downloads and rsync will work. Hover your mouse over the rsync link to see the rsync address.
For example, if I was to use the kernel.org mirror it's rsync://mirrors.kernel.org/centos
then navigate the http or ftp directory structure to see the full path to the iso. Then run rsync
rsync -Pv rsync://mirrors.kernel.org/centos/5.1/isos/i386/CentOS-5.1- i386-bin-DVD.iso .
You should get upto 50% savings. At least I did with the CD iso's in the past. The capital P is --partial --progress.
Brilliant. Thanks for the tip!
On 12/18/07, Von centos.list@eyestreet.com wrote:
Brilliant. Thanks for the tip!
Thank Andrew Tridgell. He's the brilliant one that wrote rsync and Samba. :)
--On Tuesday, December 18, 2007 1:29 PM -0800 Robert Arkiletian robark@gmail.com wrote:
rsync -Pv rsync://mirrors.kernel.org/centos/5.1/isos/i386/CentOS-5.1-i386-bin-DVD.i so .
You should get upto 50% savings. At least I did with the CD iso's in the past. The capital P is --partial --progress.
My results:
CentOS-5.1-i386-bin-DVD.iso 3845230592 100% 4.93MB/s 0:12:24 (xfer#1, to-check=0/1)
sent 487931 bytes received 1797247739 bytes 2086750.63 bytes/sec total size is 3845230592 speedup is 2.14
24F3AFE0D@[10.170.7.6]> X-Rcpt-To: centos@centos.org
Kenneth Porter wrote on Tue, 18 Dec 2007 15:01:31 -0800:
speedup is 2.14
Ah, and I was wondering all the time what that means. Seems to be the reciprocal downloaded:keptfromseed ratio?
Kai
On Tuesday 18 December 2007 03:05:52 Robert Arkiletian wrote:
On Dec 17, 2007 5:16 PM, Kenneth Porter shiva@sewingwitch.com wrote:
--On Monday, December 17, 2007 4:58 PM -0800 centos@911networks.com wrote:
Why use torrents? With torrents I get around 25Kb/sec.
Sounds like something is throttling your torrent connection. Start by using a non-standard torrent port to escape traffic shaping by naive throttles.
I think the EFF was accusing Comcast of doing this. http://arstechnica.com/news.ars/post/20071128-eff-study-reveals-evidence-of -comcasts-bittorrent-interference.html
Over here in the UK most if not all the ISP's will throttle people that they think are file sharing. Try changing the port that the bittorrent software is using. Remember to do it for UDP and TCP packets on the same port and open and / or open the same port on your firewall.
With places such as utah.edu [I am in North America] I got 320Kb/sec steady. It took me 3hr and a bit to download the 5.1 dvd. As far as I understand it, Utah and the other mirrors donated the bandwidth to the community.
Torrents have the benefit of sharing the cost over many community contributors.
Also don't forget that many mirrors offer rsync. If you rename your 5.0 DVD to the 5.1 version and do an rsync it will save lots of bandwidth.
-
John Bowden -
Kai Schaetzl -
Kenneth Porter -
Robert Arkiletian -
Tom Diehl -
Von