Hi,
I've discovered a bug in rsync which leads to increased CPU usage and slower transfers in many situations.
When syncing with compression (-z), certain file types should not be compressed during the transfer because they are already compressed. The file types which are not to be compressed can be seen in the man page section --skip-compress.
Unfortunately skipping the default file types doesn't work and all transferred data is being compressed during the transfer. This is true for all versions since 3.1.0.
Steps to Reproduce: 1. run 'rm -f Z ; rsync -azv alpha:z.gz Z'
Actual results, transferred data was compressed during the transfer: sent 43 bytes received 63,873 bytes 25,566.40 bytes/sec total size is 628,952 speedup is 9.84
Expected results: No compression should happen for .gz file.
Additional info: Note that the source file 'z.gz' was an ascii text file to show clearly that compression took place.
Please the following BZ for more info: https://bugzilla.samba.org/show_bug.cgi?id=14323 https://bugzilla.redhat.com/show_bug.cgi?id=1816528
I tried to locate the bug in the code but failed. That said I'm not a C developer and therefore need some help from someone who understands C better than me. Any help is much appreciated!
Thanks and kind regards, Simon
On Wed, 25 Mar 2020 14:49:24 +0100 Simon Matter via CentOS centos@centos.org wrote:
Hi,
I've discovered a bug in rsync which leads to increased CPU usage and slower transfers in many situations.
When syncing with compression (-z)
Tbh, using -z with rsync is almost always a bad idea (unless you're on some pre-historic type of network link..).
/Peter
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
________________________________ From: CentOS centos-bounces@centos.org on behalf of Peter Kjellström cap@nsc.liu.se Sent: Wednesday, March 25, 2020 9:34 AM To: Simon Matter simon.matter@invoca.ch Cc: centos@centos.org centos@centos.org Subject: [EXTERNAL] Re: [CentOS] Need help to fix bug in rsync
On Wed, 25 Mar 2020 14:49:24 +0100 Simon Matter via CentOS centos@centos.org wrote:
Hi,
I've discovered a bug in rsync which leads to increased CPU usage and slower transfers in many situations.
When syncing with compression (-z)
Tbh, using -z with rsync is almost always a bad idea (unless you're on some pre-historic type of network link..).
/Peter _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.comhttp://www..com
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttp://subscribe.harriscomputer.com/.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing small files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
P.
On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing small files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
I really don't understand the discussion here. What is wrong with using -z with rsync? We're using rsync with -z for backups and just don't want to waste bandwidth for nothing. We have better use for our bandwidth and it makes quite a difference when backing up terabytes of data.
The only reason why I asked for help is because we don't want to double compress data which is already compressed. This is what currently is broken in rsync without manually specifying a skip-compress list. Fixing it would help all those who don't know it's broken now.
Thanks, Simon
On Wed, 2020-03-25 at 19:15 +0100, Simon Matter via CentOS wrote:
On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing small files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
I really don't understand the discussion here. What is wrong with using -z with rsync? We're using rsync with -z for backups and just don't want to waste bandwidth for nothing. We have better use for our bandwidth and it makes quite a difference when backing up terabytes of data.
I don't really care if you use -z, but you asked for the rationale, and I gave you it. I'm not telling you what you should do.
I'll try and make it simpler - if rsync takes 1 second to compress the file, then 1 second to decompress the file, and the whole transfer of the file takes 11 seconds uncompressed vs 10 seconds compressed, then dealing with file takes overall 12 seconds compressed, vs 11 seconds uncompressed. It's not worth it.
But as I said it depends on your network and your machine speeds. It's up to you to decide what is best in your own situation.
P.
I appreciate the reply - it keeps me from wondering "is there something I should be concerned about?". We use a co-location facility where we pay for bandwidth utilization so it's still an issue.
________________________________ From: CentOS centos-bounces@centos.org on behalf of Pete Biggs pete@biggs.org.uk Sent: Wednesday, March 25, 2020 1:32 PM To: centos@centos.org centos@centos.org Subject: [EXTERNAL] Re: [CentOS] Need help to fix bug in rsync
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.comhttp://www..com
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttp://subscribe.harriscomputer.com/.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
On Wed, 2020-03-25 at 19:15 +0100, Simon Matter via CentOS wrote:
On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing small files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
I really don't understand the discussion here. What is wrong with using -z with rsync? We're using rsync with -z for backups and just don't want to waste bandwidth for nothing. We have better use for our bandwidth and it makes quite a difference when backing up terabytes of data.
I don't really care if you use -z, but you asked for the rationale, and I gave you it. I'm not telling you what you should do.
I'll try and make it simpler - if rsync takes 1 second to compress the file, then 1 second to decompress the file, and the whole transfer of the file takes 11 seconds uncompressed vs 10 seconds compressed, then dealing with file takes overall 12 seconds compressed, vs 11 seconds uncompressed. It's not worth it.
But as I said it depends on your network and your machine speeds. It's up to you to decide what is best in your own situation.
P.
_______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
I appreciate the reply - it keeps me from wondering "is there something I should be concerned about?". We use a co-location facility where we pay for bandwidth utilization so it's still an issue.
Hi,
You are right, bandwidth always costs, always.
Who pays for it can differ a lot, but there is always someone who pays.
It can be an individual, a company, the tax payer, whoever, but bandwidth is never free.
Thanks, Simon
From: CentOS centos-bounces@centos.org on behalf of Pete Biggs pete@biggs.org.uk Sent: Wednesday, March 25, 2020 1:32 PM To: centos@centos.org centos@centos.org Subject: [EXTERNAL] Re: [CentOS] Need help to fix bug in rsync
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.comhttp://www..com
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttp://subscribe.harriscomputer.com/.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
On Wed, 2020-03-25 at 19:15 +0100, Simon Matter via CentOS wrote:
On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing
small
files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
I really don't understand the discussion here. What is wrong with using -z with rsync? We're using rsync with -z for backups and just don't want to waste bandwidth for nothing. We have better use for our bandwidth and it makes quite a difference when backing up terabytes of data.
I don't really care if you use -z, but you asked for the rationale, and I gave you it. I'm not telling you what you should do.
I'll try and make it simpler - if rsync takes 1 second to compress the file, then 1 second to decompress the file, and the whole transfer of the file takes 11 seconds uncompressed vs 10 seconds compressed, then dealing with file takes overall 12 seconds compressed, vs 11 seconds uncompressed. It's not worth it.
But as I said it depends on your network and your machine speeds. It's up to you to decide what is best in your own situation.
P.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
On Thu, 26 Mar, 2020 at 18:12:50 -0600, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
What you see is something the rest of us can not, because you don't provide links. What you regard as 'minimal' I have no idea.
well, sorry, I thought it was somewhat "self-explaining", since that terminology was used up until Centos 7 (see
links), andof course I meant the official download page.
minimal: as in approx 3Gb or so that fits on a regular 4-5Gb rewritable DVD as with Centos 7
download from : https://www.centos.org/download/
anything else blatantly obvious you need me to have spelled out?
Ron
On 3/26/20 6:27 PM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:12:50 -0600, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
What you see is something the rest of us can not, because you don't provide links. What you regard as 'minimal' I have no idea. _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Thu, 26 Mar, 2020 at 18:39:56 -0600, R C wrote:
well, sorry, I thought it was somewhat "self-explaining", since that terminology was used up until Centos 7 (see
links), andof course I meant the official download page.
minimal: as in approx 3Gb or so that fits on a regular 4-5Gb rewritable DVD as with Centos 7
download from : https://www.centos.org/download/
anything else blatantly obvious you need me to have spelled out?
Ron
On 3/26/20 6:27 PM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:12:50 -0600, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
What you see is something the rest of us can not, because you don't provide links. What you regard as 'minimal' I have no idea.
Yes, perhaps you could explain to us why you are top posting.
So your idea of minimal is 3GB. Mine is much less. The minimal ISO of CentOS 7 is < 1GB:
just to piss off people like you,
you might know the type, they never really answer any questions, lurk around on a forum like this
acting like a (self-proclaimed) genius lecturing and patronizing people, as if they have to be worthy
to get an answer from you (which you typically never do anyway), formulated by your delusional standards.
If your mindset is like that, why don't you just not answer to begin with, nobody is really interested would
be my best bet.
and no, my idea of minimal is not 3GB, try and read the thread again. However your personality comes to mind.
On 3/27/20 9:35 AM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:39:56 -0600, R C wrote:
well, sorry, I thought it was somewhat "self-explaining", since that terminology was used up until Centos 7 (see
links), andof course I meant the official download page.
minimal: as in approx 3Gb or so that fits on a regular 4-5Gb rewritable DVD as with Centos 7
download from : https://www.centos.org/download/
anything else blatantly obvious you need me to have spelled out?
Ron
On 3/26/20 6:27 PM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:12:50 -0600, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
What you see is something the rest of us can not, because you don't provide links. What you regard as 'minimal' I have no idea.
Yes, perhaps you could explain to us why you are top posting.
So your idea of minimal is 3GB. Mine is much less. The minimal ISO of CentOS 7 is < 1GB:
http://ftp.heanet.ie/pub/centos/7.7.1908/isos/x86_64/ _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Fri, Mar 27, 2020, 2:33 PM R C cjvijf@gmail.com wrote:
just to piss off people like you,
you might know the type, they never really answer any questions, lurk around on a forum like this
acting like a (self-proclaimed) genius lecturing and patronizing people, as if they have to be worthy
to get an answer from you (which you typically never do anyway), formulated by your delusional standards.
If your mindset is like that, why don't you just not answer to begin with, nobody is really interested would
be my best bet.
and no, my idea of minimal is not 3GB, try and read the thread again. However your personality comes to mind.
On 3/27/20 9:35 AM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:39:56 -0600, R C wrote:
well, sorry, I thought it was somewhat "self-explaining", since that terminology was used up until Centos 7 (see
links), andof course I meant the official download page.
minimal: as in approx 3Gb or so that fits on a regular 4-5Gb rewritable
DVD
as with Centos 7
download from : https://www.centos.org/download/
anything else blatantly obvious you need me to have spelled out?
Ron
On 3/26/20 6:27 PM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:12:50 -0600, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
What you see is something the rest of us can not, because you don't provide links. What you regard as 'minimal' I have no idea.
Yes, perhaps you could explain to us why you are top posting.
So your idea of minimal is 3GB. Mine is much less. The minimal ISO of
CentOS 7 is < 1GB:
http://ftp.heanet.ie/pub/centos/7.7.1908/isos/x86_64/ _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
I know I rarely post but can we keep it civil on the list, with everything that's going on in the world is it really necessary? If it bothers you that much just delete the email it's awful easy to do these days and poof it's gone from your inbox, problem solved.
There is no need for that level of discourse on this mail list.
:)
--On Friday, March 27, 2020 4:00 PM -0500 Tom Bishop bishoptf@gmail.com wrote:
I know I rarely post but can we keep it civil on the list, with everything that's going on in the world is it really necessary?
I urge everyone to apply Hanlon's Razor. ;)
Also, please trim. We don't need to see all that went before. This is a mailing list, not a business email with CC's coming and going that requires the entire exchange be carried in every message to maintain context. With a mailing list, new subscribers can go look at the archives to get the context, so trimming is important. Just give the briefest context necessary to see the relevance of your addition.
Oh please. It was all in light of people being anal.
Therefore…
I highly suggest that people refrain from being anal on the list. People can also go to the archives to see that top or bottom posting…nobody cares.
On Mar 27, 2020, at 8:46 PM, Kenneth Porter shiva@sewingwitch.com wrote:
--On Friday, March 27, 2020 4:00 PM -0500 Tom Bishop bishoptf@gmail.com wrote:
I know I rarely post but can we keep it civil on the list, with everything that's going on in the world is it really necessary?
I urge everyone to apply Hanlon's Razor. ;)
Also, please trim. We don't need to see all that went before. This is a mailing list, not a business email with CC's coming and going that requires the entire exchange be carried in every message to maintain context. With a mailing list, new subscribers can go look at the archives to get the context, so trimming is important. Just give the briefest context necessary to see the relevance of your addition.
Cheers, Bee
Am 28.03.20 um 03:20 schrieb Bee.Lists:
Oh please. It was all in light of people being anal.
Therefore…
I highly suggest that people refrain from being anal on the list. People can also go to the archives to see that top or bottom posting…nobody cares.
This is valid as also that somebody cares is valid.
Please use your energy (to all subscribers) to answer unanswered questions on the list instead using impolitement as a pressure blowout.
The transition to additional services like forums, feeds and other 3rd party exchange portals, the live for a mailing list gets harder. It would be great to make this here as pleasant as possible.
-- Leon
Then I care that others stop harassing people on remedial stuff. Because I care. So that means you need to follow suit.
It’s like being married.
On Mar 28, 2020, at 7:29 AM, Leon Fauster via CentOS centos@centos.org wrote:
This is valid as also that somebody cares is valid.
Cheers, Bee
On Fri, 27 Mar, 2020 at 13:33:02 -0600, R C wrote:
just to piss off people like you,
you might know the type, they never really answer any questions, lurk around on a forum like this
acting like a (self-proclaimed) genius lecturing and patronizing people, as if they have to be worthy
to get an answer from you (which you typically never do anyway), formulated by your delusional standards.
If your mindset is like that, why don't you just not answer to begin with, nobody is really interested would
be my best bet.
and no, my idea of minimal is not 3GB, try and read the thread again.
You stated: "minimal: as in approx 3Gb or so".
However your personality comes to mind.
On 3/27/20 9:35 AM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:39:56 -0600, R C wrote:
well, sorry, I thought it was somewhat "self-explaining", since that terminology was used up until Centos 7 (see
links), andof course I meant the official download page.
minimal: as in approx 3Gb or so that fits on a regular 4-5Gb rewritable DVD as with Centos 7
download from : https://www.centos.org/download/
anything else blatantly obvious you need me to have spelled out?
Ron
On 3/26/20 6:27 PM, Liam O'Toole wrote:
On Thu, 26 Mar, 2020 at 18:12:50 -0600, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
What you see is something the rest of us can not, because you don't provide links. What you regard as 'minimal' I have no idea.
Yes, perhaps you could explain to us why you are top posting.
So your idea of minimal is 3GB. Mine is much less. The minimal ISO of CentOS 7 is < 1GB:
http://ftp.heanet.ie/pub/centos/7.7.1908/isos/x86_64/ _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
At 05:12 PM 3/26/2020, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
Ron:
The ISO called 'boot' is the equivalent. When you get to selecting the software, choose 'minimal', or whichever option seems to work best for you. I chose 'minimal', because I have scripts that install just what I want.
It's beyond me why it takes 600 megs, but at least it fits on a CD (if anyone still uses them). And it does work on at least one physical box and a VirtualBox VM on windows.
David
Hi David,
right, that's the easies, have something minimal (or 'boot' if it is called that), that you can just install a basic machine with, and
install everything else as needed while connected to a network.
Yup, 600Mb is kinda stiff. I am installing it on a physical box.
thanks!
Ron
On 3/26/20 6:41 PM, david wrote:
At 05:12 PM 3/26/2020, R C wrote:
Hello,
is there a minimal install for Centos 8?
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
thanks,
Ron
Ron:
The ISO called 'boot' is the equivalent. When you get to selecting the software, choose 'minimal', or whichever option seems to work best for you. I chose 'minimal', because I have scripts that install just what I want.
It's beyond me why it takes 600 megs, but at least it fits on a CD (if anyone still uses them). And it does work on at least one physical box and a VirtualBox VM on windows.
David
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hi R C.
The resulting installation is smaller than the image that you download through FTP, etc. When I first installed this OS during version 5, I was amazed that the installation took a mere fraction of the time it took to go through the menu. Heh.
Anyway, give it a whirl as it’s quick. I’m assuming you mean a headless installation.
Oh and I’m top posting because it’s logical.
On Mar 26, 2020, at 8:12 PM, R C cjvijf@gmail.com wrote:
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
Cheers, Bee
Hi,
someone pointed out the 'boot' iso, it's approx 600Mb, which works if you can connect to a network. I used that.
In previous versions, I just downloaded the 'minimal install iso', mostly because it is a habbit because my network connection used to
be a lot slower, so I downloaded them over night.
I tend to install in different ways at work, just depends. Of course you can get the 7.something or 8.something iso, mount it on a loop
and export it to use it as an install, or http, or kickstart etc etc.. but if not too big, having a disk with it comes in handy.
(especially if it is not supported anymore, and you really need that old disto, because of that 'thing' you need to run *lol*)
It's not headless, I have a few machines on a kvm switch, but after the initial install I use as if they are.
Yeah I don't care much about top or bottom either, if someone likes top, it's top, if someone likes bottom it's bottom *lol*
I think posting on top is easier than bottom than shredded in the middle, less likely overlook stuff that way.... whatever you choose, some
just like top, others get po-ed by it.. can't make everyone happy all the tme.
thanks for the tip !
Ron
On 3/27/20 3:08 PM, Bee.Lists wrote:
Hi R C.
The resulting installation is smaller than the image that you download through FTP, etc. When I first installed this OS during version 5, I was amazed that the installation took a mere fraction of the time it took to go through the menu. Heh.
Anyway, give it a whirl as it’s quick. I’m assuming you mean a headless installation.
Oh and I’m top posting because it’s logical.
On Mar 26, 2020, at 8:12 PM, R C cjvijf@gmail.com wrote:
What I see is a 7.4Gb and a 8Gb iso, that can't be 'minimal'
Cheers, Bee
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
That's why I asked, I wanted to know if there was something inherently bad with "-z". I had a situation where Postgresql was replicating 16M files every few minutes ("log shipping") on approximately 10 systems, got behind which resulted in almost continuous file transfer (of mostly null 16M files) and saturated the common link. Specifying compression with file transfer cut transmission time by 5-10x resolving the problem.
________________________________ From: CentOS centos-bounces@centos.org on behalf of Simon Matter via CentOS centos@centos.org Sent: Wednesday, March 25, 2020 1:15 PM To: CentOS mailing list centos@centos.org Subject: [EXTERNAL] Re: [CentOS] Need help to fix bug in rsync
&g
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.comhttp://www..com
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttp://subscribe.harriscomputer.com/.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
t; On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing small files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
I really don't understand the discussion here. What is wrong with using -z with rsync? We're using rsync with -z for backups and just don't want to waste bandwidth for nothing. We have better use for our bandwidth and it makes quite a difference when backing up terabytes of data.
The only reason why I asked for help is because we don't want to double compress data which is already compressed. This is what currently is broken in rsync without manually specifying a skip-compress list. Fixing it would help all those who don't know it's broken now.
Thanks, Simon
_______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Am 25.03.20 um 19:15 schrieb Simon Matter via CentOS:
On Wed, 2020-03-25 at 14:39 +0000, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I think the "rationale" is that at some point the compression/decompression takes longer than the time reduction from sending a compressed file. It depends on the relative speeds of the machines and the network.
You have most to gain from compressing large files, but if they are already compressed, then you have nothing to gain from just doing small files.
It obviously depends on your network speed and if you have a metered connection, but does anyone really have such an ancient network connection still these days - I mean if you have fast enough machines at both ends to do rapid compression/decompression, it seems unlikely that you will have a damp piece of string connecting them.
I really don't understand the discussion here. What is wrong with using -z with rsync? We're using rsync with -z for backups and just don't want to waste bandwidth for nothing. We have better use for our bandwidth and it makes quite a difference when backing up terabytes of data.
The only reason why I asked for help is because we don't want to double compress data which is already compressed. This is what currently is broken in rsync without manually specifying a skip-compress list. Fixing it would help all those who don't know it's broken now.
Until this is fixed; as a workaround I would do a two-pass transfer with filters via ".rsync-filter" file and then using rsync -azvF for everything with high compression ratio and rsync -av for all, including compressed data. So, ".rsync-filter" includes the exclude statements for compressed formats. This all makes only sense if the compression ratio is higher then the meta data transfer of the second run ...
-- Leon
On Wed, 25 Mar 2020, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I can't speak to that, but the obvious workaround is to use ssh's compression instead of rsync's:
rsync -av -e 'ssh -C' remotehost:remote.file local.file
On Wed, 25 Mar 2020, Leroy Tennison wrote:
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something.
I can't speak to that, but the obvious workaround is to use ssh's compression instead of rsync's:
rsync -av -e 'ssh -C' remotehost:remote.file local.file
From what I understand that's not really the same as having compression
inside rsync makes rsync transfers even better than only compressing the transport.
Thanks, Simon
On Wed, Mar 25, 2020 at 02:49:24PM +0100, Simon Matter via CentOS wrote:
Hi,
I've discovered a bug in rsync which leads to increased CPU usage and slower transfers in many situations.
When syncing with compression (-z), certain file types should not be compressed during the transfer because they are already compressed. The file types which are not to be compressed can be seen in the man page section --skip-compress.
Unfortunately skipping the default file types doesn't work and all transferred data is being compressed during the transfer. This is true for all versions since 3.1.0.
Steps to Reproduce:
- run 'rm -f Z ; rsync -azv alpha:z.gz Z'
Actual results, transferred data was compressed during the transfer: sent 43 bytes received 63,873 bytes 25,566.40 bytes/sec total size is 628,952 speedup is 9.84
Expected results: No compression should happen for .gz file.
Additional info: Note that the source file 'z.gz' was an ascii text file to show clearly that compression took place.
That may invalidate your testing.
rsync may not depend upon the filename extension but instead check the magic numbers within the file to determine whether to compress or not.
Jon
On Wed, Mar 25, 2020 at 02:49:24PM +0100, Simon Matter via CentOS wrote:
Hi,
I've discovered a bug in rsync which leads to increased CPU usage and slower transfers in many situations.
When syncing with compression (-z), certain file types should not be compressed during the transfer because they are already compressed. The file types which are not to be compressed can be seen in the man page section --skip-compress.
Unfortunately skipping the default file types doesn't work and all transferred data is being compressed during the transfer. This is true for all versions since 3.1.0.
Steps to Reproduce:
- run 'rm -f Z ; rsync -azv alpha:z.gz Z'
Actual results, transferred data was compressed during the transfer: sent 43 bytes received 63,873 bytes 25,566.40 bytes/sec total size is 628,952 speedup is 9.84
Expected results: No compression should happen for .gz file.
Additional info: Note that the source file 'z.gz' was an ascii text file to show clearly that compression took place.
That may invalidate your testing.
rsync may not depend upon the filename extension but instead check the magic numbers within the file to determine whether to compress or not.
One could think so but it's not the case, rsync just uses the file suffix to decide which files to compress. It even lower cases them, leaving good old UNIX .Z file behind... Also a bug IMHO but not so important.
Simon
-
Bee.Lists -
david -
Jon LaBadie -
Kenneth Porter -
Leon Fauster -
Leroy Tennison -
Liam O'Toole -
Paul Heinlein -
Pete Biggs -
Peter Kjellström -
R C -
Simon Matter -
Tom Bishop