A few users have reported we have a bad copy of yum on our mirror causing update failures. The yum package on mirror.steadfast.net had the md5sum:
0ef11f016a2cf6545ae4abb356abc498 yum-3.2.8-9.el5.centos.2.1.noarch.rpm
A copy from mirror.centos.org has:
d776014eb266b87299465cb38e0f6b96 yum-3.2.8-9.el5.centos.2.1.noarch.rpm
After replacing the RPM with the one from mirror.centos.org, it seems to verify correctly. If I delete the file from our mirror and allow rsync to fetch it again, it receives the same file as it did originally, with the wrong md5sum. The file sizes however, are the same.
Is there a bad copy of the file on msync-dvd.centos.org, which we sync from?
Thanks.
Kevin Stange wrote:
A few users have reported we have a bad copy of yum on our mirror causing update failures. The yum package on mirror.steadfast.net had the md5sum:
I forgot to mention this was at:
/centos/5/updates/x86_64/RPMS/
Additionally, I've heard reports that more than just the "yum" package were reporting errors. It seems it may be limited to x86_64, but I'd rather not have to go through and md5sum the whole tree.
Kevin
Kevin Stange wrote:
Kevin Stange wrote:
A few users have reported we have a bad copy of yum on our mirror causing update failures. The yum package on mirror.steadfast.net had the md5sum:
I forgot to mention this was at:
/centos/5/updates/x86_64/RPMS/
Additionally, I've heard reports that more than just the "yum" package were reporting errors. It seems it may be limited to x86_64, but I'd rather not have to go through and md5sum the whole tree.
I heard that there was a problem with external mirrors, I did not hear that there was a problem on msync-dvd.centos.org.
I can test all the md5sums for all yum packages if there seems to be a problem on msync-dvd.
not sure how these could be wrong as they should be hardlinks of one another (all the files in question are noarch files, so they were only built once)
Johnny Hughes wrote:
Kevin Stange wrote:
Kevin Stange wrote:
A few users have reported we have a bad copy of yum on our mirror causing update failures. The yum package on mirror.steadfast.net had the md5sum:
I forgot to mention this was at:
/centos/5/updates/x86_64/RPMS/
Additionally, I've heard reports that more than just the "yum" package were reporting errors. It seems it may be limited to x86_64, but I'd rather not have to go through and md5sum the whole tree.
I heard that there was a problem with external mirrors, I did not hear that there was a problem on msync-dvd.centos.org.
I can test all the md5sums for all yum packages if there seems to be a problem on msync-dvd.
not sure how these could be wrong as they should be hardlinks of one another (all the files in question are noarch files, so they were only built once)
Did you just fix something? I ran a new sync and it re-hardlinked a bunch of files that were previously not and there are no longer any files that aren't hardlinked.
Kevin
Kevin Stange wrote:
Johnny Hughes wrote:
Kevin Stange wrote:
Kevin Stange wrote:
A few users have reported we have a bad copy of yum on our mirror causing update failures. The yum package on mirror.steadfast.net had the md5sum:
I forgot to mention this was at:
/centos/5/updates/x86_64/RPMS/
Additionally, I've heard reports that more than just the "yum" package were reporting errors. It seems it may be limited to x86_64, but I'd rather not have to go through and md5sum the whole tree.
I heard that there was a problem with external mirrors, I did not hear that there was a problem on msync-dvd.centos.org.
I can test all the md5sums for all yum packages if there seems to be a problem on msync-dvd.
not sure how these could be wrong as they should be hardlinks of one another (all the files in question are noarch files, so they were only built once)
Did you just fix something? I ran a new sync and it re-hardlinked a bunch of files that were previously not and there are no longer any files that aren't hardlinked.
Yes, I went through and manually hard linked all the noarch files, as you were correct that some had different md5sums even though they were hard linked.
I am looking for an answer as to why the hardlink.py program might have screwed this up ... I have no idea how or why it might have done so.
I have turned off the script that hardlinks files until we can clear up what seems to cause this issue.
2 files linked together and 2 other files linked together instead of 4 files linked together?
-----Original Message----- From: centos-mirror-bounces@centos.org
[mailto:centos-mirror-bounces@centos.org] On Behalf Of H. Peter
Anvin Sent: Monday, July 21, 2008 7:22 PM To: Mailing list for CentOS mirrors. Subject: Re: [CentOS-mirror] Yum 3.2.8-9 from msync-dvd.centos.org
wrong?
Johnny Hughes wrote:
Yes, I went through and manually hard linked all the noarch files,
as
you were correct that some had different md5sums even though they
were
hard linked.
How could files have different md5sums if they are hard-linked?
-hpa
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
I have had trouble getting yum to work on a box that was fairly behind on it's updates. As a work around I ended up downloading several yum packages by hand, rpm -e the old ones and rpm -i the new ones.
I still don't see how they could have different checksums if they are hard linked.
-----Original Message----- From: centos-mirror-bounces@centos.org
[mailto:centos-mirror-bounces@centos.org] On Behalf Of H. Peter
Anvin Sent: Monday, July 21, 2008 7:28 PM To: Mailing list for CentOS mirrors. Subject: Re: [CentOS-mirror] Yum 3.2.8-9 from msync-dvd.centos.org
wrong?
Lauro, John wrote:
2 files linked together and 2 other files linked together instead
of 4
files linked together?
Then they're not hard-linked together, though.
FWIW, we have gotten these bug reports from our users, too.
-hpa _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
Lauro, John wrote:
I have had trouble getting yum to work on a box that was fairly behind on it's updates. As a work around I ended up downloading several yum packages by hand, rpm -e the old ones and rpm -i the new ones.
I still don't see how they could have different checksums if they are hard linked.
agreed ... so their must be something wrong with the hardlink.py program ...
i say this because, we only made ONE file (they were noarch files) ... they were copied to one place and hard linked into the other place.
then automated -avzH rsyncs were used to copy to other places
then hardlink.py was run against the repos ... and the x86_64 noarch files changed, but the i386 ones stayed the same.
i do not know how ... just that it did happen
-----Original Message----- From: centos-mirror-bounces@centos.org
[mailto:centos-mirror-bounces@centos.org] On Behalf Of H. Peter
Anvin Sent: Monday, July 21, 2008 7:28 PM To: Mailing list for CentOS mirrors. Subject: Re: [CentOS-mirror] Yum 3.2.8-9 from msync-dvd.centos.org
wrong?
Lauro, John wrote:
2 files linked together and 2 other files linked together instead
of 4
files linked together?
Then they're not hard-linked together, though.
FWIW, we have gotten these bug reports from our users, too.
Johnny,
Johnny Hughes wrote:
i say this because, we only made ONE file (they were noarch files) ... they were copied to one place and hard linked into the other place.
then automated -avzH rsyncs were used to copy to other places
then hardlink.py was run against the repos ... and the x86_64 noarch files changed, but the i386 ones stayed the same.
AFAICT, hardlink.py checks a number of things to determine whether or not files are eligible for hardlinking:
* size is the same * size is not zero * file mode is the same * owner user id is the same * owner group id is the same * modified time is the same (unless date hashing is disabled)
Depending on how the files were copied, and how the script was run, I could see the script failing to hardlink properly - but not how you could end up with a changed file. It seems like that would have to have happened during the copy or rsync.
For forensic purposes, does anyone have a copy of an altered RPM around to diff/bindiff against the proper version?
-Brandon
Brandon Davidson wrote:
AFAICT, hardlink.py checks a number of things to determine whether or not files are eligible for hardlinking:
- size is the same
- size is not zero
- file mode is the same
- owner user id is the same
- owner group id is the same
- modified time is the same (unless date hashing is disabled)
Depending on how the files were copied, and how the script was run, I could see the script failing to hardlink properly - but not how you could end up with a changed file. It seems like that would have to have happened during the copy or rsync.
Does it actually compare the contents?
-hpa
H. Peter Anvin wrote:
Brandon Davidson wrote:
AFAICT, hardlink.py checks a number of things to determine whether or not files are eligible for hardlinking:
- size is the same
- size is not zero
- file mode is the same
- owner user id is the same
- owner group id is the same
- modified time is the same (unless date hashing is disabled)
Does it actually compare the contents?
Yes, it looks like it does :) If it passes the above eligibility check, it then continues on to a raw block comparison:
buffer_size = 1024*1024 while 1: buffer1 = file1.read(buffer_size) buffer2 = file2.read(buffer_size) if buffer1 != buffer2: result = False break if not buffer1: result = True break
I don't claim to be a Python expert, but I don't see any obvious problems with this. There is of course the obvious window between the initial stat and the following block comparison in which the files could be modified, but it seems unlikely to be the cause of our problems in this case.
FYI, I'm looking here: http://code.google.com/p/hardlinkpy/source/browse/trunk/hardlink.py
-Brandon
-
Brandon Davidson -
H. Peter Anvin -
Johnny Hughes -
Kevin Stange -
Lauro, John