How can binaries be different when package versions are identical? (mkfs.ext3 on CentOS 5.4)
Situation: two similar servers, both with CentOS 5.4 64-bit. Making an ext3 filesystem on one takes seconds, on the other minutes.
Partition Size Server 1 Server 2
1 GB 0.7 sec 0.3 sec
4 GB 2.5 sec 1.2 sec
40 GB 15 sec 13 min
WTF
e2fsprogs RPM is e2fsprogs-1.39-23.el5 on both systems mkfs --version reports "util-linux 2.13-pre7" on both systems mkfs -V reports the same on both systems (mke2fs 1.3 (29-May-2006) Using EXT2FS Library version 1.39) file size on mkfs and mkfs.ext3 binaries is identical
However, checksum on these binaries differs across servers, and "cmp" reports files are different
I took "mkfs" and "mkfs.ext3" from server 1, put them on server 2, and got an improvement:
40 GB 15 sec 3 min
Just out of curiousity, I ran "rpm -V e2fsprogs" to see if rpm would pick up the difference and sure enough, it complained mkfs.ext3 changed.
Both of these servers have CentOS 5.4; Server 1 was installed in Dec 2009, Server 2 in April 2010.
I'd like to:
(a) account for the difference in the binaries, and (b) see if something else is different that I can make the same to get the mkfs.ext3 time down to 15 sec on both systems.
Solving (a) should shed light on (b). Any ideas?
Best, Aleksey
On Tue, 2010-06-29 at 18:07 -0700, Aleksey Tsalolikhin wrote:
Situation: two similar servers, both with CentOS 5.4 64-bit. Making an ext3 filesystem on one takes seconds, on the other minutes.
Partition Size Server 1 Server 2
1 GB 0.7 sec 0.3 sec
4 GB 2.5 sec 1.2 sec
40 GB 15 sec 13 min
WTF
e2fsprogs RPM is e2fsprogs-1.39-23.el5 on both systems mkfs --version reports "util-linux 2.13-pre7" on both systems mkfs -V reports the same on both systems (mke2fs 1.3 (29-May-2006) Using EXT2FS Library version 1.39) file size on mkfs and mkfs.ext3 binaries is identical
However, checksum on these binaries differs across servers, and "cmp" reports files are different
I took "mkfs" and "mkfs.ext3" from server 1, put them on server 2, and got an improvement:
40 GB 15 sec 3 min
Just out of curiousity, I ran "rpm -V e2fsprogs" to see if rpm would pick up the difference and sure enough, it complained mkfs.ext3 changed.
Both of these servers have CentOS 5.4; Server 1 was installed in Dec 2009, Server 2 in April 2010.
I'd like to:
(a) account for the difference in the binaries, and (b) see if something else is different that I can make the same to get the mkfs.ext3 time down to 15 sec on both systems.
Solving (a) should shed light on (b). Any ideas?
---- I think the answer is in your original statement - 'two similar servers'
Similar says, not exactly and the hard drives, controller and RAID configuration differences would be the first place to look for your answer.
Craig
On 06/29/10 6:17 PM, Craig White wrote:
Similar says, not exactly and the hard drives, controller and RAID configuration differences would be the first place to look for your answer.
indeed, first place I'd look would be for things like battery backed raid controllers with/without writeback caching.
Aleksey Tsalolikhin atsaloli.tech@gmail.com wrote:
(a) account for the difference in the binaries, and (b) see if something else is different that I can make the same to get the mkfs.ext3 time down to 15 sec on both systems. Solving (a) should shed light on (b). Any ideas?
Look into prelinking (man prelink). A prelinker from /etc/cron.daily that changes the binaries with an aim to speed up execution.
On Wednesday 30 June 2010, Spiro Harvey wrote:
Aleksey Tsalolikhin atsaloli.tech@gmail.com wrote:
(a) account for the difference in the binaries, and (b) see if something else is different that I can make the same to get the mkfs.ext3 time down to 15 sec on both systems. Solving (a) should shed light on (b). Any ideas?
Look into prelinking (man prelink). A prelinker from /etc/cron.daily that changes the binaries with an aim to speed up execution.
While prelinking would give you different checksums for the same binary on different servers it would not show up in "rpm -V" as reported. This since rpms checksumming is prelink-aware.
/Peter
On Wed, Jun 30, 2010 at 1:48 AM, Peter Kjellstrom cap@nsc.liu.se wrote:
On Wednesday 30 June 2010, Spiro Harvey wrote:
Aleksey Tsalolikhin atsaloli.tech@gmail.com wrote:
(a) account for the difference in the binaries, and (b) see if something else is different that I can make the same to get the mkfs.ext3 time down to 15 sec on both systems. Solving (a) should shed light on (b). Any ideas?
Look into prelinking (man prelink). A prelinker from /etc/cron.daily that changes the binaries with an aim to speed up execution.
Yes, actually the full "rpm -V" message mentions prelink:
[root@server2 ~]# rpm -V e2fsprogs prelink: /sbin/mkfs.ext3: at least one of file's dependencies has changed since prelinking S.?....T /sbin/mkfs.ext3 [root@server2 ~]#
I will RTFM on prelink. Thank you, Spiro, all!
Aleksey
I read up on "prelink" as suggested; and used "ldd /sbin/mkfs.ext3" to see what the dependencies (libraries) are.
There are 13 dependencies; file size is the same between servers but md5sum's are different!
Most of these libraries have other libraries they call; I finally drilled down to ld-2.5.so which is statically built.
Same thing: same file size, same datestamp, same package version; but the binary is actually different; yet rpm -V does not complain. Why?
(Both systems are running CentOS 5.4; one was deployed in December 2009, the other in April 2010.)
[root@server1 /lib64]# ls -l ld-2.5.so -rwxr-xr-x 1 root root 139416 Sep 2 2009 ld-2.5.so [root@server1 /lib64]# md5sum ld-2.5.so ad38c69452b3990852c0d3e0ea51a31b ld-2.5.so [root@server1 /lib64]# ldd ld-2.5.so statically linked [root@server1 /lib64]# rpm -q -f /lib64/ld-2.5.so glibc-2.5-42 [root@server1 /lib64]# rpm -V glibc [root@server1 /lib64]#
[root@server2 /lib64]# ls -l ld-2.5.so -rwxr-xr-x 1 root root 139416 Sep 2 2009 ld-2.5.so [root@server2 /lib64]# md5sum ld-2.5.so ddb5ad336c3cf40ee2c69b91ef7bfd04 ld-2.5.so [root@server2 /lib64]# ldd ld-2.5.so statically linked [root@server2 /lib64]# [root@server2 /lib64]# rpm -q -f /lib64/ld-2.5.so glibc-2.5-42 [root@server2 /lib64]# rpm -V glibc-2.5-42 [root@server2 /lib64]#
i think you'll need to re-read the man pages on prelink. specifically, the -y or --md5 or --sha options. that is essentially what rpm -V does, it does an "undo" of the prelink to verify the original binary file's hash; which will be the same for the same version of software from the same package. doing an md5sum/sha1sum on prelinked binaries is meaningless now.
i know this whole 'prelink' thing throws people off the first time they encounter it. especially if you're doing computer forensics and you haven't been made aware of this, it'll drive you nuts until you understand prelink. personally speaking, i think this 'optimization' comes at a cost, but one eventually gets use to it.
hope that helps... -Bond
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Aleksey Tsalolikhin Sent: Wednesday, June 30, 2010 6:00 PM To: CentOS mailing list Subject: Re: [CentOS] How can binaries be different when package versions are identical? (mkfs.ext3 on CentOS 5.4)
I read up on "prelink" as suggested; and used "ldd /sbin/mkfs.ext3" to see what the dependencies (libraries) are.
There are 13 dependencies; file size is the same between servers but md5sum's are different!
Most of these libraries have other libraries they call; I finally drilled down to ld-2.5.so which is statically built.
Same thing: same file size, same datestamp, same package version; but the binary is actually different; yet rpm -V does not complain. Why?
(Both systems are running CentOS 5.4; one was deployed in December 2009, the other in April 2010.)
[root@server1 /lib64]# ls -l ld-2.5.so -rwxr-xr-x 1 root root 139416 Sep 2 2009 ld-2.5.so [root@server1 /lib64]# md5sum ld-2.5.so ad38c69452b3990852c0d3e0ea51a31b ld-2.5.so [root@server1 /lib64]# ldd ld-2.5.so statically linked [root@server1 /lib64]# rpm -q -f /lib64/ld-2.5.so glibc-2.5-42 [root@server1 /lib64]# rpm -V glibc [root@server1 /lib64]#
[root@server2 /lib64]# ls -l ld-2.5.so -rwxr-xr-x 1 root root 139416 Sep 2 2009 ld-2.5.so [root@server2 /lib64]# md5sum ld-2.5.so ddb5ad336c3cf40ee2c69b91ef7bfd04 ld-2.5.so [root@server2 /lib64]# ldd ld-2.5.so statically linked [root@server2 /lib64]# [root@server2 /lib64]# rpm -q -f /lib64/ld-2.5.so glibc-2.5-42 [root@server2 /lib64]# rpm -V glibc-2.5-42 [root@server2 /lib64]# _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Aleksey Tsalolikhin -
Bond Masuda -
Craig White -
John R Pierce -
Peter Kjellstrom -
Spiro Harvey