I am still running CentOS 4.6 on our production systems, but I am
starting to plan the upgrade to CentOS 5.1. I have one test system
running 5.1 that is the exact same hardware configuration as my 4.6
test system. One of our builds runs about 6 times slower on the 5.1
system, even though is uses less overall CPU time. I first suspected
something wrong with the disk, but the results from bonnie++ show
that the 5.1 system is slightly faster:
Version 1.03 ------Sequential Output------ --Sequential
Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
centos4.6 16G 35933 10 21301 5
46507 6 41.8 0
Version 1.03 ------Sequential Output------ --Sequential
Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
centos5.1 16G 42015 14 21179 5
49863 4 91.6 0
Then I ran the build with "/usr/bin/time --verbose", and here are the
results (first 4.6 then 5.1):
Command being timed: "make"
User time (seconds): 32.15
System time (seconds): 3.52
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:35.88
Command being timed: "make"
User time (seconds): 22.05
System time (seconds): 3.11
Percent of CPU this job got: 11%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:31.35
As you can see from the above, there is a lot of idle time on the 5.1
system. Finally, I ran the build with "strace -c", and here are the
top ten lines of that output (again, 4.6 first and then 5.1):
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
53.81 16.804147 54916 306 58 waitpid
34.75 10.853461 82851 131 wait4
5.29 1.650844 9 177706 154581 open
1.61 0.503701 15 34408 read
0.91 0.283706 15 18607 write
0.60 0.185894 12 14919 10364 stat64
0.52 0.163340 10 16495 9079 access
0.47 0.146933 7 20581 mmap2
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
60.07 15.173924 52687 288 58 waitpid
38.50 9.724412 83831 116 wait4
0.54 0.135194 7 19199 10705 access
0.36 0.090850 54 1681 1334 execve
0.27 0.067686 5 14423 10570 stat64
0.11 0.027676 1 24832 read
0.09 0.022339 0 155810 135765 open
0.03 0.007617 159 48 unlink
Any suggestions as to what could possible be causing this? I am
fresh out of other ideas to try.
Alfred