Hi There,
I recently suffered a serious hardware failure on one of my firewalls, (motherboard died), this machine was originally a redhat 9 box, on an Athlon 2200 with 512Megs RAM, this machine was doing the following jobs for a small network at a relatives house:
Firewall (IPTABLES) Transparent proxy (squid) Sendmail smarthost IMAP mail server SAMBA file server IPSEC tunnel to my home.
The only machines I had to replace it with were an Intel SC5000 chassis with twin PIII 1GHz CPUs on SCSI RAID (1 Gig RAM) and a compaq PII 450 (512Megs) software RAID on IDE 40Gig drives.
So I installed Centos 4.2 on both, split the IMAP and SAMBA onto the SC5000 and put the firewall, Squid and sendmail smarthost on the Compaq. The system is connected to the net via a 2Meg Line.
While each of these machines easily copes with the jobs they have to do, I have noticed (or rather my relative noticed and I agreed) that web browsing now has a high latency. i.e. you go to a new web page, there is a substantial pause, before the page starts to load (substantial a few seconds, whereas previously it was instant). DNS is all ok and working fine, there are no delays there. The DSL line shows very little latency from the net and outgoing from the lan.
I note that squid uses a fair amout of CPU on the PII450 from time to time, but can't seem to get a handle on what is causing the delay. The only thing I can think of is that the PII450 running squid is just slow compared to the old Athlon box and I am reaping the "benefits" of that. I do find that hard to believe though, since the box isn't running X or anthing else extra. I also notice that sometimes some downloads, particularly large ones just seem to grind to a halt after a few megs, and sometimes then carry on a bit later if left, right up to full speed of the line. During this slowdown, the latency of the line even with large pings is fine, so I am sure the connection is good.
Is Centos4.2 now so heavy that a PII450 is not enough for a transparent proxy and smarthost....
Any comments/ideas are welcome...
Pete
Peter Farrow peter@farrows.org wrote:
web browsing now has a high latency ... DNS is all ok ... I note that squid uses a fair amout of CPU on the PII450 from time to time, but can't seem to get a handle on what
is
causing the delay.
Could be differences in the disk / disk controller. Squid is very storage intensive, although 512MiB RAM should mitigate much of that.
Run "hdparm -i" and "hdparm" (no options) on each of your IDE disks and check for DMA settings, and what the controller currently set at (the asterisk next to the modes).
The transfer rate from the disks is ok, I added some options in /etc/sysconfig/hardisk after the original install, but found they weren't needed,
[root@ ~]# hdparm -t /dev/hdb
/dev/hdb: Timing buffered disk reads: 62 MB in 3.07 seconds = 20.17 MB/sec [root@ ~]# hdparm -t /dev/hda
/dev/hda: Timing buffered disk reads: 62 MB in 3.02 seconds = 20.51 MB/sec
/dev/hdb: multcount = 16 (on) IO_support = 0 (default 16-bit) unmaskirq = 0 (off) using_dma = 1 (on) keepsettings = 0 (off) readonly = 0 (off) readahead = 256 (on) geometry = 38792/16/63, sectors = 20020396032, start = 0 [root@lionel ~]# hdparm /dev/hda
/dev/hda: multcount = 16 (on) IO_support = 0 (default 16-bit) unmaskirq = 0 (off) using_dma = 1 (on) keepsettings = 0 (off) readonly = 0 (off) readahead = 256 (on) geometry = 38792/16/63, sectors = 20020396032, start = 0
The controller ignores setting IO bit to 32bit , so hdparm -d1 /dev/hda does nothing... and the disks are 20gigs not 40,they are Western Digital WD200 hdparm -i /dev/hda
/dev/hda:
Model=WDC WD200EB-00CPF0, FwRev=06.04G06, SerialNo=WD-WCAAU5009049 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=40 BuffType=DualPortCache, BuffSize=2048kB, MaxMultSect=16, MultSect=16 CurCHS=17475/15/63, CurSects=16513875, LBA=yes, LBAsects=39102336 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 *udma2 udma3 udma4 udma5 AdvancedPM=no WriteCache=enabled Drive conforms to: device does not report version:
* signifies the current active mode hdparm -i /dev/hdb
/dev/hdb:
Model=WDC WD200EB-00CPF0, FwRev=06.04G06, SerialNo=WD-WCAAU5108990 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=40 BuffType=DualPortCache, BuffSize=2048kB, MaxMultSect=16, MultSect=16 CurCHS=17475/15/63, CurSects=16513875, LBA=yes, LBAsects=39102336 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 *udma2 udma3 udma4 udma5 AdvancedPM=no WriteCache=enabled Drive conforms to: device does not report version:
* signifies the current active mode
I'm running software RAID and they are both on one IDE controller, but this is because the installer detects the drive geometry differently for each if they are on separate controller, which would be the best option, but as I am doing mirroring I want the geometry the same. Furthermore no amount of changing in the BIOS affects the detected geometry by Anaconda. I've seen this quite a lot on Compaqs. I have just bought a Compaq motherboard from Ebay with a 933MHz PIII for £12.95 (= 20 bucks), to replace the PII 450 motherboard, so I guess I'll know if it is the CPU in a day or two.
Any other ideas?
Here is top:
top - 23:55:32 up 3 days, 7:46, 2 users, load average: 0.05, 0.03, 0.00 Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie Cpu(s): 1.6% us, 2.3% sy, 0.0% ni, 95.8% id, 0.0% wa, 0.3% hi, 0.0% si Mem: 515708k total, 449256k used, 66452k free, 92000k buffers Swap: 1020024k total, 916k used, 1019108k free, 205364k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3276 root 16 0 7528 2288 1828 S 2.6 0.4 0:01.07 sshd 3466 root 17 0 3464 952 748 R 1.3 0.2 0:00.33 top 1 root 16 0 2708 568 476 S 0.0 0.1 0:01.92 init 2 root 34 19 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/0 3 root 5 -10 0 0 0 S 0.0 0.0 0:01.37 events/0 4 root 5 -10 0 0 0 S 0.0 0.0 0:00.04 khelper 5 root 5 -10 0 0 0 S 0.0 0.0 0:00.03 kblockd/0 6 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush 31 root 15 0 0 0 0 S 0.0 0.0 0:13.89 pdflush 33 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0 28 root 15 0 0 0 0 S 0.0 0.0 0:00.21 kapmd 32 root 15 0 0 0 0 S 0.0 0.0 0:32.34 kswapd0 107 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod 207 root 15 0 0 0 0 S 0.0 0.0 0:00.00 md6_raid1 209 root 15 0 0 0 0 S 0.0 0.0 0:00.72 md1_raid1 211 root 15 0 0 0 0 S 0.0 0.0 0:00.01 md2_raid1 213 root 15 0 0 0 0 S 0.0 0.0 0:00.01 md4_raid1 215 root 15 0 0 0 0 S 0.0 0.0 0:00.40 md5_raid1 217 root 15 0 0 0 0 S 0.0 0.0 0:04.02 md3_raid1 218 root 15 0 0 0 0 S 0.0 0.0 0:00.00 md0_raid1 219 root 15 0 0 0 0 S 0.0 0.0 0:07.55 kjournald
Note that I put X back on the box with VNC in xinetd, since it seems to make no difference to the box performance...
P.
Bryan J. Smith wrote:
Peter Farrow peter@farrows.org wrote:
web browsing now has a high latency ... DNS is all ok ... I note that squid uses a fair amout of CPU on the PII450 from time to time, but can't seem to get a handle on what
is
causing the delay.
Could be differences in the disk / disk controller. Squid is very storage intensive, although 512MiB RAM should mitigate much of that.
Run "hdparm -i" and "hdparm" (no options) on each of your IDE disks and check for DMA settings, and what the controller currently set at (the asterisk next to the modes).
Peter Farrow peter@farrows.org wrote:
[root@ ~]# hdparm -t /dev/hdb /dev/hdb: Timing buffered disk reads: 62 MB in 3.07 seconds = 20.17 MB/sec [root@ ~]# hdparm -t /dev/hda /dev/hda: Timing buffered disk reads: 62 MB in 3.02 seconds = 20.51 MB/sec
Ouch! They are on the same channel! Furthermore, you're only getting the cache speed. Try "-Tt" instead. And try running those 2 commands simultaneously! You're going to see more than a 50% drop (more like an 80%!).
UDMA modes: udma0 udma1 *udma2 udma3 udma4 udma5 UDMA modes: udma0 udma1 *udma2 udma3 udma4 udma5
Ultra DMA mode 2 (33MHz) is a good sign.
I'm running software RAID and they are both on one IDE controller, but this is because the installer detects the drive geometry differently for each if they are on separate controller,
If it's Award BIOS, then set the geometry to "LBA" in the BIOS.
Furthermore, _regardless_ of what the BIOS says, once you partition with LBA, if you move it to another controller, the partition table will _still_ be LBA when Linux loads the partitions.
which would be the best option, but as I am doing mirroring
I
want the geometry the same.
The geometry is _already_ the same once the partition table has been created. Linux _ignores_ the BIOS' geometry if it was partitioned differently.
Furthermore no amount of changing in the BIOS affects the detected geometry by Anaconda. I've seen this quite a lot
on
Compaqs.
Oh, a Compaq. Yeah, broken BIOS.
But _regardless_, you've already got the correct geometry. You can now move channels, Linux will read the partition table and use its geometry -- not what the BIOS says.
ATA DMA was _never_ designed for master/slave, that's an old EIDE PIO configuration. Drives only allow it to be compatible, but it's not recommended at all.
No,
hdparm -t gives you the uncached speed, -T gives you the cached speed: hdparm -Tt /dev/hda
/dev/hda: Timing cached reads: 472 MB in 2.00 seconds = 235.68 MB/sec Timing buffered disk reads: 62 MB in 3.06 seconds = 20.23 MB/sec
hdparm -Tt /dev/hdb
/dev/hdb: Timing cached reads: 480 MB in 2.00 seconds = 239.80 MB/sec Timing buffered disk reads: 62 MB in 3.03 seconds = 20.46 MB/sec
The disks are set the logical block assignments already, and anaconda, detects the number of cylinders differently in the install, which makes mirroring non symmetrical, trust me when on channel 1 and 2, I've checked this in great detail, furthermore, simultaneous requests to both drives isn't affecting performance as much as you imagine, here are the results of a simultaneous hdparm test:
Timing cached reads: Timing cached reads: 272 MB in 2.01 seconds = 135.08 MB/sec 340 MB in 2.01 seconds = 168.76 MB/sec Timing buffered disk reads: Timing buffered disk reads: 46 MB in 3.08 seconds = 14.93 MB/sec 46 MB in 3.08 seconds = 14.92 MB/sec
[5]- Done hdparm -Tt /dev/hda [6]+ Done hdparm -Tt /dev/hdb
Also the old machine had them on the same channel two for the same reason.....
This isn't the issue, its something else.......
P.
Bryan J. Smith wrote:
Peter Farrow peter@farrows.org wrote:
[root@ ~]# hdparm -t /dev/hdb /dev/hdb: Timing buffered disk reads: 62 MB in 3.07 seconds = 20.17 MB/sec [root@ ~]# hdparm -t /dev/hda /dev/hda: Timing buffered disk reads: 62 MB in 3.02 seconds = 20.51 MB/sec
Ouch! They are on the same channel! Furthermore, you're only getting the cache speed. Try "-Tt" instead. And try running those 2 commands simultaneously! You're going to see more than a 50% drop (more like an 80%!).
UDMA modes: udma0 udma1 *udma2 udma3 udma4 udma5 UDMA modes: udma0 udma1 *udma2 udma3 udma4 udma5
Ultra DMA mode 2 (33MHz) is a good sign.
I'm running software RAID and they are both on one IDE controller, but this is because the installer detects the drive geometry differently for each if they are on separate controller,
If it's Award BIOS, then set the geometry to "LBA" in the BIOS.
Furthermore, _regardless_ of what the BIOS says, once you partition with LBA, if you move it to another controller, the partition table will _still_ be LBA when Linux loads the partitions.
which would be the best option, but as I am doing mirroring
I
want the geometry the same.
The geometry is _already_ the same once the partition table has been created. Linux _ignores_ the BIOS' geometry if it was partitioned differently.
Furthermore no amount of changing in the BIOS affects the detected geometry by Anaconda. I've seen this quite a lot
on
Compaqs.
Oh, a Compaq. Yeah, broken BIOS.
But _regardless_, you've already got the correct geometry. You can now move channels, Linux will read the partition table and use its geometry -- not what the BIOS says.
ATA DMA was _never_ designed for master/slave, that's an old EIDE PIO configuration. Drives only allow it to be compatible, but it's not recommended at all.
Peter Farrow peter@farrows.org wrote:
No, hdparm -t gives you the uncached speed, -T gives you the cached speed:
You're right, I just looked at your numbers incorrectly. I was wondering how you got 62MB out of the Ultra33 interface, but didn't see it was for 3 seconds (doh!).
The disks are set the logical block assignments already, and anaconda, detects the number of cylinders differently
in
the install, which makes mirroring non symmetrical, trust
me
when on channel 1 and 2, I've checked this in great detail,
First off, if you use Anaconda with the master/slave on the primary, and the BIOS is LBA, then the partition table _will_ be created with LBA. After that point, if you move the disk to the secondary channel, it _will_ continue to be LBA -- as Linux will read the geometry from the _disk_, not the BIOS.
Secondly, you can _force_ different geometry by pulling up fdisk and entering "x". Then manually set the "h" (heads) to 255, and the reduced "c" (cylinders) appropriately to match. Hit "r" to return to the main menu and "o" to install a new partition table.
_Everyone_ needs to know how to do this -- especially when XP forces 255/63 in conflict with the BIOS on regular occassions. Ironically, this is no longer LBA48 compliant, and heads can exceed 255 now (don't get me started).
furthermore, simultaneous requests to both drives isn't affecting performance as much as you imagine, here are the results of a simultaneous hdparm test: Timing buffered disk reads: Timing buffered disk reads: 46 MB in 3.08 seconds = 14.93 MB/sec 46 MB in 3.08 seconds = 14.92 MB/sec
First off, that's a 25% drop right there!
As I said, you're not only seeing a 50% drop because of the sharing, but additional overhead that could be up to 80% less. I'm sure if you make a sustained set of transfers, it would really kill it. The resetting of the ATA bus continually is part of the problem -- although you're using the same type of disk/mode, so at least it's not as bad as it could be.
(E.g., putting a PIO CD-ROM on the same channel as a DMA hard drive typically results in far more than an 80% loss with all the reset overhead).
Regardless, 15MBps is about 3-4x _slower_ than modern disks.
Also the old machine had them on the same channel two for the same reason..... This isn't the issue, its something else.......
Oh, why didn't you mention the old config was the same?
-- Bryan
P.S. In a world with cheap, off-chipset Ultra33 cards (sub-$10 at many resellers) for additional channels (such as optical drives), why ever put two hard drives on the same channel?
Hi there,
In my experience with 20Gig drives of this age on a machine of this age, 20Megs/sec is about what I would expect....
25% drop is a lot, but its ok....
I know I can force the CHS in fdisk but I don't like doing that, it seems that only some machines have the BIOS issue, putting them on the same channel *always* fixes this issue.....
Since the transfer tate of the interface (i.e. cache test) is more than 10x the transfer rate of the disk, although not optimal by any stretch, I don't really see a problem of having them on the same channel for this application.
I should get the PIII board tomorrow, so I'll report back if it changes anything!
whats the command line switch to do the sustained test for more than the default number of seconds?
P.
Bryan J. Smith wrote:
Peter Farrow peter@farrows.org wrote:
No, hdparm -t gives you the uncached speed, -T gives you the cached speed:
You're right, I just looked at your numbers incorrectly. I was wondering how you got 62MB out of the Ultra33 interface, but didn't see it was for 3 seconds (doh!).
The disks are set the logical block assignments already, and anaconda, detects the number of cylinders differently
in
the install, which makes mirroring non symmetrical, trust
me
when on channel 1 and 2, I've checked this in great detail,
First off, if you use Anaconda with the master/slave on the primary, and the BIOS is LBA, then the partition table _will_ be created with LBA. After that point, if you move the disk to the secondary channel, it _will_ continue to be LBA -- as Linux will read the geometry from the _disk_, not the BIOS.
Secondly, you can _force_ different geometry by pulling up fdisk and entering "x". Then manually set the "h" (heads) to 255, and the reduced "c" (cylinders) appropriately to match. Hit "r" to return to the main menu and "o" to install a new partition table.
_Everyone_ needs to know how to do this -- especially when XP forces 255/63 in conflict with the BIOS on regular occassions. Ironically, this is no longer LBA48 compliant, and heads can exceed 255 now (don't get me started).
furthermore, simultaneous requests to both drives isn't affecting performance as much as you imagine, here are the results of a simultaneous hdparm test: Timing buffered disk reads: Timing buffered disk reads: 46 MB in 3.08 seconds = 14.93 MB/sec 46 MB in 3.08 seconds = 14.92 MB/sec
First off, that's a 25% drop right there!
As I said, you're not only seeing a 50% drop because of the sharing, but additional overhead that could be up to 80% less. I'm sure if you make a sustained set of transfers, it would really kill it. The resetting of the ATA bus continually is part of the problem -- although you're using the same type of disk/mode, so at least it's not as bad as it could be.
(E.g., putting a PIO CD-ROM on the same channel as a DMA hard drive typically results in far more than an 80% loss with all the reset overhead).
Regardless, 15MBps is about 3-4x _slower_ than modern disks.
Also the old machine had them on the same channel two for the same reason..... This isn't the issue, its something else.......
Oh, why didn't you mention the old config was the same?
-- Bryan
P.S. In a world with cheap, off-chipset Ultra33 cards (sub-$10 at many resellers) for additional channels (such as optical drives), why ever put two hard drives on the same channel?
BTW have you seen this before:
P.
Bryan J. Smith wrote:
Peter Farrow peter@farrows.org wrote:
No, hdparm -t gives you the uncached speed, -T gives you the cached speed:
You're right, I just looked at your numbers incorrectly. I was wondering how you got 62MB out of the Ultra33 interface, but didn't see it was for 3 seconds (doh!).
The disks are set the logical block assignments already, and anaconda, detects the number of cylinders differently
in
the install, which makes mirroring non symmetrical, trust
me
when on channel 1 and 2, I've checked this in great detail,
First off, if you use Anaconda with the master/slave on the primary, and the BIOS is LBA, then the partition table _will_ be created with LBA. After that point, if you move the disk to the secondary channel, it _will_ continue to be LBA -- as Linux will read the geometry from the _disk_, not the BIOS.
Secondly, you can _force_ different geometry by pulling up fdisk and entering "x". Then manually set the "h" (heads) to 255, and the reduced "c" (cylinders) appropriately to match. Hit "r" to return to the main menu and "o" to install a new partition table.
_Everyone_ needs to know how to do this -- especially when XP forces 255/63 in conflict with the BIOS on regular occassions. Ironically, this is no longer LBA48 compliant, and heads can exceed 255 now (don't get me started).
furthermore, simultaneous requests to both drives isn't affecting performance as much as you imagine, here are the results of a simultaneous hdparm test: Timing buffered disk reads: Timing buffered disk reads: 46 MB in 3.08 seconds = 14.93 MB/sec 46 MB in 3.08 seconds = 14.92 MB/sec
First off, that's a 25% drop right there!
As I said, you're not only seeing a 50% drop because of the sharing, but additional overhead that could be up to 80% less. I'm sure if you make a sustained set of transfers, it would really kill it. The resetting of the ATA bus continually is part of the problem -- although you're using the same type of disk/mode, so at least it's not as bad as it could be.
(E.g., putting a PIO CD-ROM on the same channel as a DMA hard drive typically results in far more than an 80% loss with all the reset overhead).
Regardless, 15MBps is about 3-4x _slower_ than modern disks.
Also the old machine had them on the same channel two for the same reason..... This isn't the issue, its something else.......
Oh, why didn't you mention the old config was the same?
-- Bryan
P.S. In a world with cheap, off-chipset Ultra33 cards (sub-$10 at many resellers) for additional channels (such as optical drives), why ever put two hard drives on the same channel?
By any change do you have a speed/duplex mismatch? What is the output of mii-tool <device name> or ethtool <device name>?
Do you have a hub or a switch? What speeds does it support?
You can also do point to point bandwidth testing with Iperf http://dast.nlanr.net/Projects/Iperf/
Hope this helps.
Barry
I upgraded the motherboard to a 933 MHz PIII, same ram and disk arrangement,
Problem went away...
Seems a PII 450 can't cut it on Centos 4.2.... but it is an old machine anyway... P.
Bryan J. Smith wrote:
Peter Farrow peter@farrows.org wrote:
web browsing now has a high latency ... DNS is all ok ... I note that squid uses a fair amout of CPU on the PII450 from time to time, but can't seem to get a handle on what
is
causing the delay.
Could be differences in the disk / disk controller. Squid is very storage intensive, although 512MiB RAM should mitigate much of that.
Run "hdparm -i" and "hdparm" (no options) on each of your IDE disks and check for DMA settings, and what the controller currently set at (the asterisk next to the modes).
Peter Farrow wrote:
I upgraded the motherboard to a 933 MHz PIII, same ram and disk arrangement,
Problem went away...
Seems a PII 450 can't cut it on Centos 4.2.... but it is an old machine anyway...
That's good news. I have some older P3-733/512mbRAM 1RU machines that were lightly used that are being replaced. I was a bit nervous about deploying a couple of them as firewall machines, but I suspect they'll still outperform an ancient Cisco PIX 520 for satellite office duty.
Cheers,
Hi Chris,
The biggest beef was the fact that Squid would just seem to stall on some downloads, for example MS updates on the client machines would all halt consistently after doing about 10% of the latest .net udpates. Clear the cache on squid and empty the browser caches, clear it all out and start again, and it still did it everytime.
Changed the motherboard to a faster one and the problem goes away....bizarre to say the least....
Load average on the machine was always low (<.2 )
The 933 PIII with the sames disks (just transplanted) and the same network cards and RAM, works fine, the PII 450 on Red Hat 9 also works fine. It might be a 2.6 kernel issue with PIIs maybe (a bit far fetched I agree but possible)
With the PIII installed its like the internet connection has 1/10 the latency...
P.
Chris Mauritz wrote:
Peter Farrow wrote:
I upgraded the motherboard to a 933 MHz PIII, same ram and disk arrangement,
Problem went away...
Seems a PII 450 can't cut it on Centos 4.2.... but it is an old machine anyway...
That's good news. I have some older P3-733/512mbRAM 1RU machines that were lightly used that are being replaced. I was a bit nervous about deploying a couple of them as firewall machines, but I suspect they'll still outperform an ancient Cisco PIX 520 for satellite office duty.
Cheers,
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Barry Brimer -
Bryan J. Smith -
Chris Mauritz -
Peter Farrow