[CentOS] 3Ware 9550SX and latency/system responsiveness

Tue Sep 25 14:36:45 UTC 2007

Simon Banton wrote:
> 
> At 13:35 -0400 24/9/07, Ross S. W. Walker wrote:
> >Ok, so here is the command I would use:
> 
> Thanks - here are the results (tried CentOS 4.5 and RHEL5, with tests 
> on sdb when configured as both RAID 0 and as RAID 1):
> 
> >Sequential reads:
> >disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 300 -r /dev/sdX
> 
> CentOS 4.5, RAID 0:
> | 2007/09/25-14:26:58 | STAT  | 13944 | v1.2.8 | /dev/sdb | Total 
> read throughput: 50249728.0B/s (47.92MB/s), IOPS 12268.0/s.
> | 2007/09/25-14:26:58 | END   | 13944 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Ok, this is a 2 disk RAID0? If so then this is ok, not the fastest
config (60MB/s for fast drives) but mid-level SATA performance.

> CentOS 4.5, RAID 1:
> | 2007/09/25-14:20:06 | STAT  | 13807 | v1.2.8 | /dev/sdb | Total 
> read throughput: 44994150.4B/s (42.91MB/s), IOPS 10984.9/s.
> | 2007/09/25-14:20:06 | END   | 13807 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Statisically equivalent to RAID0, which is a good sign, as it means the
3ware is doing striped reads off a RAID1.

> RHEL5, RAID 0:
> | 2007/09/25-11:07:46 | STAT  | 2835 | v1.2.8 | /dev/sdb | Total read 
> throughput: 2405171.2B/s (2.29MB/s), IOPS 587.2/s.
> | 2007/09/25-11:07:46 | END   | 2835 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Ok there is a problem here with the driver on RHEL5, are you running
the latest version off of 3ware's site?

Can you send the output of a modinfo <driver name>?

> RHEL5, RAID 1:
> | 2007/09/25-11:35:53 | STAT  | 3022 | v1.2.8 | /dev/sdb | Total read 
> throughput: 2461696.0B/s (2.35MB/s), IOPS 601.0/s.
> | 2007/09/25-11:35:53 | END   | 3022 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Same bad result here too... at least it's consistently bad, definitely
points to a bad driver.

> >Sequential writes:
> >disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 300 -w /dev/sdX
> 
> CentOS 4.5, RAID 0:
> | 2007/09/25-14:28:19 | STAT  | 13951 | v1.2.8 | /dev/sdb | Total 
> write throughput: 66150946.1B/s (63.09MB/s), IOPS 16150.1/s.
> | 2007/09/25-14:28:19 | END   | 13951 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Good write performance here, the BBU cache is definitely helping.

> CentOS 4.5, RAID 1:
> | 2007/09/25-14:21:52 | STAT  | 13815 | v1.2.8 | /dev/sdb | Total 
> write throughput: 53170039.5B/s (50.71MB/s), IOPS 12981.0/s.
> | 2007/09/25-14:21:52 | END   | 13815 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Also good write performance here with the BBU cache, RAID1 is
going to be slower by nature as it writes twice for each write,
but the BBU cache is minimizing the hurt.

> RHEL5, RAID 0:
> | 2007/09/25-11:13:44 | STAT  | 2850 | v1.2.8 | /dev/sdb | Total 
> write throughput: 66031616.0B/s (62.97MB/s), IOPS 16121.0/s.
> | 2007/09/25-11:13:44 | END   | 2850 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Write performance on RHEL 5 doesn't seem to be affected here,
maybe it's only read performance, maybe the BBU cache is hiding
the problem.

> RHEL5, RAID 1:
> | 2007/09/25-11:36:36 | STAT  | 3031 | v1.2.8 | /dev/sdb | Total 
> write throughput: 56870229.3B/s (54.24MB/s), IOPS 13884.3/s.
> | 2007/09/25-11:36:36 | END   | 3031 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Same thing here, good write performance.

> >Random reads:
> >disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 300 -r /dev/sdX
> 
> CentOS 4.5, RAID 0:
> | 2007/09/25-14:28:59 | STAT  | 13958 | v1.2.8 | /dev/sdb | Total 
> read throughput: 504217.6B/s (0.48MB/s), IOPS 123.1/s.
> | 2007/09/25-14:28:59 | END   | 13958 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

And here is where the difference between a 15K drive and a 7200
RPM drive appears, though with RAID0 one would expect to see
around 1MB, what chunk size does it use 64K?

> CentOS 4.5, RAID 1:
> | 2007/09/25-14:23:14 | STAT  | 13822 | v1.2.8 | /dev/sdb | Total 
> read throughput: 549570.2B/s (0.52MB/s), IOPS 134.2/s.
> | 2007/09/25-14:23:14 | END   | 13822 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

This is the correct performance of a RAID1 for 7200 RPM drives.

> RHEL5, RAID 0:
> | 2007/09/25-11:16:21 | STAT  | 2875 | v1.2.8 | /dev/sdb | Total read 
> throughput: 273612.8B/s (0.26MB/s), IOPS 66.8/s.
> | 2007/09/25-11:16:21 | END   | 2875 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

This also shows a serious performance degradation here, the numbers
should be similar to RHEL 4.5 numbers.

> RHEL5, RAID 1:
> | 2007/09/25-11:39:20 | STAT  | 3042 | v1.2.8 | /dev/sdb | Total read 
> throughput: 546816.0B/s (0.52MB/s), IOPS 133.5/s.
> | 2007/09/25-11:39:20 | END   | 3042 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

This is an oddity, and is inconsistent. I would have expected this
number to be low too, but it is showing normal throughput for this
configuration. I wouldn't put any faith in this and if you ran it
3 times in a row it will probably post slow numbers 2 out of the 3
times.

> >Random writes:
> >disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 300 -w /dev/sdX
> 
> CentOS 4.5, RAID 0:
> | 2007/09/25-14:29:34 | STAT  | 13965 | v1.2.8 | /dev/sdb | Total 
> write throughput: 1379532.8B/s (1.32MB/s), IOPS 336.8/s.
> | 2007/09/25-14:29:34 | END   | 13965 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Ok, well the write-back cache is helping here as it should,
without it these numbers would be around 320KB/s

> CentOS 4.5, RAID 1:
> | 2007/09/25-14:24:15 | STAT  | 13829 | v1.2.8 | /dev/sdb | Total 
> write throughput: 782199.5B/s (0.75MB/s), IOPS 191.0/s.
> | 2007/09/25-14:24:15 | END   | 13829 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Also good write-cache performance for this config, it would
probably be around 160KB/s without it.

> RHEL5, RAID 0:
> | 2007/09/25-11:19:21 | STAT  | 2894 | v1.2.8 | /dev/sdb | Total 
> write throughput: 1377894.4B/s (1.31MB/s), IOPS 336.4/s.
> | 2007/09/25-11:19:21 | END   | 2894 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Statistically the same as the RHEL 4.5 numbers.

> RHEL5 RAID 1:
> | 2007/09/25-11:40:08 | STAT  | 3049 | v1.2.8 | /dev/sdb | Total 
> write throughput: 798310.4B/s (0.76MB/s), IOPS 194.9/s.
> | 2007/09/25-11:40:08 | END   | 3049 | v1.2.8 | /dev/sdb | 
> Test Done (Passed)

Same here.

> I'm not sure what to make of it, mind you.

Well bottom line, there is something very wrong with the 3ware
drivers on the RHEL 5 implementation.

I recommend making sure they are the latest drivers from the
manufacturer and if they are, base your implementation on RHEL 4.5.

Post the modinfo <driver name> to the list just in case somebody
else knows of any issues with the version you are running.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.