Simon Banton wrote:
At 13:35 -0400 24/9/07, Ross S. W. Walker wrote:
Ok, so here is the command I would use:
Thanks - here are the results (tried CentOS 4.5 and RHEL5, with tests on sdb when configured as both RAID 0 and as RAID 1):
Sequential reads: disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 300 -r /dev/sdX
CentOS 4.5, RAID 0: | 2007/09/25-14:26:58 | STAT | 13944 | v1.2.8 | /dev/sdb | Total read throughput: 50249728.0B/s (47.92MB/s), IOPS 12268.0/s. | 2007/09/25-14:26:58 | END | 13944 | v1.2.8 | /dev/sdb | Test Done (Passed)
Ok, this is a 2 disk RAID0? If so then this is ok, not the fastest config (60MB/s for fast drives) but mid-level SATA performance.
CentOS 4.5, RAID 1: | 2007/09/25-14:20:06 | STAT | 13807 | v1.2.8 | /dev/sdb | Total read throughput: 44994150.4B/s (42.91MB/s), IOPS 10984.9/s. | 2007/09/25-14:20:06 | END | 13807 | v1.2.8 | /dev/sdb | Test Done (Passed)
Statisically equivalent to RAID0, which is a good sign, as it means the 3ware is doing striped reads off a RAID1.
RHEL5, RAID 0: | 2007/09/25-11:07:46 | STAT | 2835 | v1.2.8 | /dev/sdb | Total read throughput: 2405171.2B/s (2.29MB/s), IOPS 587.2/s. | 2007/09/25-11:07:46 | END | 2835 | v1.2.8 | /dev/sdb | Test Done (Passed)
Ok there is a problem here with the driver on RHEL5, are you running the latest version off of 3ware's site?
Can you send the output of a modinfo <driver name>?
RHEL5, RAID 1: | 2007/09/25-11:35:53 | STAT | 3022 | v1.2.8 | /dev/sdb | Total read throughput: 2461696.0B/s (2.35MB/s), IOPS 601.0/s. | 2007/09/25-11:35:53 | END | 3022 | v1.2.8 | /dev/sdb | Test Done (Passed)
Same bad result here too... at least it's consistently bad, definitely points to a bad driver.
Sequential writes: disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 300 -w /dev/sdX
CentOS 4.5, RAID 0: | 2007/09/25-14:28:19 | STAT | 13951 | v1.2.8 | /dev/sdb | Total write throughput: 66150946.1B/s (63.09MB/s), IOPS 16150.1/s. | 2007/09/25-14:28:19 | END | 13951 | v1.2.8 | /dev/sdb | Test Done (Passed)
Good write performance here, the BBU cache is definitely helping.
CentOS 4.5, RAID 1: | 2007/09/25-14:21:52 | STAT | 13815 | v1.2.8 | /dev/sdb | Total write throughput: 53170039.5B/s (50.71MB/s), IOPS 12981.0/s. | 2007/09/25-14:21:52 | END | 13815 | v1.2.8 | /dev/sdb | Test Done (Passed)
Also good write performance here with the BBU cache, RAID1 is going to be slower by nature as it writes twice for each write, but the BBU cache is minimizing the hurt.
RHEL5, RAID 0: | 2007/09/25-11:13:44 | STAT | 2850 | v1.2.8 | /dev/sdb | Total write throughput: 66031616.0B/s (62.97MB/s), IOPS 16121.0/s. | 2007/09/25-11:13:44 | END | 2850 | v1.2.8 | /dev/sdb | Test Done (Passed)
Write performance on RHEL 5 doesn't seem to be affected here, maybe it's only read performance, maybe the BBU cache is hiding the problem.
RHEL5, RAID 1: | 2007/09/25-11:36:36 | STAT | 3031 | v1.2.8 | /dev/sdb | Total write throughput: 56870229.3B/s (54.24MB/s), IOPS 13884.3/s. | 2007/09/25-11:36:36 | END | 3031 | v1.2.8 | /dev/sdb | Test Done (Passed)
Same thing here, good write performance.
Random reads: disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 300 -r /dev/sdX
CentOS 4.5, RAID 0: | 2007/09/25-14:28:59 | STAT | 13958 | v1.2.8 | /dev/sdb | Total read throughput: 504217.6B/s (0.48MB/s), IOPS 123.1/s. | 2007/09/25-14:28:59 | END | 13958 | v1.2.8 | /dev/sdb | Test Done (Passed)
And here is where the difference between a 15K drive and a 7200 RPM drive appears, though with RAID0 one would expect to see around 1MB, what chunk size does it use 64K?
CentOS 4.5, RAID 1: | 2007/09/25-14:23:14 | STAT | 13822 | v1.2.8 | /dev/sdb | Total read throughput: 549570.2B/s (0.52MB/s), IOPS 134.2/s. | 2007/09/25-14:23:14 | END | 13822 | v1.2.8 | /dev/sdb | Test Done (Passed)
This is the correct performance of a RAID1 for 7200 RPM drives.
RHEL5, RAID 0: | 2007/09/25-11:16:21 | STAT | 2875 | v1.2.8 | /dev/sdb | Total read throughput: 273612.8B/s (0.26MB/s), IOPS 66.8/s. | 2007/09/25-11:16:21 | END | 2875 | v1.2.8 | /dev/sdb | Test Done (Passed)
This also shows a serious performance degradation here, the numbers should be similar to RHEL 4.5 numbers.
RHEL5, RAID 1: | 2007/09/25-11:39:20 | STAT | 3042 | v1.2.8 | /dev/sdb | Total read throughput: 546816.0B/s (0.52MB/s), IOPS 133.5/s. | 2007/09/25-11:39:20 | END | 3042 | v1.2.8 | /dev/sdb | Test Done (Passed)
This is an oddity, and is inconsistent. I would have expected this number to be low too, but it is showing normal throughput for this configuration. I wouldn't put any faith in this and if you ran it 3 times in a row it will probably post slow numbers 2 out of the 3 times.
Random writes: disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 300 -w /dev/sdX
CentOS 4.5, RAID 0: | 2007/09/25-14:29:34 | STAT | 13965 | v1.2.8 | /dev/sdb | Total write throughput: 1379532.8B/s (1.32MB/s), IOPS 336.8/s. | 2007/09/25-14:29:34 | END | 13965 | v1.2.8 | /dev/sdb | Test Done (Passed)
Ok, well the write-back cache is helping here as it should, without it these numbers would be around 320KB/s
CentOS 4.5, RAID 1: | 2007/09/25-14:24:15 | STAT | 13829 | v1.2.8 | /dev/sdb | Total write throughput: 782199.5B/s (0.75MB/s), IOPS 191.0/s. | 2007/09/25-14:24:15 | END | 13829 | v1.2.8 | /dev/sdb | Test Done (Passed)
Also good write-cache performance for this config, it would probably be around 160KB/s without it.
RHEL5, RAID 0: | 2007/09/25-11:19:21 | STAT | 2894 | v1.2.8 | /dev/sdb | Total write throughput: 1377894.4B/s (1.31MB/s), IOPS 336.4/s. | 2007/09/25-11:19:21 | END | 2894 | v1.2.8 | /dev/sdb | Test Done (Passed)
Statistically the same as the RHEL 4.5 numbers.
RHEL5 RAID 1: | 2007/09/25-11:40:08 | STAT | 3049 | v1.2.8 | /dev/sdb | Total write throughput: 798310.4B/s (0.76MB/s), IOPS 194.9/s. | 2007/09/25-11:40:08 | END | 3049 | v1.2.8 | /dev/sdb | Test Done (Passed)
Same here.
I'm not sure what to make of it, mind you.
Well bottom line, there is something very wrong with the 3ware drivers on the RHEL 5 implementation.
I recommend making sure they are the latest drivers from the manufacturer and if they are, base your implementation on RHEL 4.5.
Post the modinfo <driver name> to the list just in case somebody else knows of any issues with the version you are running.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.