I have a handful of new systems where I've seen unexpectedly low disk performance on an Avago SAS controller, when using CentOS 7. It looked like a regression, so I installed CentOS 6 on one of them and reloaded CentOS 7 on the other. Immediately after install, a difference is apparent in the RAID rebuild speed. The CentOS 6 system is initializing its software RAID5 array at somewhere around 120MB/s, while the CentOS 7 system is rebuilding at around 55MB/s.
It took a while to get an older system installed, and I wasn't able to boot CentOS 6 under UEFI, to that's one difference between the systems. I also set the elevator on CentOS 7 to cfq, to match the CentOS 6 system. That didn't have any apparent effect.
Other than that, the kernel reports "MSI-X vectors supported: 96" on C6, but "MSI-X vectors supported: 8" on C7. I don't know if that's a difference that would affect performance.
Does anyone have experience with these controllers? Any pointers while I continue tracking this down?
Output from lspci follows, as well as dmesg output regarding the MPT controller and the RAID array status on C6 and C7.
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: LSI Logic / Symbios Logic Device 30e0 Physical Slot: 1 Flags: bus master, fast devsel, latency 0, IRQ 24 I/O ports at e000 [size=256] Memory at fb240000 (64-bit, non-prefetchable) [size=64K] Memory at fb200000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at fb100000 [disabled] [size=1M] Capabilities: [50] Power Management version 3 Capabilities: [68] Express Endpoint, MSI 00 Capabilities: [d0] Vital Product Data Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [c0] MSI-X: Enable+ Count=96 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [1e0] #19 Capabilities: [1c0] Power Budgeting <?> Capabilities: [190] #16 Capabilities: [148] Alternative Routing-ID Interpretation (ARI) Kernel driver in use: mpt3sas
CentOS 6:
# dmesg | grep -i mpt mpt3sas version 09.102.00.00-rh loaded mpt3sas 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26 mpt3sas 0000:01:00.0: setting latency timer to 64 mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32772632 kB) mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 12, max_msix_vectors: -1 mpt3sas 0000:01:00.0: irq 60 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 61 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 62 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 63 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 64 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 65 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 66 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 67 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 68 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 69 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 70 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 71 for MSI/MSI-X mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 60 mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 61 mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 62 mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 63 mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 64 mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 65 mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 66 mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 67 mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 68 mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 69 mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 70 mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 71 mpt3sas_cm0: iomem(0x00000000fb240000), mapped(0xffffc90013ee0000), size(65536) mpt3sas_cm0: ioport(0x000000000000e000), size(256) mpt3sas_cm0: sending message unit reset !! mpt3sas_cm0: message unit reset: SUCCESS mpt3sas_cm0: Allocated physical memory: size(17971 kB) mpt3sas_cm0: Current Controller Queue Depth(9979),Max Controller Queue Depth(10240) mpt3sas_cm0: Scatter Gather Elements per IO(128) mpt3sas_cm0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00) mpt3sas_cm0: Protocol=( scsi0 : Fusion MPT SAS Host mpt3sas_cm0: sending port enable !! mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aee5510), phys(8) mpt3sas_cm0: port enable: SUCCESS
# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md0 : active raid1 sda2[0] sdb2[1] 999360 blocks super 1.0 [2/2] [UU]
md1 : active raid5 sdf3[6] sde3[4] sdd3[3] sdc3[2] sda3[0] sdb3[1] 19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] [====>................] recovery = 22.1% (864374932/3905691648) finish=417.0min speed=121538K/sec bitmap: 1/30 pages [4KB], 65536KB chunk
unused devices: <none>
CentOS 7:
# dmesg | grep -i mpt mpt3sas version 04.100.00.00 loaded mpt3sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32674176 kB) mpt3sas0: MSI-X vectors supported: 8, no of cores: 12, max_msix_vectors: 8 mpt3sas 0000:01:00.0: irq 45 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 46 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 47 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 49 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 50 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 51 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 52 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 53 for MSI/MSI-X mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 45 mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 46 mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 47 mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 49 mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 50 mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 51 mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 52 mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 53 mpt3sas0: iomem(0x00000000fb240000), mapped(0xffffc90013180000), size(65536) mpt3sas0: ioport(0x000000000000e000), size(256) mpt3sas0: sending message unit reset !! mpt3sas0: message unit reset: SUCCESS mpt3sas0: Allocated physical memory: size(17329 kB) mpt3sas0: Current Controller Queue Depth(9979),Max Controller Queue Depth(10240) mpt3sas0: Scatter Gather Elements per IO(128) mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00) mpt3sas0: Protocol=( scsi host0: Fusion MPT SAS Host mpt3sas0: sending port enable !! mpt3sas0: host_add: handle(0x0001), sas_addr(0x500605b00aee5ca0), phys(8) mpt3sas0: port enable: SUCCESS
# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md125 : active raid1 sda1[0] sdb1[1] 194496 blocks super 1.0 [2/2] [UU]
md126 : active raid5 sdf3[6] sda3[0] sde3[4] sdd3[3] sdc3[2] sdb3[1] 19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] [=========>...........] recovery = 47.6% (1862704880/3905691648) finish=569.6min speed=59772K/sec bitmap: 2/30 pages [8KB], 65536KB chunk
md127 : active raid1 sda2[0] sdb2[1] 999360 blocks super 1.0 [2/2] [UU]
unused devices: <none>
On Nov 4, 2016, at 11:08 PM, Gordon Messmer gordon.messmer@gmail.com wrote:
LSI Logic / Symbios Logic SAS3008
What firmware is/are on the cards? The driver version difference between C6 and C7 could interact badly with old firmware. What firmware does $vendor suggest?
Does C7 in BIOS mode work any better?
Run `lsiget` and compare lsi tech support bundles.
On 11/05/2016 06:40 AM, Steven Tardy wrote:
What firmware is/are on the cards?
In the dmesg output I included: mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00)
The driver version difference between C6 and C7 could interact badly with old firmware.
I suspected that was the case, and tried running elrepo's "kmod-mpt3sas" from their testing repo. C7 has version 04.100.00.00, while elrepo's appears to be 13.100.00.00. That didn't change anything.
What firmware does $vendor suggest?
Haven't asked yet, since the same firmware produces different behavior on the two OSs.
Does C7 in BIOS mode work any better?
That'll be one of my next tests.
Run `lsiget` and compare lsi tech support bundles.
I'll look around for that, too. No vendor tools are installed, currently.
On 11/5/2016 12:08 PM, Gordon Messmer wrote:
On 11/05/2016 06:40 AM, Steven Tardy wrote:
What firmware is/are on the cards?
In the dmesg output I included: mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00)
The driver version difference between C6 and C7 could interact badly with old firmware.
I suspected that was the case, and tried running elrepo's "kmod-mpt3sas" from their testing repo. C7 has version 04.100.00.00, while elrepo's appears to be 13.100.00.00. That didn't change anything.
What firmware does $vendor suggest?
Haven't asked yet, since the same firmware produces different behavior on the two OSs.
I know with the SAS2 chips (SAS2008, SAS2308, etc), FreeBSD strongly recommends the firmware revision matches the driver revision, so if the driver is P20, the firmware also should be P20. I don't have any SAS3 stuff.
After running a long series of benchmarks, it looks like there is a very significant performance difference between systems using LVM and systems not using LVM under CentOS 7. I'd appreciate it if anyone else can confirm these results.
There are a couple of other surprising aspects of the results, especially that the default storage configuration (LVM and 512k blocks) appears to be the worst configuration possible, when using RAID5 (and probably RAID6).
https://plus.google.com/+GordonMessmer/posts/H5DuyP1LHPU?sfc=true
For the archive's sake:
https://plus.google.com/+GordonMessmer/posts/H5DuyP1LHPU?sfc=false https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false
During testing, I found that under CentOS 7.2, the default storage layout for software RAID5 was the worst possible configuration. 512k blocks perform noticeably worse than 64k blocks, and LVM had a tremendous impact as well.
After upgrading to CentOS 7.3, which features a newer mpt3sas driver, performance improved significantly. LVM's overhead has decreased to an acceptable level (from my point of view).
Because chunk size has a significant impact on performance, I'm going to file an RFE to change the default chunk size for RAID5 and RAID6 volumes.
-
Gordon Messmer -
John R Pierce -
Steven Tardy