I have a handful of new systems where I've seen unexpectedly low disk performance on an Avago SAS controller, when using CentOS 7. It looked like a regression, so I installed CentOS 6 on one of them and reloaded CentOS 7 on the other. Immediately after install, a difference is apparent in the RAID rebuild speed. The CentOS 6 system is initializing its software RAID5 array at somewhere around 120MB/s, while the CentOS 7 system is rebuilding at around 55MB/s.
It took a while to get an older system installed, and I wasn't able to boot CentOS 6 under UEFI, to that's one difference between the systems. I also set the elevator on CentOS 7 to cfq, to match the CentOS 6 system. That didn't have any apparent effect.
Other than that, the kernel reports "MSI-X vectors supported: 96" on C6, but "MSI-X vectors supported: 8" on C7. I don't know if that's a difference that would affect performance.
Does anyone have experience with these controllers? Any pointers while I continue tracking this down?
Output from lspci follows, as well as dmesg output regarding the MPT controller and the RAID array status on C6 and C7.
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: LSI Logic / Symbios Logic Device 30e0 Physical Slot: 1 Flags: bus master, fast devsel, latency 0, IRQ 24 I/O ports at e000 [size=256] Memory at fb240000 (64-bit, non-prefetchable) [size=64K] Memory at fb200000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at fb100000 [disabled] [size=1M] Capabilities: [50] Power Management version 3 Capabilities: [68] Express Endpoint, MSI 00 Capabilities: [d0] Vital Product Data Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [c0] MSI-X: Enable+ Count=96 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [1e0] #19 Capabilities: [1c0] Power Budgeting <?> Capabilities: [190] #16 Capabilities: [148] Alternative Routing-ID Interpretation (ARI) Kernel driver in use: mpt3sas
CentOS 6:
# dmesg | grep -i mpt mpt3sas version 09.102.00.00-rh loaded mpt3sas 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26 mpt3sas 0000:01:00.0: setting latency timer to 64 mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32772632 kB) mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 12, max_msix_vectors: -1 mpt3sas 0000:01:00.0: irq 60 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 61 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 62 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 63 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 64 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 65 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 66 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 67 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 68 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 69 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 70 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 71 for MSI/MSI-X mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 60 mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 61 mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 62 mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 63 mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 64 mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 65 mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 66 mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 67 mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 68 mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 69 mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 70 mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 71 mpt3sas_cm0: iomem(0x00000000fb240000), mapped(0xffffc90013ee0000), size(65536) mpt3sas_cm0: ioport(0x000000000000e000), size(256) mpt3sas_cm0: sending message unit reset !! mpt3sas_cm0: message unit reset: SUCCESS mpt3sas_cm0: Allocated physical memory: size(17971 kB) mpt3sas_cm0: Current Controller Queue Depth(9979),Max Controller Queue Depth(10240) mpt3sas_cm0: Scatter Gather Elements per IO(128) mpt3sas_cm0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00) mpt3sas_cm0: Protocol=( scsi0 : Fusion MPT SAS Host mpt3sas_cm0: sending port enable !! mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aee5510), phys(8) mpt3sas_cm0: port enable: SUCCESS
# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md0 : active raid1 sda2[0] sdb2[1] 999360 blocks super 1.0 [2/2] [UU]
md1 : active raid5 sdf3[6] sde3[4] sdd3[3] sdc3[2] sda3[0] sdb3[1] 19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] [====>................] recovery = 22.1% (864374932/3905691648) finish=417.0min speed=121538K/sec bitmap: 1/30 pages [4KB], 65536KB chunk
unused devices: <none>
CentOS 7:
# dmesg | grep -i mpt mpt3sas version 04.100.00.00 loaded mpt3sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32674176 kB) mpt3sas0: MSI-X vectors supported: 8, no of cores: 12, max_msix_vectors: 8 mpt3sas 0000:01:00.0: irq 45 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 46 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 47 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 49 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 50 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 51 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 52 for MSI/MSI-X mpt3sas 0000:01:00.0: irq 53 for MSI/MSI-X mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 45 mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 46 mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 47 mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 49 mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 50 mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 51 mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 52 mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 53 mpt3sas0: iomem(0x00000000fb240000), mapped(0xffffc90013180000), size(65536) mpt3sas0: ioport(0x000000000000e000), size(256) mpt3sas0: sending message unit reset !! mpt3sas0: message unit reset: SUCCESS mpt3sas0: Allocated physical memory: size(17329 kB) mpt3sas0: Current Controller Queue Depth(9979),Max Controller Queue Depth(10240) mpt3sas0: Scatter Gather Elements per IO(128) mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00) mpt3sas0: Protocol=( scsi host0: Fusion MPT SAS Host mpt3sas0: sending port enable !! mpt3sas0: host_add: handle(0x0001), sas_addr(0x500605b00aee5ca0), phys(8) mpt3sas0: port enable: SUCCESS
# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md125 : active raid1 sda1[0] sdb1[1] 194496 blocks super 1.0 [2/2] [UU]
md126 : active raid5 sdf3[6] sda3[0] sde3[4] sdd3[3] sdc3[2] sdb3[1] 19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] [=========>...........] recovery = 47.6% (1862704880/3905691648) finish=569.6min speed=59772K/sec bitmap: 2/30 pages [8KB], 65536KB chunk
md127 : active raid1 sda2[0] sdb2[1] 999360 blocks super 1.0 [2/2] [UU]
unused devices: <none>