[CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

[Kelly Lesperance]

The HBA is an HP H220.

We haven’t really benchmarked individual drives – all 12 drives are utilized in one RAID-10 array, I’m unsure how we would test individual drives without breaking the array.  

Trying ‘hdparm -tT /dev/sda’ now – it’s been running for 25 minutes so far… 

Kelly

On 2016-05-25, 2:12 PM, "centos-bounces at centos.org on behalf of Dennis Jacobfeuerborn" <centos-bounces at centos.org on behalf of dennisml at conversis.de> wrote:

>What is the HBA the drives are attached to?
>Have you done a quick benchmark on a single disk to check if this is a
>raid problem or further down the stack?
>
>Regards,
>  Dennis
>
>On 25.05.2016 19:26, Kelly Lesperance wrote:
>> [merging]
>> 
>> The HBA the drives are attached to has no configuration that I’m aware of.  We would have had to accidentally change 23 of them ☺
>> 
>> Thanks,
>> 
>> Kelly
>> 
>> On 2016-05-25, 1:25 PM, "Kelly Lesperance" <klesperance at blackberry.com> wrote:
>> 
>>> They are:
>>>
>>> [root at r1k1 ~] # hdparm -I /dev/sda
>>>
>>> /dev/sda:
>>>
>>> ATA device, with non-removable media
>>> 	Model Number:       MB4000GCWDC                             
>>> 	Serial Number:      S1Z06RW9            
>>> 	Firmware Revision:  HPGD    
>>> 	Transport:          Serial, SATA Rev 3.0
>>>
>>> Thanks,
>>>
>>> Kelly
>> 
>> 
>> On 2016-05-25, 1:23 PM, "centos-bounces at centos.org on behalf of m.roth at 5-cent.us" <centos-bounces at centos.org on behalf of m.roth at 5-cent.us> wrote:
>> 
>>> Kelly Lesperance wrote:
>>>> I’ve posted this on the forums at
>>>> https://www.centos.org/forums/viewtopic.php?f=47&t=57926&p=244614#p244614
>>>> - posting to the list in the hopes of getting more eyeballs on it.
>>>>
>>>> We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs:
>>>>
>>>> 2x E5-2650
>>>> 128 GB RAM
>>>> 12 x 4 TB 7200 RPM SATA drives connected to an HP H220 HBA
>>>> Dual port 10 GB NIC
>>>>
>>>> The drives are configured as one large RAID-10 volume with mdadm,
>>>> filesystem is XFS. The OS is not installed on the drive - we PXE boot a
>>>> CentOS image we've built with minimal packages installed, and do the OS
>>>> configuration via puppet. Originally, the hosts were running CentOS 6.5,
>>>> with Kafka 0.8.1, without issue. We recently upgraded to CentOS 7.2 and
>>>> Kafka 0.9, and that's when the trouble started.
>>> <SNIP>
>>> One more stupid question: could the configuration of the card for how the
>>> drives are accessed been accidentally changed?
>>>
>>>          mark
>>>
>>> _______________________________________________
>>> CentOS mailing list
>>> CentOS at centos.org
>>> https://lists.centos.org/mailman/listinfo/centos
>> 
>> _______________________________________________
>> CentOS mailing list
>> CentOS at centos.org
>> https://lists.centos.org/mailman/listinfo/centos
>> 
>
>_______________________________________________
>CentOS mailing list
>CentOS at centos.org
>https://lists.centos.org/mailman/listinfo/centos