[CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Wed May 25 19:13:37 UTC 2016
Kelly Lesperance <klesperance at blackberry.com>

Hdparm didn’t get far:

[root at r1k1 ~] # hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   Alarm clock
[root at r1k1 ~] #

On 2016-05-25, 2:44 PM, "Kelly Lesperance" <klesperance at blackberry.com> wrote:

>The HBA is an HP H220.
>
>We haven’t really benchmarked individual drives – all 12 drives are utilized in one RAID-10 array, I’m unsure how we would test individual drives without breaking the array.  
>
>Trying ‘hdparm -tT /dev/sda’ now – it’s been running for 25 minutes so far… 
>
>Kelly
>
>On 2016-05-25, 2:12 PM, "centos-bounces at centos.org on behalf of Dennis Jacobfeuerborn" <centos-bounces at centos.org on behalf of dennisml at conversis.de> wrote:
>
>>What is the HBA the drives are attached to?
>>Have you done a quick benchmark on a single disk to check if this is a
>>raid problem or further down the stack?
>>
>>Regards,
>>  Dennis
>>
>>On 25.05.2016 19:26, Kelly Lesperance wrote:
>>> [merging]
>>> 
>>> The HBA the drives are attached to has no configuration that I’m aware of.  We would have had to accidentally change 23 of them ☺
>>> 
>>> Thanks,
>>> 
>>> Kelly
>>> 
>>> On 2016-05-25, 1:25 PM, "Kelly Lesperance" <klesperance at blackberry.com> wrote:
>>> 
>>>> They are:
>>>>
>>>> [root at r1k1 ~] # hdparm -I /dev/sda
>>>>
>>>> /dev/sda:
>>>>
>>>> ATA device, with non-removable media
>>>> 	Model Number:       MB4000GCWDC                             
>>>> 	Serial Number:      S1Z06RW9            
>>>> 	Firmware Revision:  HPGD    
>>>> 	Transport:          Serial, SATA Rev 3.0
>>>>
>>>> Thanks,
>>>>
>>>> Kelly
>>> 
>>> 
>>> On 2016-05-25, 1:23 PM, "centos-bounces at centos.org on behalf of m.roth at 5-cent.us" <centos-bounces at centos.org on behalf of m.roth at 5-cent.us> wrote:
>>> 
>>>> Kelly Lesperance wrote:
>>>>> I’ve posted this on the forums at
>>>>> https://www.centos.org/forums/viewtopic.php?f=47&t=57926&p=244614#p244614
>>>>> - posting to the list in the hopes of getting more eyeballs on it.
>>>>>
>>>>> We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs:
>>>>>
>>>>> 2x E5-2650
>>>>> 128 GB RAM
>>>>> 12 x 4 TB 7200 RPM SATA drives connected to an HP H220 HBA
>>>>> Dual port 10 GB NIC
>>>>>
>>>>> The drives are configured as one large RAID-10 volume with mdadm,
>>>>> filesystem is XFS. The OS is not installed on the drive - we PXE boot a
>>>>> CentOS image we've built with minimal packages installed, and do the OS
>>>>> configuration via puppet. Originally, the hosts were running CentOS 6.5,
>>>>> with Kafka 0.8.1, without issue. We recently upgraded to CentOS 7.2 and
>>>>> Kafka 0.9, and that's when the trouble started.
>>>> <SNIP>
>>>> One more stupid question: could the configuration of the card for how the
>>>> drives are accessed been accidentally changed?
>>>>
>>>>          mark
>>>>
>>>> _______________________________________________
>>>> CentOS mailing list
>>>> CentOS at centos.org
>>>> https://lists.centos.org/mailman/listinfo/centos
>>> 
>>> _______________________________________________
>>> CentOS mailing list
>>> CentOS at centos.org
>>> https://lists.centos.org/mailman/listinfo/centos
>>> 
>>
>>_______________________________________________
>>CentOS mailing list
>>CentOS at centos.org
>>https://lists.centos.org/mailman/listinfo/centos
>