[CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Thu May 26 13:07:39 UTC 2016
Kelly Lesperance <klesperance at blackberry.com>

Hi Charles,

	Looks to me like all of the drives are performing roughly the same – there’s certainly not 1 that sticks out (also note this is happening on all 23 nodes in the cluster).

Thanks!

Kelly

[root at r1k1.kafka.log10.blackberry sys] # cat /proc/mdstat 
Personalities : [raid10] 
md127 : active raid10 sdc[2] sdh[7] sdb[1] sdf[5] sde[13] sdg[12] sdj[9] sdk[10] sda[0] sdl[11] sdd[3] sdi[8]
      23441323008 blocks super 1.2 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
      [>....................]  check =  0.0% (618944/23441323008) finish=108288.4min speed=3607K/sec
      
unused devices: <none>
[root at r1k1.kafka.log10.blackberry sys] # iostat -xdmc 1 10
Linux 3.10.0-327.18.2.el7.x86_64 (r1k1.kafka.log10.blackberry) 	05/26/16 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.76    0.07    2.48    0.16    0.00   84.53

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.01     0.56    0.26   26.44     0.06    11.30   871.39     9.67  362.22    5.14  365.71   6.68  17.83
sdk               0.01     0.56    0.26   26.56     0.06    11.30   867.70     9.53  355.45    5.05  358.84   6.58  17.65
sdc               0.01     0.46    0.26   26.34     0.06    11.29   874.67     9.73  365.89    4.86  369.38   6.81  18.11
sdd               0.01     0.46    0.20   26.34     0.07    11.29   876.98    10.40  391.99    5.33  394.93   7.17  19.02
sda               0.01     0.49    0.26   26.53     0.06    11.29   868.24     9.48  353.91    4.96  357.36   6.57  17.61
sdj               0.01     0.56    0.20   26.44     0.07    11.30   873.73    10.04  376.87    5.48  379.68   6.91  18.40
sdl               0.01     0.56    0.20   26.56     0.07    11.30   869.99     9.77  365.16    5.92  367.92   6.72  17.99
sdh               0.01     0.57    0.21   26.79     0.07    11.30   862.30     9.65  357.60    5.27  360.31   6.63  17.90
sde               0.01     0.47    0.26   26.13     0.06    11.29   881.38    10.60  401.47    6.62  405.41   7.35  19.41
sdf               0.01     0.47    0.20   26.13     0.07    11.29   883.71     9.53  361.85    5.24  364.64   6.73  17.73
sdg               0.01     0.57    0.26   26.79     0.06    11.30   859.99    10.15  375.20    5.26  378.82   6.86  18.57
sdb               0.01     0.49    0.20   26.53     0.07    11.29   870.69     9.85  368.48    5.35  371.23   6.79  18.15
md127             0.00     0.00    2.51  156.82     0.77    67.77   881.06     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          25.51    0.03    4.37    1.05    0.00   69.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     1.00    8.00   30.00     0.50    14.18   791.16     1.06   28.03    0.50   35.37   6.97  26.50
sdk               0.00     0.00    8.00   30.00     0.50    14.52   809.47     0.93   24.32    0.00   30.80   7.87  29.90
sdc               0.00     1.00    9.00   32.00     0.56    15.21   787.90     1.13   27.54    0.67   35.09   6.90  28.30
sdd               0.00     1.00   10.00   32.00     0.62    15.21   772.19     1.29   30.69    0.70   40.06   6.76  28.40
sda               0.00     0.00    8.00   38.00     0.50    15.54   714.09     1.40   30.35    0.38   36.66   7.91  36.40
sdj               0.00     1.00    8.00   30.00     0.50    14.18   791.16     1.05   27.68    0.50   34.93   7.00  26.60
sdl               0.00     0.00    8.00   30.00     0.50    14.52   809.47     0.90   23.61    0.25   29.83   7.66  29.10
sdh               0.00     1.00   13.00   34.00     0.81    14.11   650.04     1.17   24.98    0.31   34.41   6.60  31.00
sde               0.00     0.00   16.00   31.00     1.00    14.54   676.94     1.20   25.45    0.31   38.42   7.13  33.50
sdf               0.00     0.00   16.00   31.00     1.00    14.54   676.94     1.19   25.38    0.31   38.32   5.57  26.20
sdg               0.00     1.00   13.00   34.00     0.81    14.11   650.04     1.22   25.98    0.31   35.79   6.70  31.50
sdb               0.00     0.00    8.00   38.00     0.50    15.54   714.09     1.31   28.41    0.25   34.34   8.02  36.90
md127             0.00     0.00    0.00  198.00     0.00    86.59   895.60     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          21.31    0.00    2.99    0.00    0.00   75.69

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00    7.00     0.50     3.50   546.13     0.13    8.47    6.25   11.00   8.47  12.70
sdk               0.00     0.00    8.00    8.00     0.50     3.98   574.00     0.16    9.88    1.00   18.75  10.06  16.10
sdc               0.00     0.00    8.00    8.00     0.50     4.00   576.00     0.12    7.25    0.62   13.88   7.25  11.60
sdd               0.00     0.00    8.00    8.00     0.50     4.00   576.00     0.12    7.44    0.50   14.38   7.44  11.90
sda               0.00     0.00    8.00    8.00     0.50     4.00   576.00     0.13    8.00    0.50   15.50   8.00  12.80
sdj               0.00     0.00    8.00    7.00     0.50     3.50   546.13     0.18   12.20    9.25   15.57  12.20  18.30
sdl               0.00     0.00    8.00    9.00     0.50     4.48   600.47     0.11    6.94    1.00   12.22   6.59  11.20
sdh               0.00     0.00    8.00    9.00     0.50     3.51   482.82     0.10    6.12    0.50   11.11   6.12  10.40
sde               0.00     0.00    8.00    9.00     0.50     4.00   542.59     0.16    9.65    0.25   18.00   9.65  16.40
sdf               0.00     0.00    8.00    9.00     0.50     4.00   542.59     0.13    7.65    0.25   14.22   7.65  13.00
sdg               0.00     0.00    8.00    9.00     0.50     3.51   482.82     0.13    7.59    0.50   13.89   7.59  12.90
sdb               0.00     0.00    8.00    8.00     0.50     4.00   576.00     0.11    6.62    2.12   11.12   6.62  10.60
md127             0.00     0.00    0.00   49.00     0.00    23.00   961.14     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.70    4.21    4.24    0.00    0.00   72.85

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00   15.00     0.50     6.09   586.78     0.22    9.17    0.25   13.93   6.26  14.40
sdk               0.00     0.00    8.00   14.00     0.50     5.55   563.27     0.25   11.68    2.38   17.00   7.59  16.70
sdc               0.00     0.00    8.00   13.00     0.50     6.50   682.67     0.15    7.00    0.25   11.15   6.00  12.60
sdd               0.00     0.00    8.00   13.00     0.50     6.50   682.67     0.17    7.95    0.25   12.69   6.86  14.40
sda               0.00     0.00    8.00   14.00     0.50     6.50   652.00     0.26   11.77    0.62   18.14   7.86  17.30
sdj               0.00     0.00    8.00   15.00     0.50     6.09   586.78     0.34   14.35    2.00   20.93   9.87  22.70
sdl               0.00     0.00    8.00   13.00     0.50     5.05   541.33     0.25   11.86    0.50   18.85   7.57  15.90
sdh               0.00     0.00   10.00   17.00     0.62     7.14   589.04     0.33   12.19    0.60   19.00   7.41  20.00
sde               0.00     0.00    8.00   18.00     0.50     6.68   565.85     0.31   11.77    0.25   16.89   7.00  18.20
sdf               0.00     0.00    8.00   18.00     0.50     6.68   565.85     0.42   16.12    2.25   22.28   9.96  25.90
sdg               0.00     0.00   10.00   17.00     0.62     7.14   589.04     0.33   12.30    0.60   19.18   6.59  17.80
sdb               0.00     0.00    8.00   14.00     0.50     6.50   652.00     0.27   12.14    2.25   17.79   8.00  17.60
md127             0.00     0.00    0.00   91.00     0.00    38.47   865.76     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.69    0.03    3.08    0.03    0.00   80.16

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00   18.00   14.00     1.12     7.00   520.00     0.15    4.84    0.50   10.43   4.62  14.80
sdk               0.00     0.00   16.00   14.00     1.00     7.00   546.13     0.14    4.77    0.38    9.79   4.77  14.30
sdc               0.00     0.00   16.00   13.00     1.00     6.50   529.66     0.14    5.00    0.38   10.69   5.00  14.50
sdd               0.00     0.00   16.00   13.00     1.00     6.50   529.66     0.15    5.10    0.38   10.92   5.10  14.80
sda               0.00     0.00   16.00   18.00     1.00     7.54   514.59     0.21    6.12    1.31   10.39   6.26  21.30
sdj               0.00     0.00   18.00   14.00     1.12     7.00   520.00     0.13    4.25    0.50    9.07   4.03  12.90
sdl               0.00     0.00   16.00   14.00     1.00     7.00   546.13     0.13    4.47    0.31    9.21   4.47  13.40
sdh               7.00     0.00   10.00   13.00     1.06     6.50   673.39     0.10    4.57    0.50    7.69   4.57  10.50
sde               6.00     0.00   10.00   13.00     1.00     6.50   667.83     0.15    6.35    0.60   10.77   6.35  14.60
sdf               6.00     0.00   10.00   13.00     1.00     6.50   667.83     0.14    6.22    0.60   10.54   6.22  14.30
sdg               7.00     0.00   10.00   13.00     1.06     6.50   673.39     0.10    4.39    0.50    7.38   4.39  10.10
sdb               0.00     0.00   16.00   19.00     1.00     7.57   501.71     0.13    3.77    0.31    6.68   3.77  13.20
md127             0.00     0.00    0.00   85.00     0.00    40.57   977.60     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.73    0.00    5.91    0.06    0.00   71.30

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi             242.00     0.00 1048.00    1.00    80.62     0.01   157.43     4.38    4.18    4.17    8.00   0.37  38.50
sdk             334.00     0.00  954.00    0.00    80.50     0.00   172.81     7.27    7.62    7.62    0.00   0.45  43.20
sdc             294.00     0.00  994.00    0.00    80.50     0.00   165.86     5.56    5.59    5.59    0.00   0.40  39.80
sdd             249.00     0.00 1039.00    0.00    80.50     0.00   158.68     4.11    3.95    3.95    0.00   0.37  38.00
sda             268.00     0.00 1020.00   11.00    80.50     0.18   160.26     5.47    5.31    5.14   21.36   0.58  60.20
sdj             253.00     0.00 1037.00    1.00    80.62     0.01   159.10     4.42    4.26    4.26    3.00   0.37  38.80
sdl             257.00     0.00 1031.00    0.00    80.50     0.00   159.91     5.13    4.98    4.98    0.00   0.37  38.30
sdh             224.00     0.00 1064.00    1.00    80.50     0.00   154.81     3.80    3.57    3.57   10.00   0.36  38.30
sde             247.00     0.00 1041.00    0.00    80.50     0.00   158.37     4.96    4.77    4.77    0.00   0.37  38.60
sdf             220.00     0.00 1068.00    0.00    80.50     0.00   154.37     3.70    3.47    3.47    0.00   0.33  35.40
sdg             242.00     0.00 1046.00    1.00    80.50     0.00   157.47     5.05    4.82    4.81   13.00   0.39  40.80
sdb             239.00     0.00 1049.00   10.00    80.50     0.15   155.97     4.77    4.51    4.43   12.10   0.45  47.90
md127             0.00     0.00    0.00   13.00     0.00     0.17    26.46     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.14    0.03    6.00    0.00    0.00   65.83

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     5.00   12.00   37.00     0.75     0.29    43.27     1.13   23.04    0.67   30.30   4.98  24.40
sdk               0.00    17.00   16.00   34.00     1.00     0.34    54.88     2.95   59.02    0.56   86.53   3.98  19.90
sdc               0.00     4.00    8.00   33.00     0.50     0.25    37.66     1.14   27.88    0.25   34.58   5.66  23.20
sdd               0.00     4.00    8.00   33.00     0.50     0.25    37.66     0.52   12.83    0.25   15.88   4.02  16.50
sda               0.00     3.00   16.00   21.00     1.00     0.17    64.86     0.26    7.14    1.06   11.76   3.92  14.50
sdj               0.00     5.00   12.00   37.00     0.75     0.29    43.27     0.84   17.24    0.42   22.70   4.47  21.90
sdl               0.00    17.00   16.00   34.00     1.00     0.34    54.88     2.98   59.56    0.56   87.32   3.92  19.60
sdh               0.00     4.00    8.00   26.00     0.50     0.20    41.88     0.67   19.71    1.75   25.23   4.50  15.30
sde               0.00     4.00    8.00   22.00     0.50     0.19    47.20     0.39   12.83    2.38   16.64   3.93  11.80
sdf               0.00     4.00    8.00   22.00     0.50     0.19    47.20     0.35   11.60    2.25   15.00   3.67  11.00
sdg               0.00     4.00    8.00   26.00     0.50     0.20    41.88     0.67   19.62    1.50   25.19   4.12  14.00
sdb               0.00     3.00   16.00   21.00     1.00     0.17    64.86     0.42   11.27    1.00   19.10   6.73  24.90
md127             0.00     0.00    0.00  210.00     0.00     1.44    14.06     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35.57    0.00   10.34    0.00    0.00   54.08

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    9.00   11.00     0.56     0.04    62.00     0.14    7.00    1.78   11.27   6.50  13.00
sdk               0.00     0.00    8.00   11.00     0.50     0.05    58.95     0.10    5.26    0.88    8.45   5.26  10.00
sdc               0.00     0.00   16.00   16.00     1.00     0.07    68.75     0.17    5.44    0.44   10.44   5.44  17.40
sdd               0.00     0.00   16.00   16.00     1.00     0.07    68.75     0.17    5.38    1.00    9.75   5.38  17.20
sda               0.00     0.00    8.00   16.00     0.50     0.06    48.08     0.20    8.54    0.75   12.44   5.42  13.00
sdj               0.00     0.00    9.00   11.00     0.56     0.04    62.00     0.13    6.65    0.44   11.73   5.90  11.80
sdl               0.00     0.00    8.00   11.00     0.50     0.05    58.95     0.12    6.16    1.62    9.45   6.16  11.70
sdh               0.00     0.00   16.00   30.00     1.00     0.11    49.63     0.32    6.85    0.44   10.27   4.39  20.20
sde               0.00     0.00   16.00    6.00     1.00     0.02    95.27     0.10    4.41    0.44   15.00   4.41   9.70
sdf               0.00     0.00   16.00    6.00     1.00     0.02    95.27     0.14    6.59    4.06   13.33   6.55  14.40
sdg               0.00     0.00   16.00   29.00     1.00     0.11    50.56     0.31    6.80    0.44   10.31   4.82  21.70
sdb               0.00     0.00    8.00   16.00     0.50     0.06    48.08     0.24   10.17    0.62   14.94   6.75  16.20
md127             0.00     0.00    0.00   89.00     0.00     0.36     8.24     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35.50    0.03    7.43    0.00    0.00   57.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi              74.00     0.00   21.00    7.00     5.94     1.53   546.00     0.47   16.71   17.57   14.14   4.89  13.70
sdk              70.00     0.00   26.00   10.00     6.00     1.41   421.56     0.41   10.94   11.73    8.90   4.33  15.60
sdc              77.00     0.00   11.00    9.00     5.50     1.57   723.60     0.64   32.00   42.64   19.00  10.65  21.30
sdd              77.00     0.00   11.00    9.00     5.50     1.57   723.60     1.19   59.60   96.36   14.67  12.10  24.20
sda              71.00     1.00   24.00   11.00     5.94     1.53   437.09     0.51   14.46   14.38   14.64   5.09  17.80
sdj              74.00     0.00   21.00    7.00     5.94     1.53   546.00     0.58   20.79   20.57   21.43   7.04  19.70
sdl              70.00     0.00   26.00   11.00     6.00     1.91   437.84     0.39   10.54   11.04    9.36   4.32  16.00
sdh              77.00     0.00   11.00    7.00     5.50     1.52   798.67     0.43   24.17   33.82    9.00   6.61  11.90
sde              77.00     0.00   11.00    6.00     5.50     1.52   845.18     0.58   34.24   36.91   29.33  13.71  23.30
sdf              77.00     0.00   11.00    6.00     5.50     1.52   845.18     0.60   35.35   45.36   17.00  10.06  17.10
sdg              77.00     0.00   11.00    8.00     5.50     1.52   757.05     0.43   22.95   32.00   10.50   6.89  13.10
sdb              71.00     1.00   24.00   11.00     5.94     1.53   437.09     0.60   17.14   13.67   24.73   9.03  31.60
md127             0.00     0.00    0.00   52.00     0.00     9.57   376.96     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          27.06    0.03    6.00    0.00    0.00   66.91

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi              14.00     0.00   10.00    9.00     1.50     4.06   599.58     0.13    6.84    2.60   11.56   6.63  12.60
sdk              14.00     0.00   10.00   10.00     1.50     5.00   665.60     0.13    7.05    2.50   11.60   6.35  12.70
sdc              14.00     0.00   11.00   10.00     1.56     4.01   543.24     0.15    7.33    1.00   14.30   7.24  15.20
sdd              14.00     0.00   11.00   10.00     1.56     4.01   543.24     0.15    7.14    1.00   13.90   7.05  14.80
sda              14.00     0.00   11.00   10.00     1.56     4.20   561.52     0.12    5.38    0.91   10.30   5.43  11.40
sdj              14.00     0.00   10.00    9.00     1.50     4.06   599.58     0.26   13.68    3.60   24.89  13.47  25.60
sdl              14.00     0.00   10.00    9.00     1.50     4.50   646.74     0.13    6.63    1.30   12.56   6.47  12.30
sdh              13.00     0.00   11.00    9.00     1.50     4.00   563.60     0.11    5.70    1.18   11.22   5.55  11.10
sde              14.00     0.00   10.00    8.00     1.50     4.00   625.78     0.09    4.78    1.10    9.38   4.67   8.40
sdf              14.00     0.00   10.00    8.00     1.50     4.00   625.78     0.14    8.06    4.00   13.12   7.17  12.90
sdg              13.00     0.00   11.00    9.00     1.50     4.00   563.60     0.14    7.00    1.91   13.22   6.80  13.60
sdb              14.00     0.00   11.00   10.00     1.56     4.20   561.52     0.17    7.67    1.73   14.20   7.71  16.20
md127             0.00     0.00    0.00   56.00     0.00    25.27   924.14     0.00    0.00    0.00    0.00   0.00   0.00



On 2016-05-25, 5:43 PM, "centos-bounces at centos.org on behalf of cpolish at surewest.net" <centos-bounces at centos.org on behalf of cpolish at surewest.net> wrote:

>On 2016-05-25 19:13, Kelly Lesperance wrote:
>> Hdparm didn’t get far:
>> 
>> [root at r1k1 ~] # hdparm -tT /dev/sda
>> 
>> /dev/sda:
>>  Timing cached reads:   Alarm clock
>> [root at r1k1 ~] #
>
>Hi Kelly,
>
>Try running 'iostat -xdmc 1'. Look for a single drive that has
>substantially greater await than ~10msec. If all the drives 
>except one are taking 6-8msec, but one is very much more, you've
>got a drive that drags down the whole array's performance.
>
>Ignore the very first output from the command - it's an
>average of the disk subsystem since boot.
>
>Post a representative output along with the contents /proc/mdstat.
>
>Good luck,
>-- 
>Charles Polisher
>
>_______________________________________________
>CentOS mailing list
>CentOS at centos.org
>https://lists.centos.org/mailman/listinfo/centos