Hi Users
I am using RHEL 6.5 on my server.
From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on "wa" time.
Here is the df output about the disk space in my system:
********** [root@localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1915071844 103227908 1714563936 6% / tmpfs 32931472 0 32931472 0% /dev/shm **********
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
Thanks Jatin
On 01/28/2015 01:32 PM, Jatin Davey wrote:
Hi Users
I am using RHEL 6.5 on my server.
From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on "wa" time.
Here is the df output about the disk space in my system:
[root@localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1915071844 103227908 1714563936 6% / tmpfs 32931472 0 32931472 0% /dev/shm
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
you could use iotop or vmstat to see what processes are causing the IO.
+1
And remember that I/O is more than just disk. The atop monitor gives you information like top and htop, but also provides a lot of I/O information as well including network. Perhaps your server is the target of a network-based DDOS attack which can cause lots of I/O wait time. Also look at top to see what numbers are in the si and hi columns. THese stand for software and hardware interrupts. If one of those is high you can also narrow it down.
Hope this helps.
On 01/28/2015 07:51 AM, Nicolas Thierry-Mieg wrote:
On 01/28/2015 01:32 PM, Jatin Davey wrote:
Hi Users
I am using RHEL 6.5 on my server.
From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on "wa" time.
Here is the df output about the disk space in my system:
[root@localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1915071844 103227908 1714563936 6% / tmpfs 32931472 0 32931472 0% /dev/shm
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
you could use iotop or vmstat to see what processes are causing the IO.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
--
David P. Both, RHCE Millennium Technology Consulting LLC Raleigh, NC, USA 919-389-8678
dboth@millennium-technology.com
www.millennium-technology.com www.databook.bz - Home of the DataBook for Linux DataBook is a Registered Trademark of David Both
This communication may be unlawfully collected and stored by the National Security Agency (NSA) in secret. The parties to this email do not consent to the retrieving or storing of this communication and any related metadata, as well as printing, copying, re-transmitting, disseminating, or otherwise using it. If you believe you have received this communication in error, please delete it immediately.
On 01/28/2015 03:32 PM, Jatin Davey wrote:
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
Fisrt of all, you should ensure how fast are your disks. There are several methods to check that: https://www.thomas-krenn.com/en/wiki/Linux_I/O_Performance_Tests_using_dd But there are several aother way to achieve measurement.
If you notice some poor performance, then you can assume your disks are slow.
On 1/28/2015 4:32 AM, Jatin Davey wrote:
I am using RHEL 6.5 on my server.
From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on "wa" time.
Here is the df output about the disk space in my system:
[root@localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1915071844 103227908 1714563936 6% / tmpfs 32931472 0 32931472 0% /dev/shm
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
it appears you have everything on a single file system on a single 2TB drive? I'm going to guess thats a desktop grade SATA disk spinning at most at 7200rpm (many are slower, 'green' disks are often 5400rpm).
what sort of workload are you running? our high disk IO servers, doing things like transactional databases, have many 10000rpm or 15000 rpm drives on a SAS2 raid card with 1GB or 2GB of write-back-cache (battery or flash backed).
On 1/28/2015 9:37 AM, John R Pierce wrote:
On 1/28/2015 4:32 AM, Jatin Davey wrote:
....
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
...
oops, left off the last part. To improve disk IO performance, use more/faster disks, or do less disk IO. Since you've given us no clues about your hardware configuration OR software workload, its hard to be more specific.
On 1/28/2015 11:07 PM, John R Pierce wrote:
On 1/28/2015 4:32 AM, Jatin Davey wrote:
I am using RHEL 6.5 on my server.
From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on "wa" time.
Here is the df output about the disk space in my system:
[root@localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1915071844 103227908 1714563936 6% / tmpfs 32931472 0 32931472 0% /dev/shm
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
it appears you have everything on a single file system on a single 2TB drive?
[Jatin] Yes , entire disk is mounted on a single file system. Will this cause high wait time for I/O operations ? Do you think splitting it into multiple partitions would help ? Appreciate if you can enlighten me in this regard or provide pointers to it.
I'm going to guess thats a desktop grade SATA disk spinning at most at 7200rpm (many are slower, 'green' disks are often 5400rpm).
[Jatin] Yes , it is a SATA disk. I am not sure of the speed. Can you tell me how to find out this information ? Additionally we are using RAID 10 configuration with 4 disks.
what sort of workload are you running?
[Jatin] Our workload is a bit high on disk reads / writes and hence i am seeing slowness in it when using the top command. We basically run VMs on the RHEL 6.5 host. Our target is to house 5-6 application VMs on this host and ship this as a product on a rack server.
our high disk IO servers, doing things like transactional databases, have many 10000rpm or 15000 rpm drives on a SAS2 raid card with 1GB or 2GB of write-back-cache (battery or flash backed).
Hi John
Please see my responses inline. Appreciate your inputs in this regard.
Thanks Jatin
On Thu, Jan 29, 2015 at 7:07 AM, Jatin Davey jashokda@cisco.com wrote:
[Jatin] Yes , entire disk is mounted on a single file system. Will this cause high wait time for I/O operations ? Do you think splitting it into multiple partitions would help ? Appreciate if you can enlighten me in this regard or provide pointers to it.
The extra time goes to move the head back and forth from one file to the next as different applications write to different files. Different partitions may make things even worse if they happen to force those locations to be farther apart. What you need are more drives with activity spread over them so you have a better chance of a head being near where it needs to be. Adding RAM can help to a certain extent by allowing more buffering. Faster disks will help too, but won't magically eliminate seek time - moving heads is still going to be your slowest computer operation.
[Jatin] Our workload is a bit high on disk reads / writes and hence i am seeing slowness in it when using the top command. We basically run VMs on the RHEL 6.5 host. Our target is to house 5-6 application VMs on this host and ship this as a product on a rack server.
You can micro-manage this by putting a VM image on its own drive so you don't lose seek time between the images. Or make a raid with a very large number of members and hope the odds work out.
On 1/30/2015 1:53 AM, Gordon Messmer wrote:
On 01/29/2015 05:07 AM, Jatin Davey wrote:
Yes , it is a SATA disk. I am not sure of the speed. Can you tell me how to find out this information ? Additionally we are using RAID 10 configuration with 4 disks.
What RAID controller are you using?
# lspci | grep RAID
[Jatin] [root@localhost ~]# lspci | grep RAID 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02)
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 1/29/2015 7:21 PM, Jatin Davey wrote:
[root@localhost ~]# lspci | grep RAID 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02)
to get info out of those, you need to install MegaCli64 from LSI Logic, which has the ugliest command lines and output you've ever seen.
I use the python script below, which I put in /root/bin/lsi-raidinfo and chmod +x it, then run it like this..
# lsi-raidinfo -- Controllers -- -- ID | Model c0 | LSI MegaRAID SAS 9261-8i
-- Volumes -- -- ID | Type | Size | Status | InProgress volume c0u0 | RAID1 1x2 | 2727G | Optimal | None volume c0u1 | RAID6 1x8 | 16370G | Optimal | None volume c0u2 | RAID6 1x8 | 16370G | Optimal | None
-- Disks -- -- Encl:Slot | vol-span-unit | Model | Status disk 8:0 | 0-0-0 | ST33000650NS 0003 | Online, Spun Up disk 8:1 | 0-0-1 | ST33000650NS 0003 | Online, Spun Up disk 8:2 | 1-0-0 | ST33000650NS 0003 | Online, Spun Up disk 8:3 | 1-0-1 | ST33000650NS 0003 | Online, Spun Up disk 8:4 | 1-0-2 | ST33000650NS 0003 | Online, Spun Up disk 8:5 | 1-0-3 | ST33000650NS 0003 | Online, Spun Up disk 8:6 | 1-0-4 | ST33000650NS 0003 | Online, Spun Up disk 8:7 | 1-0-5 | ST33000650NS 0003 | Online, Spun Up disk 8:8 | 1-0-6 | ST33000650NS 0003 | Online, Spun Up disk 8:9 | 1-0-7 | ST33000650NS 0003 | Online, Spun Up disk 8:10 | 2-0-0 | ST33000650NS 0003 | Online, Spun Up disk 8:11 | 2-0-1 | ST33000650NS 0003 | Online, Spun Up disk 8:12 | 2-0-2 | ST33000650NS 0003 | Online, Spun Up disk 8:13 | 2-0-3 | ST33000650NS 0003 | Online, Spun Up disk 8:14 | 2-0-4 | ST33000650NS 0003 | Online, Spun Up disk 8:15 | 2-0-5 | ST33000650NS 0003 | Online, Spun Up disk 8:16 | 2-0-6 | ST33000650NS 0003 | Online, Spun Up disk 8:17 | 2-0-7 | ST33000650NS 0003 | Online, Spun Up disk 8:18 | x-x-x | ST33000650NS 0003 | Hotspare, Spun down disk 8:19 | x-x-x | ST33000650NS 0003 | Hotspare, Spun down
(yes, that system has 20 disks, in 3 raids + 2 hot spares)
*****************start of my script*********** #!/usr/bin/python
# megaclisas-status 0.6 # renamed lsi-raidinfo # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Pulse 2; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, # MA 02110-1301, USA. # # Copyright (C) 2007-2009 Adam Cecile (Le_Vert)
## modified by johnpuskar@gmail.com 08/14/11 # fixed for LSI 9285-8e on Openfiler
## modified by pierce@hogranch.com 2012-01-05 # fixed for newer version of megacli output on RHEL6/CentOS6 # output format extended to show raid span-unit and rebuild % complete
import os import re import sys
if len(sys.argv) > 2: print 'Usage: lsi-raidinfo [-d]' sys.exit(1)
# if argument -d, only print disk info printarray = True printcontroller = True if len(sys.argv) > 1: if sys.argv[1] == '-d': printarray = False printcontroller = False else: print 'Usage: lsi-raidinfo [-d]' sys.exit(1)
# Get command output def getOutput(cmd): output = os.popen(cmd) lines = [] for line in output: if not re.match(r'^$',line.strip()): lines.append(line.strip()) return lines
def returnControllerNumber(output): for line in output: if re.match(r'^Controller Count.*$',line.strip()): return int(line.split(':')[1].strip().strip('.'))
def returnControllerModel(output): for line in output: if re.match(r'^Product Name.*$',line.strip()): return line.split(':')[1].strip()
def returnArrayNumber(output): i = 0 for line in output: if re.match(r'^Virtual (Drive|Disk).*$',line.strip()): i += 1 return i
def returnArrayInfo(output,controllerid,arrayid): id = 'c'+str(controllerid)+'u'+str(arrayid) # print 'DEBUG: id = '+str(id) operationlinennumber = False linenumber = 0 units = 1 type = 'JBOD' span = 0 size = 0 for line in output: if re.match(r'^RAID Level.*$',line.strip()): type = line.strip().split(':')[1].strip() type = 'RAID' + type.split(',')[0].split('-')[1].strip() # print 'debug: type = '+str(type) if re.match(r'^Number.*$',line.strip()): units = line.strip().split(':')[1].strip() if re.match(r'^Span Depth.*$',line.strip()): span = line.strip().split(':')[1].strip() if re.match(r'^Size.*$',line.strip()): # Size reported in MB if re.match(r'^.*MB$',line.strip().split(':')[1]): size = line.strip().split(':')[1].strip('MB').strip() size = str(int(round((float(size) / 1000))))+'G' # Size reported in TB elif re.match(r'^.*TB$',line.strip().split(':')[1]): size = line.strip().split(':')[1].strip('TB').strip() size = str(int(round((float(size) * 1000))))+'G' # Size reported in GB (default) else: size = line.strip().split(':')[1].strip('GB').strip() size = str(int(round((float(size)))))+'G' if re.match(r'^State.*$',line.strip()): state = line.strip().split(':')[1].strip() if re.match(r'^Ongoing Progresses.*$',line.strip()): operationlinennumber = linenumber linenumber += 1 if operationlinennumber: inprogress = output[operationlinennumber+1] else: inprogress = 'None' if span > 1: type = type+'0' type = type + ' ' + str(span) + 'x' + str(units) return [id,type,size,state,inprogress]
def returnDiskInfo(output,controllerid,currentarrayid): arrayid = False oldarrayid = False olddiskid = False table = [] state = 'Offline' model = 'Unknown' enclnum = 'Unknown' slotnum = 'Unknown' enclsl = 'Unknown'
firstDisk = True for line in output: if re.match(r'Firmware state: .*$',line.strip()): state = line.split(':')[1].strip() if re.match(r'Rebuild',state): cmd2 = '/opt/MegaRAID/MegaCli/MegaCli64 pdrbld showprog physdrv['+str(enclnum)+':'+str(slotnum)+'] a'+str(controllerid)+' nolog' ll = getOutput(cmd2) state += ' completed ' + re.sub(r'Rebuild Progress.*Completed', '', ll[0]).strip(); if re.match(r'Slot Number: .*$',line.strip()): slotnum = line.split(':')[1].strip() if re.match(r'Inquiry Data: .*$',line.strip()): model = line.split(':')[1].strip() model = re.sub(' +', ' ', model) model = re.sub('Hotspare Information', '', model).strip() #remove bogus output from firmware 12.12 if re.match(r"(Drive|Disk)'s postion: .*$",line.strip()): spans = line.split(',') span = re.sub(r"(Drive|Disk).*DiskGroup:", '', spans[0]).strip()+'-' span += spans[1].split(':')[1].strip()+'-' span += spans[2].split(':')[1].strip() if re.match(r'Enclosure Device ID: [0-9]+$',line.strip()): if firstDisk == True: firstDisk = False else: enclsl = str(enclnum)+':'+str(slotnum) table.append([str(enclsl), span, model, state]) span = 'x-x-x' enclnum = line.split(':')[1].strip() # Last disk of last array enclsl = str(enclnum)+':'+str(slotnum) table.append([str(enclsl), span, model, state]) arraytable = [] for disk in table: arraytable.append(disk) return arraytable
cmd = '/opt/MegaRAID/MegaCli/MegaCli64 adpcount nolog' output = getOutput(cmd) controllernumber = returnControllerNumber(output)
bad = False
# List available controller if printcontroller: print '-- Controllers --' print '-- ID | Model' controllerid = 0 while controllerid < controllernumber: cmd = '/opt/MegaRAID/MegaCli/MegaCli64 adpallinfo a'+str(controllerid)+' nolog' output = getOutput(cmd) controllermodel = returnControllerModel(output) print 'c'+str(controllerid)+' | '+controllermodel controllerid += 1 print ''
if printarray: controllerid = 0 print '-- Volumes --' print '-- ID | Type | Size | Status | InProgress' # print 'controller number'+str(controllernumber) while controllerid < controllernumber: arrayid = 0 cmd = '/opt/MegaRAID/MegaCli/MegaCli64 ldinfo lall a'+str(controllerid)+' nolog' output = getOutput(cmd) arraynumber = returnArrayNumber(output) # print 'array number'+str(arraynumber) while arrayid < arraynumber: cmd = '/opt/MegaRAID/MegaCli/MegaCli64 ldinfo l'+str(arrayid)+' a'+str(controllerid)+' nolog' # print 'DEBUG: running '+str(cmd) output = getOutput(cmd) # print 'DEBUG: output '+str(output) arrayinfo = returnArrayInfo(output,controllerid,arrayid) print 'volume '+arrayinfo[0]+' | '+arrayinfo[1]+' | '+arrayinfo[2]+' | '+arrayinfo[3]+' | '+arrayinfo[4] if not arrayinfo[3] == 'Optimal': bad = True arrayid += 1 controllerid += 1 print ''
print '-- Disks --' print '-- Encl:Slot | vol-span-unit | Model | Status'
controllerid = 0 while controllerid < controllernumber: arrayid = 0 cmd = '/opt/MegaRAID/MegaCli/MegaCli64 ldinfo lall a'+str(controllerid)+' nolog' output = getOutput(cmd) arraynumber = returnArrayNumber(output) while arrayid<arraynumber: #grab disk arrayId info cmd = '/opt/MegaRAID/MegaCli/MegaCli64 pdlist a'+str(controllerid)+' nolog' #print 'debug: running '+str(cmd) output = getOutput(cmd) arraydisk = returnDiskInfo(output,controllerid,arrayid)
for array in arraydisk: print 'disk '+array[0]+' | '+array[1]+' | '+array[2]+' | '+array[3] arrayid += 1 controllerid += 1
if bad: print '\nThere is at least one disk/array in a NOT OPTIMAL state.' sys.exit(1) *****************end of the script**************
On 1/30/2015 9:44 AM, John R Pierce wrote:
On 1/29/2015 7:21 PM, Jatin Davey wrote:
[root@localhost ~]# lspci | grep RAID 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02)
to get info out of those, you need to install MegaCli64 from LSI Logic, which has the ugliest command lines and output you've ever seen.
I use the python script below, which I put in /root/bin/lsi-raidinfo and chmod +x it, then run it like this..
# lsi-raidinfo -- Controllers -- -- ID | Model c0 | LSI MegaRAID SAS 9261-8i
-- Volumes -- -- ID | Type | Size | Status | InProgress volume c0u0 | RAID1 1x2 | 2727G | Optimal | None volume c0u1 | RAID6 1x8 | 16370G | Optimal | None volume c0u2 | RAID6 1x8 | 16370G | Optimal | None
-- Disks -- -- Encl:Slot | vol-span-unit | Model | Status disk 8:0 | 0-0-0 | ST33000650NS 0003 | Online, Spun Up disk 8:1 | 0-0-1 | ST33000650NS 0003 | Online, Spun Up disk 8:2 | 1-0-0 | ST33000650NS 0003 | Online, Spun Up disk 8:3 | 1-0-1 | ST33000650NS 0003 | Online, Spun Up disk 8:4 | 1-0-2 | ST33000650NS 0003 | Online, Spun Up disk 8:5 | 1-0-3 | ST33000650NS 0003 | Online, Spun Up disk 8:6 | 1-0-4 | ST33000650NS 0003 | Online, Spun Up disk 8:7 | 1-0-5 | ST33000650NS 0003 | Online, Spun Up disk 8:8 | 1-0-6 | ST33000650NS 0003 | Online, Spun Up disk 8:9 | 1-0-7 | ST33000650NS 0003 | Online, Spun Up disk 8:10 | 2-0-0 | ST33000650NS 0003 | Online, Spun Up disk 8:11 | 2-0-1 | ST33000650NS 0003 | Online, Spun Up disk 8:12 | 2-0-2 | ST33000650NS 0003 | Online, Spun Up disk 8:13 | 2-0-3 | ST33000650NS 0003 | Online, Spun Up disk 8:14 | 2-0-4 | ST33000650NS 0003 | Online, Spun Up disk 8:15 | 2-0-5 | ST33000650NS 0003 | Online, Spun Up disk 8:16 | 2-0-6 | ST33000650NS 0003 | Online, Spun Up disk 8:17 | 2-0-7 | ST33000650NS 0003 | Online, Spun Up disk 8:18 | x-x-x | ST33000650NS 0003 | Hotspare, Spun down disk 8:19 | x-x-x | ST33000650NS 0003 | Hotspare, Spun down
(yes, that system has 20 disks, in 3 raids + 2 hot spares)
*****************start of my script*********** #!/usr/bin/python
# megaclisas-status 0.6 # renamed lsi-raidinfo # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Pulse 2; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, # MA 02110-1301, USA. # # Copyright (C) 2007-2009 Adam Cecile (Le_Vert)
## modified by johnpuskar@gmail.com 08/14/11 # fixed for LSI 9285-8e on Openfiler
## modified by pierce@hogranch.com 2012-01-05 # fixed for newer version of megacli output on RHEL6/CentOS6 # output format extended to show raid span-unit and rebuild % complete
import os import re import sys
if len(sys.argv) > 2: print 'Usage: lsi-raidinfo [-d]' sys.exit(1)
# if argument -d, only print disk info printarray = True printcontroller = True if len(sys.argv) > 1: if sys.argv[1] == '-d': printarray = False printcontroller = False else: print 'Usage: lsi-raidinfo [-d]' sys.exit(1)
# Get command output def getOutput(cmd): output = os.popen(cmd) lines = [] for line in output: if not re.match(r'^$',line.strip()): lines.append(line.strip()) return lines
def returnControllerNumber(output): for line in output: if re.match(r'^Controller Count.*$',line.strip()): return int(line.split(':')[1].strip().strip('.'))
def returnControllerModel(output): for line in output: if re.match(r'^Product Name.*$',line.strip()): return line.split(':')[1].strip()
def returnArrayNumber(output): i = 0 for line in output: if re.match(r'^Virtual (Drive|Disk).*$',line.strip()): i += 1 return i
def returnArrayInfo(output,controllerid,arrayid): id = 'c'+str(controllerid)+'u'+str(arrayid) # print 'DEBUG: id = '+str(id) operationlinennumber = False linenumber = 0 units = 1 type = 'JBOD' span = 0 size = 0 for line in output: if re.match(r'^RAID Level.*$',line.strip()): type = line.strip().split(':')[1].strip() type = 'RAID' + type.split(',')[0].split('-')[1].strip() # print 'debug: type = '+str(type) if re.match(r'^Number.*$',line.strip()): units = line.strip().split(':')[1].strip() if re.match(r'^Span Depth.*$',line.strip()): span = line.strip().split(':')[1].strip() if re.match(r'^Size.*$',line.strip()): # Size reported in MB if re.match(r'^.*MB$',line.strip().split(':')[1]): size = line.strip().split(':')[1].strip('MB').strip() size = str(int(round((float(size) / 1000))))+'G' # Size reported in TB elif re.match(r'^.*TB$',line.strip().split(':')[1]): size = line.strip().split(':')[1].strip('TB').strip() size = str(int(round((float(size) * 1000))))+'G' # Size reported in GB (default) else: size = line.strip().split(':')[1].strip('GB').strip() size = str(int(round((float(size)))))+'G' if re.match(r'^State.*$',line.strip()): state = line.strip().split(':')[1].strip() if re.match(r'^Ongoing Progresses.*$',line.strip()): operationlinennumber = linenumber linenumber += 1 if operationlinennumber: inprogress = output[operationlinennumber+1] else: inprogress = 'None' if span > 1: type = type+'0' type = type + ' ' + str(span) + 'x' + str(units) return [id,type,size,state,inprogress]
def returnDiskInfo(output,controllerid,currentarrayid): arrayid = False oldarrayid = False olddiskid = False table = [] state = 'Offline' model = 'Unknown' enclnum = 'Unknown' slotnum = 'Unknown' enclsl = 'Unknown'
firstDisk = True for line in output: if re.match(r'Firmware state: .*$',line.strip()): state = line.split(':')[1].strip() if re.match(r'Rebuild',state): cmd2 = '/opt/MegaRAID/MegaCli/MegaCli64 pdrbldshowprog physdrv['+str(enclnum)+':'+str(slotnum)+'] a'+str(controllerid)+' nolog' ll = getOutput(cmd2) state += ' completed ' + re.sub(r'Rebuild Progress.*Completed', '', ll[0]).strip(); if re.match(r'Slot Number: .*$',line.strip()): slotnum = line.split(':')[1].strip() if re.match(r'Inquiry Data: .*$',line.strip()): model = line.split(':')[1].strip() model = re.sub(' +', ' ', model) model = re.sub('Hotspare Information', '', model).strip() #remove bogus output from firmware 12.12 if re.match(r"(Drive|Disk)'s postion: .*$",line.strip()): spans = line.split(',') span = re.sub(r"(Drive|Disk).*DiskGroup:", '', spans[0]).strip()+'-' span += spans[1].split(':')[1].strip()+'-' span += spans[2].split(':')[1].strip() if re.match(r'Enclosure Device ID: [0-9]+$',line.strip()): if firstDisk == True: firstDisk = False else: enclsl = str(enclnum)+':'+str(slotnum) table.append([str(enclsl), span, model, state]) span = 'x-x-x' enclnum = line.split(':')[1].strip() # Last disk of last array enclsl = str(enclnum)+':'+str(slotnum) table.append([str(enclsl), span, model, state]) arraytable = [] for disk in table: arraytable.append(disk) return arraytable
cmd = '/opt/MegaRAID/MegaCli/MegaCli64 adpcount nolog' output = getOutput(cmd) controllernumber = returnControllerNumber(output)
bad = False
# List available controller if printcontroller: print '-- Controllers --' print '-- ID | Model' controllerid = 0 while controllerid < controllernumber: cmd = '/opt/MegaRAID/MegaCli/MegaCli64 adpallinfo a'+str(controllerid)+' nolog' output = getOutput(cmd) controllermodel = returnControllerModel(output) print 'c'+str(controllerid)+' | '+controllermodel controllerid += 1 print ''
if printarray: controllerid = 0 print '-- Volumes --' print '-- ID | Type | Size | Status | InProgress' # print 'controller number'+str(controllernumber) while controllerid < controllernumber: arrayid = 0 cmd = '/opt/MegaRAID/MegaCli/MegaCli64 ldinfo lall a'+str(controllerid)+' nolog' output = getOutput(cmd) arraynumber = returnArrayNumber(output) # print 'array number'+str(arraynumber) while arrayid < arraynumber: cmd = '/opt/MegaRAID/MegaCli/MegaCli64 ldinfo l'+str(arrayid)+' a'+str(controllerid)+' nolog' # print 'DEBUG: running '+str(cmd) output = getOutput(cmd) # print 'DEBUG: output '+str(output) arrayinfo = returnArrayInfo(output,controllerid,arrayid) print 'volume '+arrayinfo[0]+' | '+arrayinfo[1]+' | '+arrayinfo[2]+' | '+arrayinfo[3]+' | '+arrayinfo[4] if not arrayinfo[3] == 'Optimal': bad = True arrayid += 1 controllerid += 1 print ''
print '-- Disks --' print '-- Encl:Slot | vol-span-unit | Model | Status'
controllerid = 0 while controllerid < controllernumber: arrayid = 0 cmd = '/opt/MegaRAID/MegaCli/MegaCli64 ldinfo lall a'+str(controllerid)+' nolog' output = getOutput(cmd) arraynumber = returnArrayNumber(output) while arrayid<arraynumber: #grab disk arrayId info cmd = '/opt/MegaRAID/MegaCli/MegaCli64 pdlist a'+str(controllerid)+' nolog' #print 'debug: running '+str(cmd) output = getOutput(cmd) arraydisk = returnDiskInfo(output,controllerid,arrayid)
for array in arraydisk: print 'disk '+array[0]+' | '+array[1]+' | '+array[2]+' |'+array[3] arrayid += 1 controllerid += 1
if bad: print '\nThere is at least one disk/array in a NOT OPTIMAL state.' sys.exit(1) *****************end of the script**************
Thanks John,
I ran your script and here is the output for it:
Start of the Output*************************** [root@localhost bin]# lsi-raidinfo sh: /opt/MegaRAID/MegaCli/MegaCli64: No such file or directory -- Controllers -- -- ID | Model
-- Volumes -- -- ID | Type | Size | Status | InProgress
-- Disks -- -- Encl:Slot | vol-span-unit | Model | Status End of Output ***********************************
Thanks Jatin
On Sun, Feb 1, 2015 at 10:31 PM, Jatin Davey jashokda@cisco.com wrote:
Thanks John,
I ran your script and here is the output for it:
Start of the Output*************************** [root@localhost bin]# lsi-raidinfo sh: /opt/MegaRAID/MegaCli/MegaCli64: No such file or directory
Do you have the MegaCLI tools installed?
"No such file or directory"
-- Controllers -- -- ID | Model
-- Volumes -- -- ID | Type | Size | Status | InProgress
-- Disks -- -- Encl:Slot | vol-span-unit | Model | Status End of Output ***********************************
Thanks Jatin
On 01/29/2015 07:21 PM, Jatin Davey wrote:
[root@localhost ~]# lspci | grep RAID 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02)
I'd expect some pretty good numbers from that with a 4 disk RAID 10. As before, SSD will perform better. If you need 2TB of space at similar cost, maybe consider 8 500GB drives instead of 4 1TB drives.
Since your workload is in VMs, you'll also see a large (potentially very large) performance benefit from a change to your filesystem layout. I recommend something closer to the default: one partition for /boot, and a second partition spanning the rest of the disk (RAID volume). The second partition should be part of an LVM group, your root filesystem should be limited to what it needs. Maybe a 40GB root FS. Use LVM backed VMs instead of file backed VMs.
When you use file backed VMs, all of your disk IO goes through the filesystem layer, and possibly through the journal, which amplifies your write count. That's costly.
Beyond that, you might just want to test the system and make sure that it's performing as expected. Figure out how much RAM is on your RAID controller and how much is available to the host system. Shut down all of your VMs and run "bonnie++" with a data size at least 2x the size of system RAM + RAID controller RAM.
On 01/28/2015 04:32 AM, Jatin Davey wrote:
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
Start by identifying your disk and controller. Assuming that this is a single SATA disk:
# smartctl -a /dev/sda | egrep 'Model:|Rate:|SATA Version' # lspci | grep SATA
Next install and run iotop. If there's something on your system that's particularly active and unnecessary, disable it.
If everything contending for your disk *is* necessary, then you may need to take steps to improve disk performance. There are a number of possibilities.
1: Use an SSD instead of a disk.
2: Use an SSD in addition to a disk. If a purely SSD storage system is too expensive, you might choose to use an SSD for one or more filesytems that you can mount where the high volume IO is occurring, and continue to use the disk for filesystems that need more data capacity and fewer IOPS.
3: Use a fast RAID array. A RAID1 array can provide better read performance than a single disk. For higher write volumes, use RAID10 on four or more disks. Avoid RAID 5 or 6, as they'll be slower than a single disk.
4: Replace poorly supported hardware. It might be that your SATA controller isn't well supported, and performs badly as a result. I used to have a desktop system with NVidia motherboard chipset. Its SATA controller was garbage. Performance with the same disk was noticeably better with a cheap PCI-E SATA controller.
On 1/29/2015 4:50 AM, Gordon Messmer wrote:
On 01/28/2015 04:32 AM, Jatin Davey wrote:
Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.
Start by identifying your disk and controller. Assuming that this is a single SATA disk:
# smartctl -a /dev/sda | egrep 'Model:|Rate:|SATA Version'
[Jatin] [root@localhost images]# smartctl -a /dev/sda | egrep 'Model:|Rate:|SATA Version' [root@localhost images]#
# lspci | grep SATA
[Jatin] [root@localhost images]# lspci | grep SATA 00:11.4 SATA controller: Intel Corporation Wellsburg sSATA Controller [AHCI mode] (rev 05) 00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA Controller [AHCI mode] (rev 05)
Next install and run iotop. If there's something on your system that's particularly active and unnecessary, disable it.
If everything contending for your disk *is* necessary, then you may need to take steps to improve disk performance. There are a number of possibilities.
1: Use an SSD instead of a disk.
2: Use an SSD in addition to a disk. If a purely SSD storage system is too expensive, you might choose to use an SSD for one or more filesytems that you can mount where the high volume IO is occurring, and continue to use the disk for filesystems that need more data capacity and fewer IOPS.
[Jatin] Thanks for these inputs. I will surely take it up with my team to see if we can incorporate new hardware for our product.
3: Use a fast RAID array. A RAID1 array can provide better read performance than a single disk. For higher write volumes, use RAID10 on four or more disks. Avoid RAID 5 or 6, as they'll be slower than a single disk.
[Jatin] We are using RAID 10 in our server with 4 disks.
4: Replace poorly supported hardware. It might be that your SATA controller isn't well supported, and performs badly as a result. I used to have a desktop system with NVidia motherboard chipset. Its SATA controller was garbage. Performance with the same disk was noticeably better with a cheap PCI-E SATA controller.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi Gordon
Please see my responses inline to your questions. Appreciate your inputs in this direction.
Thanks Jatin
-
David Both -
Gordon Messmer -
Jatin Davey -
John R Pierce -
Les Mikesell -
Mihamina RAKOTOMANDIMBY -
Nicolas Thierry-Mieg -
SilverTip257