Hi,
We have a storage server (HP DL360G5 + MSA20 (12 disks in RAID 6) on a SmartArray6400). 10 directories are exported through nfs to 10 clients (rsize=32768,wsize=32768,soft,intr,nosuid,proto=udp,vers=3). The server is apparently not doing much but... we have very high waiting IOs.
dstat show very little activity, but high 'wai'...
# dstat ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 88 12 0 0| 413k 98k| 0 0 | 0 0 | 188 132 0 1 46 53 0 0| 716k 48k| 19k 420k| 0 0 |1345 476 0 1 49 50 0 1| 492k 32k| 12k 181k| 0 0 |1269 482 0 1 63 37 0 0| 316k 159k| 58k 278k| 0 0 |1789 1562 0 0 74 26 0 0| 84k 512k|1937B 6680B| 0 0 |1200 106 0 1 44 55 0 1| 612k 80k| 14k 221k| 0 0 |1378 538 1 1 52 47 0 0| 628k 0 | 17k 318k| 0 0 |1327 520 0 1 50 49 0 0| 484k 60k| 14k 178k| 0 0 |1303 494 0 0 87 13 0 0| 124k 0 |7745B 116k| 0 0 |1083 139 0 1 59 41 0 0| 316k 60k|4828B 67k| 0 0 |1179 346
top shows that one nfsd is usualy in state 'D' (waiting).
# top -i (sorted by cpu usage) top - 18:11:28 up 207 days, 7:13, 2 users, load average: 0.99, 1.07, 1.00 Tasks: 124 total, 1 running, 123 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 54.3%id, 45.3%wa, 0.2%hi, 0.0%si, 0.0%st Mem: 3089252k total, 3068112k used, 21140k free, 928468k buffers Swap: 2008116k total, 164k used, 2007952k free, 293716k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16571 root 15 0 12708 1076 788 R 1 0.0 0:00.02 top 2580 root 15 0 0 0 0 D 0 0.0 2:36.70 nfsd
# cat /proc/net/rpc/nfsd rc 8872 34768207 38630969 fh 142 0 0 0 0 io 2432226534 884662242 th 32 394 4851.311 2437.416 370.949 238.432 542.241 4.942 2.239 1.000 0.427 0.541 ra 64 3876274 5025 3724 2551 2030 2036 1506 1607 1219 1154 1136249 net 73410453 73261524 0 0 rpc 73408119 0 0 0 0 proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 proc3 22 33 9503937 1315066 11670859 7139862 0 5033349 28129122 3729031 0 0 0 487614 0 1116215 0 0 2054329 21225 66 0 2351744 proc4 2 0 0 proc4ops 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Do you think nfs is the problem here? If so, is there something wrong with our config? Is it too much to have 10 dir x 10 clients, even if there is almost no traffic?
Thx, JD
John Doe wrote:
Hi,
We have a storage server (HP DL360G5 + MSA20 (12 disks in RAID 6) on a SmartArray6400). 10 directories are exported through nfs to 10 clients (rsize=32768,wsize=32768,soft,intr,nosuid,proto=udp,vers=3). The server is apparently not doing much but... we have very high waiting IOs.
How about running iostat -x ? Sounds like the system is doing a lot more than you think it is..
nate
How about running iostat -x ? Sounds like the system is doing a lot more than you think it is..
You might want to set yourself up with a performance monitoring system like Munin to give you more extensive data, as well.
If you get that far, you'll find the iostat plugin to be a bit lacking - I've written a more useful one that I'd be happy to share.
Do you think nfs is the problem here? If so, is there something wrong with our config? Is it too much to have 10 dir x 10 clients, even if there is almost no traffic?
This has very little to do with the number of clients or directories, except that in general terms more users = more work. If the workload is highly random and not easily cached, you could be seeing low throughput but also IOPS starvation. Does 12 disks in Raid6 mean 10 data disks? That's not a lot at all, especially if it is an array that doesn't do much abstraction; any given block will only exist in on a single spindle, creating potential for hot spots if we are talking large sized disks.
Can you do any analysis of the array, specifically looking for hot disks, busy disks, high latencies, etc? Do you know what your baseline should be for this array versus your 10 client workload?
John Doe wrote:
Hi,
We have a storage server (HP DL360G5 + MSA20 (12 disks in RAID 6) on a SmartArray6400). 10 directories are exported through nfs to 10 clients (rsize=32768,wsize=32768,soft,intr,nosuid,proto=udp,vers=3). The server is apparently not doing much but... we have very high waiting IOs.
dstat show very little activity, but high 'wai'...
as others have said, iostat -x N for an N like 5 (5 seconds). ignore the first sample as its the average since boot, instead, look at the ongoing 5 second interval samples, and look for high await, svctm, and %util, as well as the rrqm/s and wrqm/s numbers, rather than sec/s, as sequential access is likely the least of your problems. raid6 does pretty poorly on heavy random write workloads
Also, IBM has a neat freeware system analysis tool called NMON (originally for AIX, ported to Linux) http://www.ibm.com/developerworks/aix/library/au-analyze_aix/ works sorta like a souped up 'top' but has per file system IO stats and stuff too. it can also accumulate stats over a long period into a CSV file, and they have an excel spreadsheet that loads said CSV file and cranks out a lot of fairly useful graphs. dunno if the excel spreadsheet works in OOcalc or not.
John Doe wrote:
Hi,
We have a storage server (HP DL360G5 + MSA20 (12 disks in RAID 6) on a SmartArray6400). 10 directories are exported through nfs to 10 clients (rsize=32768,wsize=32768,soft,intr,nosuid,proto=udp,vers=3). The server is apparently not doing much but... we have very high waiting IOs.
Probably not connected, but personally I would use 'hard' and 'proto=tcp' instead of 'soft' and 'proto=udp' on the clients
James Pearson
James Pearson wrote:
John Doe wrote:
Hi,
We have a storage server (HP DL360G5 + MSA20 (12 disks in RAID 6) on a SmartArray6400). 10 directories are exported through nfs to 10 clients (rsize=32768,wsize=32768,soft,intr,nosuid,proto=udp,vers=3). The server is apparently not doing much but... we have very high waiting IOs.
Probably not connected, but personally I would use 'hard' and 'proto=tcp' instead of 'soft' and 'proto=udp' on the clients
I'd usually blame disk seek time first. If your raid level requires several drives to move their heads together and/or the data layout lands on the same drive set, consider what happens when your 10 users all want the disk head(s) to be in different places. Disk drives allow random access but they really aren't that good at it if they have to spend most of their time seeking. Raid6 is particularly bad at write performance so a different raid level might help - or if you know the data access pattern you might split the drives into different volumes that don't affect each other and arrange the data accordingly. And mounting with the async option might give much better performance - I'm not sure what the default is these days.
# iostat -m -x 10 Linux 2.6.18-8.1.6.el5 (data1.iol) 09/10/2009
. . .
avg-cpu: %user %nice %system %iowait %steal %idle 0.20 0.00 0.31 8.79 0.00 90.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util cciss/c0d0 0.00 0.20 0.92 0.51 0.00 0.00 10.29 0.04 25.29 25.93 3.71 cciss/c0d1 0.00 0.20 3.68 2.45 0.02 0.01 11.07 0.06 9.87 7.27 4.45 cciss/c0d2 0.00 0.20 0.41 2.76 0.00 0.01 8.52 0.03 9.97 2.81 0.89 cciss/c0d3 1.23 0.51 3.98 1.53 0.03 0.01 14.52 0.05 9.69 8.07 4.45 cciss/c0d4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 cciss/c0d5 0.00 0.00 1.02 0.10 0.00 0.00 8.00 0.01 9.36 9.36 1.05 cciss/c0d6 2.45 0.20 0.92 0.51 0.06 0.00 87.43 0.01 9.64 7.21 1.03 cciss/c0d7 0.00 0.00 0.31 0.10 0.00 0.00 8.00 0.01 14.25 14.25 0.58 cciss/c0d8 0.00 0.00 0.10 1.84 0.00 0.01 8.00 0.01 7.26 1.42 0.28 cciss/c0d9 0.00 0.10 0.41 3.78 0.00 0.02 9.56 0.05 12.24 1.59 0.66 cciss/c1d0 0.00 6.03 0.00 1.74 0.00 0.02 26.94 0.04 25.35 12.59 2.19
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.36 25.77 0.00 73.82
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util cciss/c0d0 0.00 0.82 4.60 0.20 0.03 0.00 13.11 0.06 13.55 13.00 6.25 cciss/c0d1 1.23 1.43 1.94 0.20 0.02 0.01 29.33 0.02 11.48 11.48 2.46 cciss/c0d2 0.00 0.00 0.82 0.00 0.00 0.00 8.00 0.01 14.00 14.00 1.15 cciss/c0d3 2.45 1.43 11.25 0.20 0.07 0.01 14.43 0.12 10.62 10.52 12.04 cciss/c0d4 0.00 1.64 7.98 0.20 0.03 0.01 10.00 0.08 9.24 9.10 7.44 cciss/c0d5 5.93 0.20 22.09 0.20 2.19 0.00 201.03 0.58 26.06 1.88 4.19 cciss/c0d6 2.45 1.12 5.62 0.41 0.08 0.01 29.15 0.06 10.66 9.66 5.83 cciss/c0d7 0.00 1.23 5.42 0.20 0.02 0.01 10.76 0.05 8.87 8.73 4.91 cciss/c0d8 0.00 0.72 3.17 0.20 0.02 0.00 13.09 0.04 12.33 12.21 4.12 cciss/c0d9 0.92 0.82 3.17 0.20 0.03 0.00 18.67 0.04 13.12 12.67 4.27 cciss/c1d0 0.00 2.66 0.41 3.07 0.00 0.01 8.29 0.28 81.91 10.94 3.80
avg-cpu: %user %nice %system %iowait %steal %idle 0.10 0.00 0.20 15.95 0.00 83.74
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util cciss/c0d0 0.00 4.19 14.72 0.51 0.10 0.02 15.52 0.17 10.98 9.05 13.79 cciss/c0d1 0.00 0.20 0.31 0.20 0.00 0.00 11.20 0.01 10.20 10.20 0.52 cciss/c0d2 0.00 0.31 0.41 0.41 0.00 0.00 13.00 0.01 7.88 7.88 0.64 cciss/c0d3 0.00 0.31 4.50 0.20 0.02 0.00 9.74 0.05 10.61 10.37 4.88 cciss/c0d4 1.23 0.31 2.76 0.41 0.28 0.00 182.71 0.06 19.97 4.00 1.27 cciss/c0d5 0.00 0.82 3.17 0.20 0.02 0.00 11.64 0.04 11.30 11.30 3.81 cciss/c0d6 2.45 0.51 3.68 0.41 0.28 0.00 143.60 0.07 16.27 5.30 2.17 cciss/c0d7 1.23 0.10 1.94 0.20 0.01 0.00 15.24 0.03 13.33 13.33 2.86 cciss/c0d8 0.00 0.31 0.51 0.41 0.00 0.00 11.56 0.01 8.00 8.00 0.74 cciss/c0d9 0.00 0.10 0.61 0.20 0.00 0.00 10.00 0.01 11.88 11.75 0.96 cciss/c1d0 0.00 3.07 0.00 1.33 0.00 0.01 19.08 0.02 16.77 13.31 1.77
I tried nmon but I did not see anything out of the ordinary... But when I try to see the NFS stats, nmon (11f-1.el5.rf) coredumps. I tried the iostat nfs option (kernel 2.6.18-8.1.6.el5, should support it) but it did not show anything. nfsstat shows no apparent activity... nfs is normaly only used to put files or modify small files (<1K) "from times to times", while http is used to get files.
From: Les Mikesell lesmikesell@gmail.com
I'd usually blame disk seek time first. If your raid level requires several drives to move their heads together and/or the data layout lands on the same drive set, consider what happens when your 10 users all want the disk head(s) to be in different places. Disk drives allow random access but they really aren't that good at it if they have to spend most of their time seeking.
That could be it... It's always a question of space vs speed...
Thx for all the answers, JD
-
Alan McKay -
Eugene Vilensky -
James Pearson -
John Doe -
John R Pierce -
Les Mikesell -
nate