[CentOS] home directory server performance issues

I’m looking for advice and considerations on how to optimally setup
and deploy an NFS-based home directory server.  In particular: (1) how
to determine hardware requirements, and (2) how to best setup and
configure the server.  We actually have a system in place, but the
performance is pretty bad---the users often experience a fair amount
of lag (1--5 seconds) when doing anything on their home directories,
including an “ls” or writing a small text file.

So now I’m trying to back-up and determine, is it simply a
configuration issue, or is the hardware inadequate?

Our scenario: we have about 25 users, mostly software developers and
analysts.  The users login to one or more of about 40 development
servers.  All users’ home directories live on a single server (no
login except root); that server does an NFSv4 export which is mounted
by all dev servers.  The home directory server hardware is a Dell R510
with dual E5620 CPUs and 8 GB RAM.  There are eight 15k 2.5” 600 GB
drives (Seagate ST3600057SS) configured in hardware RAID-6 with a
single hot spare.  RAID controller is a Dell PERC H700 w/512MB cache
(Linux sees this as a LSI MegaSAS 9260).  OS is CentOS 5.6, home
directory partition is ext3, with options “rw,data=journal,usrquota”.

I have the HW RAID configured to present two virtual disks to the OS:
/dev/sda for the OS (boot, root and swap partitions), and /dev/sdb for
the home directories.  I’m fairly certain I did not align the
partitions optimally:

[root at lnxutil1 ~]# parted -s /dev/sda unit s print

Model: DELL PERC H700 (scsi)
Disk /dev/sda: 134217599s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start    End         Size        Type     File system  Flags
 1      63s      465884s     465822s     primary  ext2         boot
 2      465885s  134207009s  133741125s  primary               lvm

[root at lnxutil1 ~]# parted -s /dev/sdb unit s print

Model: DELL PERC H700 (scsi)
Disk /dev/sdb: 5720768639s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name  Flags
 1      34s    5720768606s  5720768573s                     lvm


Can anyone confirm that the partitions are not aligned correctly, as I
suspect?  If this is true, is there any way to *quantify* the effects
of partition mis-alignment on performance?  In other words, what kind
of improvement could I expect if I rebuilt this server with the
partitions aligned optimally?

In general, what is the best way to determine the source of our
performance issues?  Right now, I’m running “iostat -dkxt 30”
re-directed to a file.  I intend to let this run for a day or so, and
write a script to produce some statistics.

Here is one iteration from the iostat process:

Time: 09:37:28 AM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00    44.09  0.03 107.76     0.13   607.40
11.27     0.89    8.27   7.27  78.35
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
sda2              0.00    44.09  0.03 107.76     0.13   607.40
11.27     0.89    8.27   7.27  78.35
sdb               0.00  2616.53  0.67 157.88     2.80 11098.83
140.04     8.57   54.08   4.21  66.68
sdb1              0.00  2616.53  0.67 157.88     2.80 11098.83
140.04     8.57   54.08   4.21  66.68
dm-0              0.00     0.00  0.03 151.82     0.13   607.26
8.00     1.25    8.23   5.16  78.35
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
dm-2              0.00     0.00  0.67 2774.84     2.80 11099.37
8.00   474.30  170.89   0.24  66.84
dm-3              0.00     0.00  0.67 2774.84     2.80 11099.37
8.00   474.30  170.89   0.24  66.84


What I observe, is that whenever sdb (home directory partition)
becomes loaded, sda (OS) often does as well.  Why is this?  I would
expect sda to generally be idle, or have minimal utilization.
According to both “free” and “vmstat”, this server is not swapping at
all.

At one point, our problems were due to a random user writing a huge
file to their home directory.  We built a second server specifically
for people to use for writing large temporary files.  Furthermore, for
all the dev servers, I used the following tc commands to rate limit
how quickly any one server can write to the home directory server (8
Mbps or 1 MB/s):

ETH_IFACE=$( route -n | grep "^0.0.0.0" | awk '{ print $8 }' )
IFACE_RATE=1000mbit
LIMIT_RATE=8mbit
TARGET_IP=1.2.3.4 # home directory server IP
tc qdisc add dev $ETH_IFACE root handle 1: htb default 1
tc class add dev $ETH_IFACE parent 1: classid 1:1 htb rate $IFACE_RATE
ceil $IFACE_RATE
tc class add dev $ETH_IFACE parent 1: classid 1:2 htb rate $LIMIT_RATE
ceil $LIMIT_RATE
tc filter add dev $ETH_IFACE parent 1: protocol ip prio 16 u32 match
ip dst $TARGET_IP flowid 1:2

The other interesting thing is that the second server I mentioned—the
one specifically designed for users to “torture”—shows very low IO
utilization, practically never going above 10%.  That server is fairly
different though: dual E5-2340 CPUs (more cores, but lower clock), 32
GB RAM.  Disk subsystem is Dell PERC 710 (LSI MegaRAID SAS 2208), and
drives are 7200 RPM 1GB (SEAGATE ST1000NM0001) in RAID-6.  The OS is
CentOS 6.3, NFS partition is ext4 with options
“rw,relatime,barrier=1,data=ordered,usrquota”.

Ultimately, I plan to rebuild the home directory server with CentOS 6
(instead of 5), and align the partitions properly.  But as of now, I
don’t have a rational reason for doing that other than the other
server with this config doesn’t have performance problems.  I’d like
to be able to say specifically (i.e. quantify) exactly where the
problems are and how they will be addressed by the upgrade/config
change.

I’ll add that we want to use the “sec=krb5p” (i.e. encrypt everything)
mount option for the home directories.  We tried that with the home
directory server, and it became virtually unusable.  But we use that
option on the other server, with no issue.  For now, as a stop-gap, we
are just using the “sec=krb5” mount option (i.e., no encryption).  The
server is still laggy, but at least usable.

Here is the output of “nfsstat –v” on the home directory server:
[root at lnxutil1 ~]# nfsstat -v
Server packet stats:
packets    udp        tcp        tcpconn
12560989   0          12544002   17146

Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
12516995   922        0          922        0

Server reply cache:
hits       misses     nocache
0          0          12512316

Server file handle cache:
lookup     anon       ncachedir  ncachedir  stale
0          0          0          0          160

Server nfs v4:
null         compound
86        0% 12516096 99%

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit
0         0% 0         0% 0         0% 449630    1% 1131528   2% 191998    0%
create       delegpurge   delegreturn  getattr      getfh        link
2053      0% 0         0% 62931     0% 11210081 29% 1638995   4% 275       0%
lock         lockt        locku        lookup       lookup_root  nverify
196       0% 0         0% 196       0% 557606    1% 0         0% 0         0%
open         openattr     open_conf    open_dgrd    putfh        putpubfh
1274780   3% 0         0% 72561     0% 618       0% 12357089 32% 0         0%
putrootfh    read         readdir      readlink     remove       rename
160       0% 1548999   4% 44760     0% 625       0% 140946    0% 4229      0%
renew        restorefh    savefh       secinfo      setattr      setcltid
134103    0% 1157086   3% 1281276   3% 0         0% 133212    0% 143       0%
setcltidconf verify       write        rellockowner
113       0% 0         0% 4896102  12% 196       0%


Let me know if I can provide any more useful information.  Thanks in
advance for any pointers!