[CentOS] Sluggish server with big array

Tue Jan 15 15:21:35 UTC 2013
Guy Boisvert <boisvert.guy at videotron.ca>

Hi!

     I am experiencing strange behavior with my new CentOS 6.3 
installation (system up to date).  I have a big 11 TB EXT4 array mounted 
on /home/data.  The server is a Tyan 2U (TA26-B3992-E) with Dual Opteron 
2216, one cpu has 4 Gigs RAM and the other has 8 Gigs (so 12 Gigs 
total).  The server uses 8 x Seagate 2TB SAS hard disks (7200 RPM) and 
all the disks are grouped together to form 1 x 500 Gigs RAID 6 array 
(for CentOS) and 1 x 11 TB RAID 6 for data storage (mounted on 
/home/data).  The controller is a LSI SAS 9265-8i:

[root at radium ~]# lspci
00:01.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge
00:02.0 Host bridge: Broadcom BCM5785 [HT1000] Legacy South Bridge
00:02.1 IDE interface: Broadcom BCM5785 [HT1000] IDE
00:02.2 ISA bridge: Broadcom BCM5785 [HT1000] LPC
00:03.0 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01)
00:03.1 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01)
00:03.2 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01)
00:04.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet 
Pro 100 (rev 10)
00:06.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-X bridge (rev b0)
00:07.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-X bridge (rev b0)
00:08.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0)
00:09.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0)
00:0a.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0)
00:0b.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0)
00:0c.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI 
ES1000 (rev 02)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
Miscellaneous Control
01:0d.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge (rev c0)
04:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5780 
Gigabit Ethernet (rev 10)
04:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5780 
Gigabit Ethernet (rev 10)
05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 
[Thunderbolt] (rev 01)


     I have KVM/QEMU running 3 VMs: 2 stored in the 500 gigs array and 
one on the 11 TB array.

     I have a couple of samba shares from the 11 TB array (all of them 
are on the 11TB array).  The strange behavior is that if i "ls -al 
/home/data", i have to wait 7-8 seconds to see the output. If i do a 
simple ls, output is instantaneous.  I see about the same delay when i 
access the samba shares from the server (related ?).  I don't know where 
to start with this!


Here are other informations:

[root at radium data]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_radium-lv_root    50G  7.4G   40G  16% /
tmpfs                 5.9G   88K  5.9G   1% /dev/shm
/dev/sda1             485M   92M  368M  21% /boot
/dev/mapper/vg_radium-lv_home    437G  4.9G  410G   2% /home
/dev/mapper/vg_radium2-lv_data    11T  6.1T  3.7T  63% /home/data


[root at radium data]# uname -a
Linux radium 2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec 19 07:05:20 UTC 
2012 x86_64 x86_64 x86_64 GNU/Linux

[root at radium data]# rpm -qa |grep samba
samba-winbind-clients-3.5.10-125.el6.x86_64
samba-3.5.10-125.el6.x86_64
samba-client-3.5.10-125.el6.x86_64
samba-winbind-3.5.10-125.el6.x86_64
samba-common-3.5.10-125.el6.x86_64

[root at radium data]# pvdisplay
   --- Physical volume ---
   PV Name               /dev/sdb
   VG Name               vg_radium2
   PV Size               10.42 TiB / not usable 4.12 MiB
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              2732799
   Free PE               0
   Allocated PE          2732799
   PV UUID               vTDqS3-wVLz-ZZfu-yfKi-dE6k-rYwr-NVMPcA

   --- Physical volume ---
   PV Name               /dev/sda2
   VG Name               vg_radium
   PV Size               499.51 GiB / not usable 2.00 MiB
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              127874
   Free PE               0
   Allocated PE          127874
   PV UUID               alKttt-X0HC-WT4X-ieju-VHqa-YcvU-gHR6Px




[root at radium ~]# lvdisplay
   --- Logical volume ---
   LV Path                /dev/vg_radium2/lv_data
   LV Name                lv_data
   VG Name                vg_radium2
   LV UUID                pDGfTc-qD3t-p5c3-FDHH-z0HS-pUKq-dClZwM
   LV Write Access        read/write
   LV Creation host, time radium.canal.qc.ca, 2012-11-28 15:52:17 -0500
   LV Status              available
   # open                 1
   LV Size                10.42 TiB
   Current LE             2732799
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:2

   --- Logical volume ---
   LV Path                /dev/vg_radium/lv_root
   LV Name                lv_root
   VG Name                vg_radium
   LV UUID                Lp5Mkd-Ht2J-XBfF-Yafx-w1HY-nIlN-1771T3
   LV Write Access        read/write
   LV Creation host, time radium.canal.qc.ca, 2012-11-28 13:57:36 -0500
   LV Status              available
   # open                 1
   LV Size                50.00 GiB
   Current LE             12800
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:0

   --- Logical volume ---
   LV Path                /dev/vg_radium/lv_home
   LV Name                lv_home
   VG Name                vg_radium
   LV UUID                VCvRSg-x1On-9GgR-09we-yIx6-uMvi-dYO2UZ
   LV Write Access        read/write
   LV Creation host, time radium.canal.qc.ca, 2012-11-28 13:57:40 -0500
   LV Status              available
   # open                 1
   LV Size                443.59 GiB
   Current LE             113560
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:3

   --- Logical volume ---
   LV Path                /dev/vg_radium/lv_swap
   LV Name                lv_swap
   VG Name                vg_radium
   LV UUID                GZorIv-iovS-n2S2-JkSe-bDUl-kwhk-E6potz
   LV Write Access        read/write
   LV Creation host, time radium.canal.qc.ca, 2012-11-28 13:58:01 -0500
   LV Status              available
   # open                 1
   LV Size                5.91 GiB
   Current LE             1514
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:1


I did directly over lvm:
mkfs.ext4 /dev/vg_radium2/lv_data


[root at radium ~]# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) select /dev/sdb
Using /dev/sdb
(parted) print
Error: /dev/sdb: unrecognised disk label


/etc/fstab
=========
/dev/mapper/vg_radium-lv_root                   / ext4    
defaults,acl                    1 1
UUID=2169688f-e830-458d-b31e-25aa0dc9c753       /boot ext4    
defaults                        1 2
/dev/mapper/vg_radium-lv_home                   /home ext4    
defaults                        1 2
/dev/mapper/vg_radium-lv_swap                   swap swap    
defaults                        0 0
tmpfs                                           /dev/shm tmpfs   
defaults                        0 0
devpts                                          /dev/pts devpts  
gid=5,mode=620                  0 0
sysfs                                           /sys sysfs   
defaults                        0 0
proc                                            /proc proc    
defaults                        0 0
/dev/mapper/vg_radium2-lv_data     /home/data      ext4 
defaults,acl,noatime            0 3




Sorry for the long post!


Guy Boisvert
IngTegration inc.
http://www.ingtegration.com