Hi!
I am experiencing strange behavior with my new CentOS 6.3 installation (system up to date). I have a big 11 TB EXT4 array mounted on /home/data. The server is a Tyan 2U (TA26-B3992-E) with Dual Opteron 2216, one cpu has 4 Gigs RAM and the other has 8 Gigs (so 12 Gigs total). The server uses 8 x Seagate 2TB SAS hard disks (7200 RPM) and all the disks are grouped together to form 1 x 500 Gigs RAID 6 array (for CentOS) and 1 x 11 TB RAID 6 for data storage (mounted on /home/data). The controller is a LSI SAS 9265-8i:
[root@radium ~]# lspci 00:01.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge 00:02.0 Host bridge: Broadcom BCM5785 [HT1000] Legacy South Bridge 00:02.1 IDE interface: Broadcom BCM5785 [HT1000] IDE 00:02.2 ISA bridge: Broadcom BCM5785 [HT1000] LPC 00:03.0 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01) 00:03.1 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01) 00:03.2 USB controller: Broadcom BCM5785 [HT1000] USB (rev 01) 00:04.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 10) 00:06.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-X bridge (rev b0) 00:07.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-X bridge (rev b0) 00:08.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0) 00:09.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0) 00:0a.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0) 00:0b.0 PCI bridge: Broadcom BCM5780 [HT2000] PCI-Express Bridge (rev b0) 00:0c.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI ES1000 (rev 02) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:0d.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge (rev c0) 04:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet (rev 10) 04:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet (rev 10) 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 01)
I have KVM/QEMU running 3 VMs: 2 stored in the 500 gigs array and one on the 11 TB array.
I have a couple of samba shares from the 11 TB array (all of them are on the 11TB array). The strange behavior is that if i "ls -al /home/data", i have to wait 7-8 seconds to see the output. If i do a simple ls, output is instantaneous. I see about the same delay when i access the samba shares from the server (related ?). I don't know where to start with this!
Here are other informations:
[root@radium data]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_radium-lv_root 50G 7.4G 40G 16% / tmpfs 5.9G 88K 5.9G 1% /dev/shm /dev/sda1 485M 92M 368M 21% /boot /dev/mapper/vg_radium-lv_home 437G 4.9G 410G 2% /home /dev/mapper/vg_radium2-lv_data 11T 6.1T 3.7T 63% /home/data
[root@radium data]# uname -a Linux radium 2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec 19 07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@radium data]# rpm -qa |grep samba samba-winbind-clients-3.5.10-125.el6.x86_64 samba-3.5.10-125.el6.x86_64 samba-client-3.5.10-125.el6.x86_64 samba-winbind-3.5.10-125.el6.x86_64 samba-common-3.5.10-125.el6.x86_64
[root@radium data]# pvdisplay --- Physical volume --- PV Name /dev/sdb VG Name vg_radium2 PV Size 10.42 TiB / not usable 4.12 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 2732799 Free PE 0 Allocated PE 2732799 PV UUID vTDqS3-wVLz-ZZfu-yfKi-dE6k-rYwr-NVMPcA
--- Physical volume --- PV Name /dev/sda2 VG Name vg_radium PV Size 499.51 GiB / not usable 2.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 127874 Free PE 0 Allocated PE 127874 PV UUID alKttt-X0HC-WT4X-ieju-VHqa-YcvU-gHR6Px
[root@radium ~]# lvdisplay --- Logical volume --- LV Path /dev/vg_radium2/lv_data LV Name lv_data VG Name vg_radium2 LV UUID pDGfTc-qD3t-p5c3-FDHH-z0HS-pUKq-dClZwM LV Write Access read/write LV Creation host, time radium.canal.qc.ca, 2012-11-28 15:52:17 -0500 LV Status available # open 1 LV Size 10.42 TiB Current LE 2732799 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2
--- Logical volume --- LV Path /dev/vg_radium/lv_root LV Name lv_root VG Name vg_radium LV UUID Lp5Mkd-Ht2J-XBfF-Yafx-w1HY-nIlN-1771T3 LV Write Access read/write LV Creation host, time radium.canal.qc.ca, 2012-11-28 13:57:36 -0500 LV Status available # open 1 LV Size 50.00 GiB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0
--- Logical volume --- LV Path /dev/vg_radium/lv_home LV Name lv_home VG Name vg_radium LV UUID VCvRSg-x1On-9GgR-09we-yIx6-uMvi-dYO2UZ LV Write Access read/write LV Creation host, time radium.canal.qc.ca, 2012-11-28 13:57:40 -0500 LV Status available # open 1 LV Size 443.59 GiB Current LE 113560 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:3
--- Logical volume --- LV Path /dev/vg_radium/lv_swap LV Name lv_swap VG Name vg_radium LV UUID GZorIv-iovS-n2S2-JkSe-bDUl-kwhk-E6potz LV Write Access read/write LV Creation host, time radium.canal.qc.ca, 2012-11-28 13:58:01 -0500 LV Status available # open 1 LV Size 5.91 GiB Current LE 1514 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1
I did directly over lvm: mkfs.ext4 /dev/vg_radium2/lv_data
[root@radium ~]# parted GNU Parted 2.1 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) select /dev/sdb Using /dev/sdb (parted) print Error: /dev/sdb: unrecognised disk label
/etc/fstab ========= /dev/mapper/vg_radium-lv_root / ext4 defaults,acl 1 1 UUID=2169688f-e830-458d-b31e-25aa0dc9c753 /boot ext4 defaults 1 2 /dev/mapper/vg_radium-lv_home /home ext4 defaults 1 2 /dev/mapper/vg_radium-lv_swap swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/mapper/vg_radium2-lv_data /home/data ext4 defaults,acl,noatime 0 3
Sorry for the long post!
Guy Boisvert IngTegration inc. http://www.ingtegration.com
On Tue, 15 Jan 2013, Guy Boisvert wrote:
I have a couple of samba shares from the 11 TB array (all ofthem are on the 11TB array). The strange behavior is that if i "ls -al /home/data", i have to wait 7-8 seconds to see the output. If i do a simple ls, output is instantaneous. I see about the same delay when i access the samba shares from the server (related ?). I don't know where to start with this!
This sounds like it might be an issue with cacheing user info.
Do you have a remote authentication server, e.g., LDAP? Are you running nscd or sssd? Does re-starting the cacheing daemon help?
Le 2013-01-15 10:35, Paul Heinlein a écrit :
On Tue, 15 Jan 2013, Guy Boisvert wrote:
I have a couple of samba shares from the 11 TB array (all of themare on the 11TB array). The strange behavior is that if i "ls -al /home/data", i have to wait 7-8 seconds to see the output. If i do a simple ls, output is instantaneous. I see about the same delay when i access the samba shares from the server (related ?). I don't know where to start with this!
This sounds like it might be an issue with cacheing user info.
Do you have a remote authentication server, e.g., LDAP? Are you running nscd or sssd? Does re-starting the cacheing daemon help?
Hi Paul,
Samba is hooked to Active Directory using Winbind. sssd is off and nscd doesn't seems to be there. As for AD (Active Directory), a simple getent passwd shows all local accounts and very quickly the AD accounts. I read in the Ubuntu mailing lists that there seemed to be some bugs with EXT4 and large arrays (very slow mounts). There was a message saying that finally the slow mount problems was fixed with kernel 3.6!
Thanks.
Le 2013-01-15 10:35, Paul Heinlein a écrit :
On Tue, 15 Jan 2013, Guy Boisvert wrote:
I have a couple of samba shares from the 11 TB array (all of themare on the 11TB array). The strange behavior is that if i "ls -al /home/data", i have to wait 7-8 seconds to see the output. If i do a simple ls, output is instantaneous. I see about the same delay when i access the samba shares from the server (related ?). I don't know where to start with this!
This sounds like it might be an issue with cacheing user info.
Do you have a remote authentication server, e.g., LDAP? Are you running nscd or sssd? Does re-starting the cacheing daemon help?
Hi Paul,
You were right: I turned off SMB, NMB & Winbind and ls -al on the large array become fast as usual. Digging a little deeper, i found that some of the sub-directories in /home/data had been restored from another server and that they showed in unknown group "10432" in ls -al . So i just did:
[root@radium home]# chgrp -R "domain users" /home/data
So in conclusion, it seems that my "ls -al" on the filesystem containing unknown GID translated into queries to Active Directory and for whatever reason, resulted in a delay of several seconds (like 10-15 seconds).
Thanks Paul again.
Guy Boisvert IngTegration inc. http://www.ingtegration.com
-
Guy Boisvert -
Paul Heinlein