Anyone seen this before?
I have a number of file systems nfs mounted onto clients running various versions of CentOS (and Upstream), although mostly they are v5.x flavors. =20
The server is a Network Appliance filer.
When the build process for this team runs, it sometimes dies because it can't find files in the automounter tree; if the engineer checks, he sometimes sees a problem, and sometimes doesn't; for example:
[pdbuild@build-c5u1: ~] ll /tools/vault/kernels ls: /tools/vault/kernels: No such file or directory [pdbuild@build-c5u1: ~] ll /tools/vault/kernels total 152K drwxr-xr-x 4 pdbuild everyone 4.0K Aug 7 2007 2.4.21-40.EL.CUSTOM.01smp/ drwxr-xr-x 7 pdbuild everyone 4.0K Jun 26 2007 2.6.16.33-1-xen/ drwxr-xr-x 7 pdbuild everyone 4.0K Aug 7 2007 2.6.16.33-xen/ drwxr-xr-x 4 pdbuild everyone 4.0K Jan 3 2008 2.6.18-53.1.4.el5/ drwxr-xr-x 4 pdbuild everyone 4.0K Feb 11 2008 2.6.18-53.1.6.el5/ drwxr-xr-x 4 pdbuild everyone 4.0K Nov 15 2007 2.6.18-53.el5/ drwxr-xr-x 4 pdbuild everyone 4.0K Sep 24 2007 2.6.18-8.el5/ drwxr-xr-x 7 pdbuild everyone 4.0K Oct 16 2007 2.6.18-xen/ [pdbuild@build-c5u1: ~] df /tools/vault/kernels Filesystem Size Used Avail Use% Mounted on nas02:/vol/tools/vault 779G 655G 124G 85% /tools/vault
Now I've seen this before where some processes don't wait for the automount= er to do its thing before continuing; they just report "fail" and move on to the failure handling.
I'm guessing that I need some magic on the automounter configuration to change this behavior, can anyone point me in the right direction?
Thanks for your time.
On Sun, Feb 01, 2009, David.Mackintosh@xdroop.com wrote:
Anyone seen this before?
I have a number of file systems nfs mounted onto clients running various versions of CentOS (and Upstream), although mostly they are v5.x flavors. =20
The server is a Network Appliance filer.
When the build process for this team runs, it sometimes dies because it can't find files in the automounter tree; if the engineer checks, he sometimes sees a problem, and sometimes doesn't; for example:
I have never used the automounter, but use the amd automounter in the am- utils package. Largely that's because that's what I know from my Caldera and SuSE days, and we have it running on OS X and FreeBSD as well.
... Bill
David.Mackintosh@xdroop.com wrote:
Anyone seen this before?
I have a number of file systems nfs mounted onto clients running various versions of CentOS (and Upstream), although mostly they are v5.x flavors. =20
The server is a Network Appliance filer.
When the build process for this team runs, it sometimes dies because it can't find files in the automounter tree; if the engineer checks, he sometimes sees a problem, and sometimes doesn't; for example:
[pdbuild@build-c5u1: ~] ll /tools/vault/kernels ls: /tools/vault/kernels: No such file or directory [pdbuild@build-c5u1: ~] ll /tools/vault/kernels total 152K
Any errors in syslog ? At my company we run automunter as well mostly on CentOS 4.x, and RHEL 4.x, though some newer systems are on 5.x. The only time I get that kind of error is if a firewall is blocking the connections, and there is always an error in syslog saying that it couldn't connect to the NAS.
e.g.
Jan 22 02:52:24 pd1-cas01 automount[32591]: >> mount: mount to NFS server 'exnas' failed: timed out (giving up). Jan 22 02:58:42 pd1-cas01 automount[23530]: >> mount: backgrounding "exnas:/exavol/system_logs/syslogs"
How many systems do you have mounting the NetApp? Here we are migrating from a BlueArc based system to an Exanet based system, NetApp's architecture limits it's performance too much for our needs(and price point). Maybe with OnTap 8.0 with the integration of GX stuff it will be better.
nate
David.Mackintosh@xdroop.com wrote:
I'm guessing that I need some magic on the automounter configuration to change this behavior, can anyone point me in the right direction?
Also what mount options are you using? I'm sure the ones that NetApp suggests.
These are what Exanet suggests for their product which we use: -rw,bg,hard,intr,tcp,vers=3,timeo=2,retrans=10,rsize=32768,wsize=32768
These are what we use for BlueArc(I asume what they suggested at the time the system is very old and was installed years before I got here): -rw,proto=tcp,vers=3,rsize=32768,wsize=32768,timeo=600,hard,intr,bg
nate
David.Mackintosh@xdroop.com wrote:
Anyone seen this before?
I have a number of file systems nfs mounted onto clients running various versions of CentOS (and Upstream), although mostly they are v5.x flavors. =20
The server is a Network Appliance filer.
When the build process for this team runs, it sometimes dies because it can't find files in the automounter tree; if the engineer checks, he sometimes sees a problem, and sometimes doesn't; for example:
[pdbuild@build-c5u1: ~] ll /tools/vault/kernels ls: /tools/vault/kernels: No such file or directory [pdbuild@build-c5u1: ~] ll /tools/vault/kernels total 152K drwxr-xr-x 4 pdbuild everyone 4.0K Aug 7 2007 2.4.21-40.EL.CUSTOM.01smp/ ... drwxr-xr-x 7 pdbuild everyone 4.0K Oct 16 2007 2.6.18-xen/ [pdbuild@build-c5u1: ~] df /tools/vault/kernels Filesystem Size Used Avail Use% Mounted on nas02:/vol/tools/vault 779G 655G 124G 85% /tools/vault
Now I've seen this before where some processes don't wait for the automounter to do its thing before continuing; they just report "fail" and move on to the failure handling.
What sort of map is being used for this mount point? i.e. what is the contents of your /etc/auto.master ?
James Pearson
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of James Pearson Sent: Monday, February 02, 2009 12:54 PM To: CentOS mailing list Subject: Re: [CentOS] Automounter issue
David.Mackintosh@xdroop.com wrote:
Anyone seen this before?
I have a number of file systems nfs mounted onto clients running various versions of CentOS (and Upstream), although mostly they are v5.x flavors. =20
The server is a Network Appliance filer.
When the build process for this team runs, it sometimes dies because it can't find files in the automounter tree; if the engineer checks, he sometimes sees a problem, and sometimes doesn't; for example:
[pdbuild@build-c5u1: ~] ll /tools/vault/kernels ls: /tools/vault/kernels: No such file or directory [pdbuild@build-c5u1: ~] ll /tools/vault/kernels total 152K drwxr-xr-x 4 pdbuild everyone 4.0K Aug 7 2007
2.4.21-40.EL.CUSTOM.01smp/
... drwxr-xr-x 7 pdbuild everyone 4.0K Oct 16 2007 2.6.18-xen/ [pdbuild@build-c5u1: ~] df /tools/vault/kernels Filesystem Size Used Avail Use% Mounted on nas02:/vol/tools/vault 779G 655G 124G 85% /tools/vault
Now I've seen this before where some processes don't wait
for the automounter
to do its thing before continuing; they just report "fail"
and move on
to the failure handling.
What sort of map is being used for this mount point? i.e. what is the contents of your /etc/auto.master ?
James Pearson
----
I have however have come to the conclusion that it is Automount itself causing this problem. I also have this problem with mounting Samba Shares. The whole client machine would lock up or either have the same error your getting. After a few hours of trying different settings I finally went to mounting the shares at boot time. My question would be is there certain settings that automount supports for auto.master for cifs and nfs verses the running Kernel.
JohnStanley
Picking up a couple of outstanding questions:
On Mon, Feb 02, 2009 at 05:53:59PM +0000, James Pearson wrote:
What sort of map is being used for this mount point? i.e. what is the contents of your /etc/auto.master ?
# ypcat -k auto.master | grep tools /tools auto.tools -hard,bg,nfsvers=3,tcp,intr,rsize=32786,wsize=32768,nosuid
There's nothing in the messages file; I am now sending *.debug to my syslog host to see if anything interesting shows up.
Nate asked how many clients are mounting; around 300 systems have the automounter map, but only 30-50 nodes can be expected to be actually using it.
Closer inspection reveals that the two systems which are having the problem the most are both v5u1. I'm starting to suspect that the root cause is an automounter issue in 5u1.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of David.Mackintosh@xdroop.com Sent: Monday, February 02, 2009 2:15 PM To: CentOS mailing list Subject: Re: [CentOS] Automounter issue
Picking up a couple of outstanding questions:
On Mon, Feb 02, 2009 at 05:53:59PM +0000, James Pearson wrote:
What sort of map is being used for this mount point? i.e.
what is the
contents of your /etc/auto.master ?
# ypcat -k auto.master | grep tools /tools auto.tools -hard,bg,nfsvers=3,tcp,intr,rsize=32786,wsize=32768,nosuid
There's nothing in the messages file; I am now sending *.debug to my syslog host to see if anything interesting shows up.
Nate asked how many clients are mounting; around 300 systems have the automounter map, but only 30-50 nodes can be expected to be actually using it.
Closer inspection reveals that the two systems which are having the problem the most are both v5u1. I'm starting to suspect that the root cause is an automounter issue in 5u1.
---------- Very Odd Indeed. I was also running 5.1 at that time also...Hmm maybe I'll give it a whirl on the latest of 5.2
JohnStanley
Ahh, here we go:
https://bugzilla.redhat.com/show_bug.cgi?id=371341 “upgrade to 5.1 breaks autofs for automounted home directories” – see especially comment #13 which describes the symptom, and #25 which explains what the fix is hidden in the records as. Fix appears to be to use kernel kernel-2.6.18-53.1.4 or higher.
-
Bill Campbell -
David.Mackintosh＠xdroop.com -
James Pearson -
John -
nate