[CentOS] odd inconsistency with nfs

Thu Jul 4 02:22:35 UTC 2013
Miranda Hawarden-Ogata <hawarden at ifa.hawaii.edu>

I'm having an interesting/odd problem with nfs (I think). We recently 
(Monday/Tuesday) upgraded our file server from an ancient redhat 7.3 
system to a shiny new centos 6.4 system. We don't see any issues between 
the other centos boxes, but things get a bit weird when we start 
mounting on the old solaris clients.

The initial symptom was that the 'tab complete' wasn't working, and then 
we noticed that typing 'ls *' in the mounted directory was bombing. I 
tried forcing the mounting back to nfs3 but it's not consistent. I've 
set up two boxes as servers and one of the solaris boxes is my client. 
Each server has two shares that are mounted on the client. Of those 
four, one of them works properly and the other three do not. I've spent 
most of the day trying to debug this and I cannot for the life of me 
tell why one share works and the rest don't. Nothing seems to be special 
about that share versus the rest.

Here are notes on how things are set up.

    on duke: (nis server)
       vi /etc/ypfiles/automap
          scrs1_bolt    -soft,intr,retrans=1 boltzmann:/scrs1_bolt
          summit_bolt   -soft,intr,retrans=1 boltzmann:/summit_bolt
          scrs1.mirror  -soft,intr,retrans=1 goblin:/scrs1.mirror
          summit.mirror -soft,intr,retrans=1 goblin:/summit.mirror
       ( cd /var/yp ; make )

    on boltzmann: (nfs server)
       df -h
          Filesystem            Size  Used Avail Use% Mounted on
          /dev/sdb2              50G   13G   37G  26% /
          tmpfs                 3.9G  1.2M  3.9G   1% /dev/shm
          /dev/sdb3             177G  188M  175G   1% /aux
          /dev/sda3             208G   44G  164G  21% /aux2
       mkdir /aux/scrs1_bolt
       mkdir /aux2/summit_bolt
       ln -s /aux/scrs1_bolt /scrs1_bolt
       ln -s /aux2/summit_bolt /summit_bolt
       chmod 777 /aux/scrs1_bolt /aux2/summit_bolt
       service nfs restart
       vi /etc/exports
          /scrs1_bolt xxx.xxx.xxx.0/24(rw,no_root_squash,sync,insecure)
          /summit_bolt xxx.xxx.xxx.0/24(rw,no_root_squash,sync,insecure)
       exportfs -rv

    on bigdog: (client)
       mkdir /tmp/test/b1 /tmp/test/b2 /tmp/test/g1 /tmp/test/g2
       touch /tmp/test/b1/nothing_is_mounted 
/tmp/test/b2/nothing_is_mounted /tmp/test/g1/nothing_is_mounted 
/tmp/test/g2/nothing_is_mounted
       mount -F nfs -o nfsvers=3 boltzmann:/summit_bolt /tmp/test/b1
       mount -F nfs -o nfsvers=3 boltzmann:/scrs1_bolt /tmp/test/b2
       mount -F nfs -o nfsvers=3 goblin:/summit.mirror /tmp/test/g1
       mount -F nfs -o nfsvers=3 goblin:/scrs1.mirror /tmp/test/g2
       ls -l /tmp/test/*
          -rw-r--r--   1 root     other          0 Jul  3 14:39 
/tmp/test/nothing_is_mounted

          /tmp/test/b1:
          total 24
          -rw-r--r--   1 root     other          0 Jul  3 12:32 SUMMIT_BOLT
          -rw-r--r--   1 root     other          0 Jul  3 09:26 
boltzmann_test_summit

          /tmp/test/b2:
          total 32
          -rw-r--r--   1 root     other          0 Jul  3 12:31 SCRS1_BOLT
          -rw-r--r--   1 root     other          0 Jul  3 09:26 
boltzmann_test_scrs1

          /tmp/test/g1:
          total 280
          -rw-r--r--   1 root     other          0 Jul  3 15:40 
.00_summit_nas_volume
          -rw-rw-r--   1 root     other          0 Jul  3 15:03 
SUMMIT_MIRROR

          /tmp/test/g2:
          total 120
          -rw-r--r--   1 root     other          0 Jul  3 15:40 
.00_scrs1_nas_volume
          -rw-rw-r--   1 root     other          0 Jul  3 15:02 SCRS1_MIRROR

       ls -la /tmp/test/b1/*
          -rw-r--r--   1 root     other          0 Jul  3 12:32 
/tmp/test/b1/SUMMIT_BOLT
          -rw-r--r--   1 root     other          0 Jul  3 09:26 
/tmp/test/b1/boltzmann_test_summit

       ls -l /tmp/test/b2/*
          ls: No match.

       ls -l /tmp/test/g1/*
          ls: No match.

       ls -l /tmp/test/g2/*
          ls: No match.

       mount
          /tmp/test/b1 on boltzmann:/summit_bolt read/write/remote on 
Wed Jul  3 15:41:11 2013
          /tmp/test/b2 on boltzmann:/scrs1_bolt read/write/remote on Wed 
Jul  3 15:41:11 2013
          /tmp/test/g1 on goblin:/summit.mirror read/write/remote on Wed 
Jul  3 15:41:11 2013
          /tmp/test/g2 on goblin:/scrs1.mirror read/write/remote on Wed 
Jul  3 15:41:11 2013

       umount -a /tmp/test/b1 /tmp/test/b2 /tmp/test/g1 /tmp/test/g2

If I use automount to access the shares on the client, the mounts are 
made by default with nfs4 and we see this same 'no match' behaviour. 
It's probably something really stupid but I'm just not seeing it...

If anyone has ideas and/or needs more info, please let me know.

-- 
Thanks!
Miranda