On Sun, Oct 23, 2016 at 9:02 AM, Larry Martell <larry.martell at gmail.com> wrote: > Hi Matt- > > Thank you for this very detailed and thoughtful reply. > > On Fri, Oct 21, 2016 at 4:43 PM, Matt Garman <matthew.garman at gmail.com> wrote: >> On Fri, Oct 21, 2016 at 4:14 AM, Larry Martell <larry.martell at gmail.com> wrote: >>> We have 1 system ruining Centos7 that is the NFS server. There are 50 >>> external machines that FTP files to this server fairly continuously. >>> >>> We have another system running Centos6 that mounts the partition the files >>> are FTP-ed to using NFS. >>> >>> There is a python script running on the NFS client machine that is reading >>> these files and moving them to a new dir on the same file system (a mv not >>> a cp). >> >> To be clear: the python script is moving files on the same NFS file >> system? E.g., something like >> >> mv /mnt/nfs-server/dir1/file /mnt/nfs-server/dir2/file >> >> where /mnt/nfs-server is the mount point of the NFS server on the >> client machine? > > Correct. > >> Or are you moving files from the CentOS 7 NFS server to the CentOS 6 NFS client? > > No the files are FTP-ed to the CentOS 7 NFS server and then processed > and moved on the CentOS 6 NFS client. > >> If the former, i.e., you are moving files to and from the same system, >> is it possible to completely eliminate the C6 client system, and just >> set up a local script on the C7 server that does the file moves? That >> would cut out a lot of complexity, and also improve performance >> dramatically. > > The problem doing that is the files are processed and loaded to MySQL > and then moved by a script that uses the Django ORM, and neither > django, nor any of the other python packages needed are installed on > the server. And since the server does not have an external internet > connection (as I mentioned in my reply to Mark) getting it set up > would require a large amount of effort. > > Also, we have this exact same setup on over 10 other systems, and it > is only this one that is having a problem. The one difference with > this one is that the sever is CentOS7 - on all the other systems both > the NFS server and client are CentOS6. > >> Also, what is the size range of these files? Are they fairly small >> (e.g. 10s of MB or less), medium-ish (100s of MB) or large (>1GB)? > > Small - They range in size from about 100K to 6M. > >>> Almost daily this script hangs while reading a file - sometimes it never >>> comes back and cannot be killed, even with -9. Other times it hangs for 1/2 >>> hour then proceeds on. >> >> Timeouts relating to NFS are the worst. >> >> >>> Coinciding with the hanging I see this message on the NFS server host: >>> >>> nfsd: peername failed (error 107) >>> >>> And on the NFS client host I see this: >>> >>> nfs: V4 server returned a bad sequence-id >>> nfs state manager - check lease failed on NFSv4 server with error 5 >> >> I've been wrangling with NFS for years, but unfortunately those >> particular messages don't ring a bell. >> >> The first thing that came to my mind is: how does the Python script >> running on the C6 client know that the FTP upload to the C7 server is >> complete? In other words, if someone is uploading "fileA", and the >> Python script starts to move "fileA" before the upload is complete, >> then at best you're setting yourself up for all kinds of confusion, >> and at worst file truncation and/or corruption. > > The python script checks the modification time of the file, and only > if it has not been modified in more then 2 minutes does it process it. > Otherwise it skips it and waits for the next run to potentially > process it. Also, the script can tell if the file is incomplete in a > few different ways. So if it has not been modified in more then 2 > minutes, the script starts to process it, but if it finds that it's > incomplete it aborts the processing and leaves it for next time. > >> Making a pure guess about those particular errors: is there any chance >> there is a network issue between the C7 server and the C6 client? >> What is the connection between those two servers? Are they physically >> adjacent to each other and on the same subnet? Or are they on >> opposite ends of the globe connected through the Internet? > > Actually both the client and server are virtual machines running on > one physical machine. The physical machine is running CentOS7. There > is nothing else running on the physical machine other then the 2 VMs. I misspoke here - the CentOS7 NFS server is running on the physical hardware, it's not a VM. The CentOS6 client is a VM. > >> Clearly two machines on the same subnet, separated only by one switch >> is the simplest case (i.e. the kind of simple LAN one might have in >> his home). But once you start crossing subnets, then routing configs >> come into play. And maybe you're using hostnames rather than IP >> addresses directly, so then name resolution comes into play (DNS or >> /etc/hosts). And each switch hop you add requires that not only your >> server network config needs to be correct, but also your switch config >> needs to be correct as well. And if you're going over the Internet, >> well... I'd probably try really hard to not use NFS in that case! :) >> >> Do you know if your NFS mount is using TCP or UDP? On the client you >> can do something like this: >> >> grep nfs /proc/mounts | less -S >> >> And then look at what the "proto=XXX" says. I expect it will be >> either "tcp" or "udp". If it's UDP, modify your /etc/fstab so that >> the options for that mountpoint include "proto=tcp". I *think* the >> default is now TCP, so this may be a non-starter. But the point is, >> based purely on the conjecture that you might have an unreliable >> network, TCP would be a better fit. > > I assume TCP, but I will check tomorrow when I am on site. > >> I hate to simply say "RTFM", but NFS is complex, and I still go back >> and re-read the NFS man page ("man nfs"). This document is long and >> very dense, but it's worth at least being familiar with its content. > > Yes, I agree. I skimmed it last week, but I will look at it in detail tomorrow. > >>> The first client message is always at the same time as the hanging starts. >>> The second client message comes 20 minutes later. >>> The server message comes 4 minutes after that. >>> Then 3 minutes later the script un-hangs (if it's going to). >> >> In my experience, delays that happen on consistent time intervals that >> are on the order of minutes tend to smell of some kind of timeout >> scenario. So the question is, what triggers the timeout state? >> >>> Can anyone shed any light on to what could be happening here and/or what I >>> could do to alleviate these issues and stop the script from hanging? >>> Perhaps some NFS config settings? We do not have any, so we are using the >>> defaults. >> >> My general rule of thumb is "defaults are generally good enough; make >> changes only if you understand their implications and you know you >> need them (or temporarily as a diagnostic tool)". > > I would like to try increasing the timeout. > >> But anyway, my hunch is that there might be a network issue. So I'd >> actually start with basic network troubleshooting. Do an "ifconfig" >> on both machines: do you see any drops or interface errors? Do >> "ethtool <interface>" on both machines to make sure both are linked up >> at the correct speed and duplex. Use a tool like netperf to check >> bandwidth between both hosts. Look at the actual detailed stats, do >> you see huge outliers or timeouts? Do the test with both TCP and UDP, >> performance should be similar with a (typically) slight gain with UDP. >> Do you see drops with UDP? >> >> What's the state of the hardware? Are they ancient machines cobbled >> together from spare parts, or reasonable decent machines? Do they >> have adequate cooling and power? Is there any chance they are >> overheating (even briefly) or possibly being fed unclean power (e.g. >> small voltage aberrations)? > > The hardware is new, and is in a rack in a server room with adequate > and monitored cooling and power. But I just found out from someone on > site that there is a disk failure, which happened back on Sept 3. The > system uses RAID, but I don't know what level. I was told it can > tolerate 3 disk failures and still keep working, but personally, I > think all bets are off until the disk has been replaced. That should > happen in the next day or 2, so we shall see. > >> Oh, also, look at the load on the two machines... are these >> purpose-built servers, or are they used for other numerous tasks? >> Perhaps one or both is overloaded. top is the tool we use >> instinctively, but also take a look at vmstat and iostat. Oh, also >> check "free", make sure neither machine is swapping (thrashing). > > I've been watching and monitoring the machines for 2 days and neither > one has had a large CPU load, not has been using much memory. > >> If >> you're not already doing this, I would recommend setting up "sar" >> (from the package "sysstat") and setting up more granular logging than >> the default. sar is kind of like a continuous >> iostat+free+top+vmstat+other system load tools rolled into one that >> continually writes this information to a database. So for example, >> next time this thing happens, you can look at the sar logs to see if >> any particular metric went significantly out-of-whack. > > That is a good idea, I will check those logs, and set up better logging. > >> >> That's all I can think of for now. Best of luck. You have my >> sympathy... I've been administering Linux both as a hobby and >> professionally for longer than I care to admit, and NFS still scares >> me. Just be thankful you're not using Kerberized NFS. ;) > > Thanks! > Larry