On Mon, Oct 24, 2016 at 7:51 AM, mark <m.roth at 5-cent.us> wrote: > On 10/24/16 03:52, Larry Martell wrote: >> >> On Fri, Oct 21, 2016 at 11:42 AM, <m.roth at 5-cent.us> wrote: >>> >>> Larry Martell wrote: >>>> >>>> On Fri, Oct 21, 2016 at 11:21 AM, <m.roth at 5-cent.us> wrote: >>>>> >>>>> Larry Martell wrote: >>>>>> >>>>>> We have 1 system ruining Centos7 that is the NFS server. There are 50 >>>>>> external machines that FTP files to this server fairly continuously. >>>>>> >>>>>> We have another system running Centos6 that mounts the partition the >>>>>> files are FTP-ed to using NFS. > > <snip> >>>>> >>>>> What filesystem? > > <snip> >>> >>> cat /etc/fstab on the systems, and see what they are. If either is xfs, >>> and assuming that the systems are on UPSes, then the fstab which controls >>> drive mounting on a system should have, instead of "defaults", >>> nobarrier,inode64. >> >> >> The server is xfs (the client is nfs). The server does have inode64 >> specified, but not nobarrier. >> >>> Note that the inode64 is relevant if the filesystem is > 2TB. >> >> >> The file system is 51TB. >> >>> The reason I say this is that we we started rolling out CentOS 7, we >>> tried >>> to put one of our user's home directory on one, and it was a disaster. >>> 100% repeatedly, untarring a 100M tarfile onto an nfs-mounted drive took >>> seven minutes, where before, it had taken 30 seconds. Timed. It took us >>> months to discover that NFS 4 tries to make transactions atomic, which is >>> fine if you're worrying about losing power or connectivity. If you're on >>> a >>> UPS, and hardwired, adding the nobarrier immediately brought it down to >>> 40 >>> seconds or so. >> >> >> We are not seeing a performance issue - do you think nobarrier would >> help with our lock up issue? I wanted to try it but my client did not >> want me to make any changes until we got the bad disk replaced. >> Unfortunately that will not happen until Wednesday. > > > Absolutely add nobarrier, and see what happens. Finally got to add nobarrier (I'll skip why it took so long), and it looks like this just caused the problem to morph a bit. On the C7 NFS server, besides having 50 external machines ftp-ing files to it, we run 2 jobs: 1 that moves files around (called image_mover) and one that changes perms on some files (called chmod_job). And on the C6 NFS client, besides the job that was hanging (called the importer), we also run a another job (called ftp_job) that ftps files to the C6 machine. The ftp_job had never hung before, but now the importer that used to hang has not (yet) hung, and the ftp_job that had not hung before now is hanging. But the system messages are different. On the C7 server there is a series of messages of the form 'task blocked for >120 seconds' with a stack trace. There is one for each of the following: nfsd, chmod_job, kworker, pure_ftpd, image_mover In each of the stack traces they are blocked on either nfs_write or nfs_flush And on the C6 client there is a similar blocked message for the ftp job, blocked on nfs_flush, then the bad sequence number message I had seen before, and at that point the ftp_job hung.