Hi guys,
we are setting NFS server on CentOS7 system. Everything working OK except speed, speed over NFS very drop... if we run dd command directly on server we are getting speed around 1,4Gbps, if we run from client connected to NFS is 200Mbps.
Do you have maybe some advice what we need to check?
Thank you!
Best, Erik
On Sat, 28 May 2022 15:43:12 +0200 Erik Frangež via CentOS wrote:
we are setting NFS server on CentOS7 system. Everything working OK except speed, speed over NFS very drop... if we run dd command directly on server we are getting speed around 1,4Gbps, if we run from client connected to NFS is 200Mbps.
Do you have maybe some advice what we need to check?
Speed of network card in the server. Speed of network card in the client. Speed of any and all routers and switches between the server and the client.
Am 28.05.22 um 18:40 schrieb Frank Cox:
On Sat, 28 May 2022 15:43:12 +0200 Erik Frangež via CentOS wrote:
we are setting NFS server on CentOS7 system. Everything working OK except speed, speed over NFS very drop... if we run dd command directly on server we are getting speed around 1,4Gbps, if we run from client connected to NFS is 200Mbps.
Do you have maybe some advice what we need to check?
Speed of network card in the server. Speed of network card in the client. Speed of any and all routers and switches between the server and the client.
and what config is in place ?
-- Leon
I would start with something like iperf to measure the actual network throughput b/w the client and server. Once you have a baseline for that, we'd have to know things like read and write buffer sizes, and sync vs async mode.
On Sat, May 28, 2022 at 03:43:12PM +0200, Erik Frangež via CentOS wrote:
Hi guys,
we are setting NFS server on CentOS7 system. Everything working OK except speed, speed over NFS very drop... if we run dd command directly on server we are getting speed around 1,4Gbps, if we run from client connected to NFS is 200Mbps.
Do you have maybe some advice what we need to check?
Thank you!
Best, Erik
On 5/28/22 9:43 AM, Erik Frangež via CentOS wrote:
Hi guys,
we are setting NFS server on CentOS7 system. Everything working OK except speed, speed over NFS very drop... if we run dd command directly on server we are getting speed around 1,4Gbps, if we run from client connected to NFS is 200Mbps.
Do you have maybe some advice what we need to check?
Make sure you have good speed of physical connection along the whole path from server to client(s), enable Jumbo packets on all switches along the path. NFS experts will add NFS specific tuning.
Valeri
Thank you!
Best, Erik
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
NFS4.x works best with newer kernels. Maybe you could try with an elrepo kernel.
Increase the number of NFS threads on the NFS server (default is 8, which is too little).
|RPCNFSDCOUNT=32|
This is an example of kernel tunning for 10Gb network, adjust to your setup:
vm.min_free_kbytes = 1048576
vm.swappiness = 10
net.core.somaxconn = 4096
# set max buffers to 64MB net.core.rmem_max = 67108864 net.core.wmem_max = 67108864
# increase Linux autotuning TCP buffer size to 32MB net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_wmem = 4096 65536 33554432
net.core.netdev_max_backlog = 250000
net.ipv4.neigh.default.unres_qlen = 100
net.ipv4.neigh.ens3f0.unres_qlen = 100
net.ipv4.neigh.ens3f1.unres_qlen = 100
fs.file-max = 98584540
You can calculate the buffer size with this tool:
https://www.switch.ch/network/tools/tcp_throughput/?do+new+calculation=do+ne... https://www.switch.ch/network/tools/tcp_throughput/?do+new+calculation=do+new+calculation
Enable readahead for the filesystem for instance:
blockdev --setra 16384 /dev/mapper/mpatha1
for x in sd[a-d]; do blockdev --setra 16384 /dev/$x ; done
It also helps if you disable flowcontrol on the network cards (and switches).
You could also try to reduce the maximum flow rate on the NFS host (set max rate to 8gbit for example)
It would be best to test the actual speed by iperf3 test as someone else already suggested. But since the performance it so bad, I would suspect that the source of the problem is in the the network, not the server.
Cheers,
Barbara
On 5/29/22 01:52, Valeri Galtsev wrote:
On 5/28/22 9:43 AM, Erik Frangež via CentOS wrote:
Hi guys,
we are setting NFS server on CentOS7 system. Everything working OK except speed, speed over NFS very drop... if we run dd command directly on server we are getting speed around 1,4Gbps, if we run from client connected to NFS is 200Mbps.
Do you have maybe some advice what we need to check?
Make sure you have good speed of physical connection along the whole path from server to client(s), enable Jumbo packets on all switches along the path. NFS experts will add NFS specific tuning.
Valeri
Thank you!
Best, Erik
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Barbara Krašovec -
Erik Frangež -
Frank Cox -
Leon Fauster -
Skylar Thompson -
Valeri Galtsev