On Mon, 14 Dec 2020 14:46:30 +0100 Patrick Bégou <Patrick.Begou at legi.grenoble-inp.fr> wrote: > Hi, > > I'm deploying a CentOS8 (not stream 😁) cluster and I have a question > about MTU on the interfaces. I have a connectX6 Mellanox interface > where I need IBoIP setup. > I've setup this interface via nmcli and set the MTU to 65520 with: > > nmcli connection modify ib0 mtu 65520 > nmcli connection up ib0 > > Running "nmcli connection show ib0" report: > > infiniband.mtu: 65520 > > But "ip addr show ib0" report a mtu of 2044: > > 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> *mtu 2044 *qdisc mq > state UP group default qlen 256 2044 is what you get with DATAGRAM mode and the native IB running with 2K packages. You can get 4092 IPoIB MTU with 4K IB. And you do want to run 4K native IB in general. To run 64K MTU you need to switch from DATAGRAM mode to CONNECTED mode for IPoIB. CONNECTED mode give the highest single stream bandwidth typically but may not give best overall performance (vs 4092 DATAGRAM mode). Cheers, Peter K