[CentOS] MTU report question

Fri Dec 18 09:25:50 UTC 2020
Peter Kjellström <cap at nsc.liu.se>

On Mon, 14 Dec 2020 14:46:30 +0100
Patrick Bégou <Patrick.Begou at legi.grenoble-inp.fr> wrote:

> Hi,
> I'm deploying a CentOS8 (not stream 😁) cluster and I have a question
> about MTU on the interfaces. I have a connectX6 Mellanox interface
> where I need IBoIP setup.
> I've setup this interface via nmcli and set the MTU to 65520 with:
>     nmcli connection modify ib0 mtu 65520
>     nmcli connection up ib0
> Running "nmcli connection show ib0" report:
>     infiniband.mtu:                         65520
> But "ip addr show ib0" report a mtu of 2044:
>     6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> *mtu 2044 *qdisc mq
> state UP group default qlen 256

2044 is what you get with DATAGRAM mode and the native IB running with
2K packages. You can get 4092 IPoIB MTU with 4K IB. And you do want to
run 4K native IB in general.

To run 64K MTU you need to switch from DATAGRAM mode to CONNECTED mode
for IPoIB. CONNECTED mode give the highest single stream bandwidth
typically but may not give best overall performance (vs 4092 DATAGRAM

 Peter K