On Thu, 21 Jan 2021 23:33:56 +0000 lejeczek via CentOS <centos at centos.org> wrote: > Hi guys. > > Hoping some net experts my stumble upon this message, I have > an IPoIB direct host to host connection and: ... > > $ iperf3 -c 10.5.5.97 > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 16.2 GBytes 13.9 Gbits/sec > 0 sender > [ 5] 0.00-10.00 sec 16.2 GBytes 13.9 > Gbits/sec receiver > > It's rather an oldish platform which hosts the link, PCIe is > only 2.0 but with link of x8 that should be able to carry > more than ~13Gbits/sec. > Infiniband is Mellanox's ConnectX-3. If you want to test the infiniband performance you ib_write_bw for example not iperf. IPoIB will always be quite a bit slower than native IB. That said, if you want to optimize IPoIB for performance make sure you're running connected mode not datagram mode. /Peter