[CentOS] Infiniband special ops?

Thu Jan 21 23:33:56 UTC 2021
lejeczek <peljasz at yahoo.co.uk>

Hi guys.

Hoping some net experts my stumble upon this message, I have 
an IPoIB direct host to host connection and:

-> $ ethtool ib1
Settings for ib1:
     Supported ports: [  ]
     Supported link modes:   Not reported
     Supported pause frame use: No
     Supports auto-negotiation: No
     Supported FEC modes: Not reported
     Advertised link modes:  Not reported
     Advertised pause frame use: No
     Advertised auto-negotiation: No
     Advertised FEC modes: Not reported
     Speed: 40000Mb/s
     Duplex: Full
     Auto-negotiation: on
     Port: Other
     PHYAD: 255
     Transceiver: internal
     Link detected: yes

and that's both ends, both hosts, yet:

 > $ iperf3 -c 10.5.5.97
Connecting to host 10.5.5.97, port 5201
[  5] local 10.5.5.49 port 56874 connected to 10.5.5.97 port 
5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.36 GBytes  11.6 Gbits/sec    0   
2.50 MBytes
[  5]   1.00-2.00   sec  1.87 GBytes  16.0 Gbits/sec    0   
2.50 MBytes
[  5]   2.00-3.00   sec  1.84 GBytes  15.8 Gbits/sec    0   
2.50 MBytes
[  5]   3.00-4.00   sec  1.83 GBytes  15.7 Gbits/sec    0   
2.50 MBytes
[  5]   4.00-5.00   sec  1.61 GBytes  13.9 Gbits/sec    0   
2.50 MBytes
[  5]   5.00-6.00   sec  1.60 GBytes  13.8 Gbits/sec    0   
2.50 MBytes
[  5]   6.00-7.00   sec  1.56 GBytes  13.4 Gbits/sec    0   
2.50 MBytes
[  5]   7.00-8.00   sec  1.52 GBytes  13.1 Gbits/sec    0   
2.50 MBytes
[  5]   8.00-9.00   sec  1.52 GBytes  13.1 Gbits/sec    0   
2.50 MBytes
[  5]   9.00-10.00  sec  1.52 GBytes  13.1 Gbits/sec    0   
2.50 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.2 GBytes  13.9 Gbits/sec 
0             sender
[  5]   0.00-10.00  sec  16.2 GBytes  13.9 
Gbits/sec                  receiver

It's rather an oldish platform which hosts the link, PCIe is 
only 2.0 but with link of x8 that should be able to carry 
more than ~13Gbits/sec.
Infiniband is Mellanox's ConnectX-3.

Any thoughts on how to track the bottleneck or any thoughts 
I'll appreciate much.
thanks, L