Hi
We are trying to configure RDMA for an infiniband connection between our data server (running CentOS 6.8) and our compute nodes (running CentOS 6.6). We have been trying to follow the instructions in
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
however we are getting conflicting information on whether the RDMA is correctly configured. Some of what we have done and some of the data are below. Can you suggest what other tests and what other data we should get to debug this problem?
/etc/rdma/mlx4.conf to set the port types properly for RoCE/IBoE usage.
Edit /etc/modprobe.d/mlx4.conf to instruct the driver on which packet priority is configured for the “no-drop” service on the Ethernet switches the cards are plugged into.
vim /etc/rdma/mlx4.conf
You can find the right pci device to use for any given card by loading
# the mlx4_core module, then going to /sys/bus/pci/drivers/mlx4_core and
# seeing what possible PCI devices are listed there.The possible values
# for ports are: ib, eth, and auto.However, not all cards support all
# types, so if you get messages from the kernel that your selected port
# type isn't supported, there's nothing this script can do about it.
[root@mseas-data2 mlx4_core]# ls /sys/bus/pci/drivers/mlx4_core
0000:81:00.0bindmodulenew_idremove_idueventunbind
mstconfig -d 81:00.0 query
Device #1:
----------
Device type:ConnectX3Pro
PCI device:81:00.0
Configurations:Current
SRIOV_EN1
NUM_OF_VFS8
LINK_TYPE_P13
LINK_TYPE_P23
LOG_BAR_SIZE3
BOOT_PKEY_P10
BOOT_PKEY_P20
BOOT_OPTION_ROM_EN_P11
BOOT_VLAN_EN_P10
BOOT_RETRY_CNT_P10
LEGACY_BOOT_PROTOCOL_P11
BOOT_VLAN_P11
BOOT_OPTION_ROM_EN_P21
BOOT_VLAN_EN_P20
BOOT_RETRY_CNT_P20
LEGACY_BOOT_PROTOCOL_P21
BOOT_VLAN_P21
Needed packages:
Most if not all packages appear to be installed on server
[root@mseas-data2 ~]# service rdma status Low level hardware support loaded: mlx4_ib
Upper layer protocol modules: ib_ipoib
User space access modules: rdma_ucm ib_ucm ib_uverbs ib_umad
Connection management modules: rdma_cm ib_cm iw_cm
Configured IPoIB interfaces: none Currently active IPoIB interfaces: ib0