Hi there,
I’m working on deploying our new cluster. Masters have 5×1gbps (i210 and i350, thus using igb.ko), configured with mtu 9000, 802.3ad. Works fine *but* I can’t get DCB working (pause frame, aka flow control, which is supported by and enabled on our switches).
[root@master2 ~]# dcbtool gc eno1 dcb Command: Get Config Feature: DCB State Port: eno1 Status: Device not capable
(I get the same with ELRepo 5.2.15 kmod). Intel datasheet says flow control is available. Can’t find much about it on the web or in kernel git repo. Could someone give me a hand ?
Regards, Laurent.
On Thu, Feb 26, 2015 at 9:18 AM, Laurent Wandrebeck < l.wandrebeck@quelquesmots.fr> wrote:
Hi there,
I’m working on deploying our new cluster. Masters have 5×1gbps (i210 and i350, thus using igb.ko), configured with mtu 9000, 802.3ad. Works fine *but* I can’t get DCB working (pause frame, aka flow control, which is supported by and enabled on our switches).
[root@master2 ~]# dcbtool gc eno1 dcb Command: Get Config Feature: DCB State Port: eno1 Status: Device not capable
(I get the same with ELRepo 5.2.15 kmod). Intel datasheet says flow control is available. Can’t find much about it on the web or in kernel git repo. Could someone give me a hand ?
Regards, Laurent.
DCB requires Priority Flow Control(PFC) aka 802.1Qbb. "Flow Control" is 802.3x.
The two are often confused and not compatible.
http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-control...
Mentions "flow control" several times, but never PFC/priority-flow-control/802.1Qbb.
PFC capable switches purposefully disable 802.3x flow control. Also PFC has to negotiate between two devices/switches matching QoS/CoS/no-drop policies.
Some good reading for beginner PFC knowledge:
http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualiza...
What exactly are you trying to pause? Typically FCoE/iSCSI is set to "no-drop" and Ethernet traffic is paused/dropped in favor of storage traffic. If there is only one type/class/CoS of traffic PFC won't gain much over regular flow control/802.3x.
Hope that helps.
Steven Tardy sjt5atra@gmail.com a écrit :
DCB requires Priority Flow Control(PFC) aka 802.1Qbb. "Flow Control" is 802.3x.
The two are often confused and not compatible.
http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-control...
Mentions "flow control" several times, but never PFC/priority-flow-control/802.1Qbb.
PFC capable switches purposefully disable 802.3x flow control. Also PFC has to negotiate between two devices/switches matching QoS/CoS/no-drop policies.
Some good reading for beginner PFC knowledge:
http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualiza...
What exactly are you trying to pause? Typically FCoE/iSCSI is set to "no-drop" and Ethernet traffic is paused/dropped in favor of storage traffic. If there is only one type/class/CoS of traffic PFC won't gain much over regular flow control/802.3x.
Hope that helps.
Hello Steven,
You’ve been really helpful !
Our switches indeed do support 802.3x and not 802.1Qbb.
Ethtool telling: Supported pause frame use: Symmetric Advertised pause frame use: Symmetric
I guess (I’m more of a sysad guy than netad) we’re on the right track and have no need of DCB* and lldpad.
Actually, our masters will be metadata server for the distributed FS (RozoFS not to name it), and will export a system image via NFS to nodes (2×1gbps, 802.3ad) which are « diskless » (no disk for OS but disks for distributed FS storage only). FC (802.3x) usage is mandatory for RozoFS. There will be some other traffic due to HTCondor (nodes will be execute nodes too), syslog being centralized on masters… I know, that not the perfect config, but we had to do that way due to budget constraints. Now I need to find how to get a single image for all the nodes :) (PXE, dhcpd, dracut and yum --installroot should do the trick I hope).
Thanks again for the head’s up !
Regards, Laurent.
-
Laurent Wandrebeck -
Steven Tardy