I've got a pair of identical CentOS 6.7 servers, with SuperMicro X8DTE-F motherboards, these have 2 each Intel 82574L ethernet ports. The eth0 ports are plugged in with 10' runs of brand new cat 5e cable to a Cisco Nexxus 9000 switch (provided by the data center).
These servers keep coming up at 100baseT rather than gigE. I've swapped ports and cables with a different server, that server gets gigE but these supermicro's are stuck at 100baseT regardless of the port or cableused.
'ethtool eth0' says its gigE, but 'mii-tool -v eth0' says its 100baseT and doesn't even list gigE as an option, and the cisco switches see it as 100baseT too.
anyone else run into anything like this before?
On Mon, Oct 10, 2016 at 9:33 PM, John R Pierce pierce@hogranch.com wrote:
I've got a pair of identical CentOS 6.7 servers, with SuperMicro X8DTE-F motherboards, these have 2 each Intel 82574L ethernet ports. The eth0 ports are plugged in with 10' runs of brand new cat 5e cable to a Cisco Nexxus 9000 switch (provided by the data center).
These servers keep coming up at 100baseT rather than gigE. I've swapped ports and cables with a different server, that server gets gigE but these supermicro's are stuck at 100baseT regardless of the port or cableused.
'ethtool eth0' says its gigE, but 'mii-tool -v eth0' says its 100baseT and doesn't even list gigE as an option, and the cisco switches see it as 100baseT too.
mii-tool doesn't support gigE. Maybe, they're not negotiating: "ethtool -s eth0 autoneg on", you can renegotiate with "ethtool -r eth0"
On 10/10/2016 5:33 PM, John R Pierce wrote:
I've got a pair of identical CentOS 6.7 servers, with SuperMicro X8DTE-F motherboards, these have 2 each Intel 82574L ethernet ports. The eth0 ports are plugged in with 10' runs of brand new cat 5e cable to a Cisco Nexxus 9000 switch (provided by the data center).
These servers keep coming up at 100baseT rather than gigE. I've swapped ports and cables with a different server, that server gets gigE but these supermicro's are stuck at 100baseT regardless of the port or cableused.
'ethtool eth0' says its gigE, but 'mii-tool -v eth0' says its 100baseT and doesn't even list gigE as an option, and the cisco switches see it as 100baseT too.
anyone else run into anything like this before?
oh.
# mii-tool -v eth0 eth0: negotiated, link ok product info: vendor 00:50:43, model 11 rev 1 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
# ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: No Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes
On 10/10/2016 11:21 PM, Gordon Messmer wrote:
On 10/10/2016 09:31 PM, John R Pierce wrote:
oh.
Yeah, the entire "net-tools" package is deprecated. I tend to forget which of the two (ethtool or mii-tool) is in that set.
# Avoid using any of these: $ rpm -ql net-tools
/bin/dnsdomainname /bin/domainname /bin/hostname /bin/netstat /bin/nisdomainname /bin/ypdomainname /sbin/ether-wake /sbin/ifconfig /sbin/ipmaddr /sbin/iptunnel /sbin/mii-diag /sbin/mii-tool /sbin/nameif /sbin/plipconfig /sbin/route /sbin/slattach
ok, so the mii-* stuff is deprecated (as is route, ifconfig, netstat, and arp? sigh).
apparently the network administrator went ahead and forced the switch port to use gigE, so its no longer in the 'broken' state of autonegotiating 100baseT.
On 11/10/2016 09:14, John R Pierce wrote:
On 10/10/2016 11:21 PM, Gordon Messmer wrote:
On 10/10/2016 09:31 PM, John R Pierce wrote:
oh.
Yeah, the entire "net-tools" package is deprecated. I tend to forget which of the two (ethtool or mii-tool) is in that set.
# Avoid using any of these: $ rpm -ql net-tools
/bin/dnsdomainname /bin/domainname /bin/hostname /bin/netstat /bin/nisdomainname /bin/ypdomainname /sbin/ether-wake /sbin/ifconfig /sbin/ipmaddr /sbin/iptunnel /sbin/mii-diag /sbin/mii-tool /sbin/nameif /sbin/plipconfig /sbin/route /sbin/slattach
ok, so the mii-* stuff is deprecated (as is route, ifconfig, netstat, and arp? sigh).
apparently the network administrator went ahead and forced the switch port to use gigE, so its no longer in the 'broken' state of autonegotiating 100baseT.
Just for comparison this is a server that is set to auto negotiate
[root@sch-mwg-01 access.log]# mii-tool eth0 eth0: negotiated 100baseTx-FD, link ok [root@sch-mwg-01 access.log]# ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: off Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes
Note that mii-tool is reporting incorrectly. That server is currently processing ~500Mbit/s. Of the tools listed above, the only one I still extensively use is netstat as ss, its replacement, is IMO not very nice.
Tris
************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify postmaster@bgfl.org
The views expressed within this email are those of the individual, and not necessarily those of the organisation *************************************************************
Hi,
On Tue, Oct 11, 2016 at 6:03 AM, John R Pierce pierce@hogranch.com wrote:
I've got a pair of identical CentOS 6.7 servers, with SuperMicro X8DTE-F motherboards, these have 2 each Intel 82574L ethernet ports. The eth0 ports are plugged in with 10' runs of brand new cat 5e cable to a Cisco Nexxus 9000 switch (provided by the data center).
These servers keep coming up at 100baseT rather than gigE. I've swapped ports and cables with a different server, that server gets gigE but these supermicro's are stuck at 100baseT regardless of the port or cableused.
'ethtool eth0' says its gigE, but 'mii-tool -v eth0' says its 100baseT and doesn't even list gigE as an option, and the cisco switches see it as 100baseT too.
anyone else run into anything like this before?
This is quite strange but I need you check that if this is a problem with
the cisco switch or server NICs.
Please test that if both the server are communicating with each other at 1Gbps or not via "iperf" tool.
If above gives result of 1Gbps then it will eliminate the NICs problem then you know that it is a problem with cisco switch only.
--Regards Ashishkumar S. Yadav
-- john r pierce, recycling bits in santa cruz
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 10/11/2016 9:03 PM, Ashish Yadav wrote:
Please test that if both the server are communicating with each other at 1Gbps or not via "iperf" tool.
If above gives result of 1Gbps then it will eliminate the NICs problem then you know that it is a problem with cisco switch only.
after they forced the cisco ports to gigE, I was seeing 200-400Mbps in iPerf, which was odd. servers were both very lightly loaded.
BUT... the switch ports kept going offline on us. Note I have no admin access to the switch, its managed by IT so I have to go through channels to get anything. I asked what error codes were causing the ports to go offline but haven't heard back. as of right now, both servers are offline, (I can reach their IPMI management controller, and remotely log onto the console just fine, but the ports show no link). When I was in the DC yesterday, I switched ports, same problem, I also switched the network cable with a different (HP) server, it had no problems on the same cable+port thats giving these supermicro servers problems.
I'd chalk it up to a bad NIC, but two identical servers with two nic's each all have this problem, so its got to be something else, some weirdness with the 82574L as implemented on these SuperMicro X8DTE-F servers running CentOS 6.7 ?!? In our old DC, these servers ran rock solid for several years without any network issues at all, in that rack I had a Netgear JGS524
On 12.10.2016 06:26, John R Pierce wrote:
On 10/11/2016 9:03 PM, Ashish Yadav wrote:
Please test that if both the server are communicating with each other at 1Gbps or not via "iperf" tool.
If above gives result of 1Gbps then it will eliminate the NICs problem then you know that it is a problem with cisco switch only.
after they forced the cisco ports to gigE, I was seeing 200-400Mbps in iPerf, which was odd. servers were both very lightly loaded.
BUT... the switch ports kept going offline on us. Note I have no admin access to the switch, its managed by IT so I have to go through channels to get anything. I asked what error codes were causing the ports to go offline but haven't heard back. as of right now, both servers are offline, (I can reach their IPMI management controller, and remotely log onto the console just fine, but the ports show no link). When I was in the DC yesterday, I switched ports, same problem, I also switched the network cable with a different (HP) server, it had no problems on the same cable+port thats giving these supermicro servers problems.
I'd chalk it up to a bad NIC, but two identical servers with two nic's each all have this problem, so its got to be something else, some weirdness with the 82574L as implemented on these SuperMicro X8DTE-F servers running CentOS 6.7 ?!? In our old DC, these servers ran rock solid for several years without any network issues at all, in that rack I had a Netgear JGS524
A while back there was an issue with this nic chipset and CentOS but I'm not sure if this still applies to CentOS 6.7: https://blog.andreas-haerter.com/2013/02/11/intel-82574l-network-nic-aspm-bu...
If this is your problem then adding "pcie_aspm=off" should fix it.
Regards, Dennis
On Oct 12, 2016, at 12:26 AM, John R Pierce pierce@hogranch.com wrote:
the switch ports kept going offline on us.
Not finding anything exactly like this... Closest I could find is CSCuu81949
Open a Cisco TAC case and upload a Nexus 9000 tech support (`tac-pac`) to investigate further.
Is "port security" enabled on these ports? Does this port double as a LOM/IPMI port? What driver is being used `ethtool -i eth#`? Any OS bonding? Any switch port-channel/vPC?
That NIC chipset is a few years old so it's not like that NIC/OS/switch is a combo that hasn't been tried/tested.
On 10/13/2016 5:57 PM, Steven Tardy wrote:
On Oct 12, 2016, at 12:26 AM, John R Pierce pierce@hogranch.com wrote:
the switch ports kept going offline on us.
Not finding anything exactly like this... Closest I could find is CSCuu81949
Open a Cisco TAC case and upload a Nexus 9000 tech support (`tac-pac`) to investigate further.
Is "port security" enabled on these ports?
I don't know, asking LAN operations
Does this port double as a LOM/IPMI port?
no, boards have dedicated IPMI port, which doesn't go offline
What driver is being used `ethtool -i eth#`?
e1000m
Any OS bonding?
no
Any switch port-channel/vPC?
I don't believe so.
That NIC chipset is a few years old so it's not like that NIC/OS/switch is a combo that hasn't been tried/tested.
yeah, about as bog stock normal a config as I've seen.
-
Alexander Dalloz -
Ashish Yadav -
Dennis Jacobfeuerborn -
Gordon Messmer -
John R Pierce -
Marcelo Roccasalva -
Steven Tardy -
Tris Hoar