Hi,
I have an interesting situation with one of our switches. It's a D-Link DGS-3100, 24 port 10/100/1000 Layer 2 Managed switch with some CentOS servers connected to it. On many of the servers I need to disable Flow Control on the switch's ports otherwise the CentOS server's doesn't connect to the switch. i.e. the Switch indicates that the LAN cable is unplugged and CentOS can simply not connect to any host on the LAN.
As soon as I disable Flow Control, CentOS "breaks".
Does anyone know why this would happen, or how / if I can enable Flow Control in CentOS as well?
On 07/07/2011 07:46 AM, Rudi Ahlers wrote:
Hi,
I have an interesting situation with one of our switches. It's a D-Link DGS-3100, 24 port 10/100/1000 Layer 2 Managed switch with some CentOS servers connected to it. On many of the servers I need to disable Flow Control on the switch's ports otherwise the CentOS server's doesn't connect to the switch. i.e. the Switch indicates that the LAN cable is unplugged and CentOS can simply not connect to any host on the LAN.
As soon as I disable Flow Control, CentOS "breaks".
Does anyone know why this would happen, or how / if I can enable Flow Control in CentOS as well?
I've got many CentOS machines connected to a few DGS-3100 (24 and 48 port versions) and I've not seen this problem before. I can't suggest what might be the problem, but I am going to guess that it's server side.
On Thu, Jul 7, 2011 at 2:31 PM, Digimer linux@alteeve.com wrote:
On 07/07/2011 07:46 AM, Rudi Ahlers wrote:
Hi,
I have an interesting situation with one of our switches. It's a D-Link DGS-3100, 24 port 10/100/1000 Layer 2 Managed switch with some CentOS servers connected to it. On many of the servers I need to disable Flow Control on the switch's ports otherwise the CentOS server's doesn't connect to the switch. i.e. the Switch indicates that the LAN cable is unplugged and CentOS can simply not connect to any host on the LAN.
As soon as I disable Flow Control, CentOS "breaks".
Does anyone know why this would happen, or how / if I can enable Flow Control in CentOS as well?
I've got many CentOS machines connected to a few DGS-3100 (24 and 48 port versions) and I've not seen this problem before. I can't suggest what might be the problem, but I am going to guess that it's server side.
Yes, I figured that much, but I don't know what to look for.
Some of the servers are Intel and other are SuperMicro, all using onboard NIC's. The Dell Windows servers connected to the same switch doesn't have this issue though
rudi
when migrating some rackmount HP servers running Centos4 from hard coded 100 meg fdx to auto gigE that we had to
1) remove this from our ifcfg-ethX files
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
2) proper CAT6 wiring
3) plug into the copper gigE switchport
4) reboot
using Cisco though...
did not find a decent plug n play solution
i.e., did not waste a lot of time looking for solutions other than the obvious cisco and centos config or network down and up interface commands
- rh
On Thu, Jul 7, 2011 at 3:22 PM, R - elists lists07@abbacomm.net wrote:
rudi
when migrating some rackmount HP servers running Centos4 from hard coded 100 meg fdx to auto gigE that we had to
- remove this from our ifcfg-ethX files
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
I don't see those options listed:
root@zaxen01:[~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0 # Intel Corporation 82566DC Gigabit Network Connection DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=00:1c:c0:75:19:ee TYPE=Ethernet IPADDR=196.34.x.x NETMASK=255.255.255.224 GATEWAY=196.34.x.x
- proper CAT6 wiring
already done :)
- plug into the copper gigE switchport
already done :)
- reboot
tried that already
using Cisco though...
I can't change a switch just for this. The other (Dell + Windows) servers on the exact same switch doesn't give me this problem.
did not find a decent plug n play solution
i.e., did not waste a lot of time looking for solutions other than the obvious cisco and centos config or network down and up interface commands
- rh
Rudi Ahlers wrote:
On Thu, Jul 7, 2011 at 3:22 PM, R - elistslists07@abbacomm.net wrote:
rudi
when migrating some rackmount HP servers running Centos4 from hard coded 100 meg fdx to auto gigE that we had to
- remove this from our ifcfg-ethX files
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
I don't see those options listed:
root@zaxen01:[~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
you're not looking in the right place, look in TFM: /usr/share/doc/initscripts-*/sysconfig.txt
On Thu, Jul 7, 2011 at 3:43 PM, Nicolas Thierry-Mieg Nicolas.Thierry-Mieg@imag.fr wrote:
Rudi Ahlers wrote:
On Thu, Jul 7, 2011 at 3:22 PM, R - elistslists07@abbacomm.net wrote:
rudi
when migrating some rackmount HP servers running Centos4 from hard coded 100 meg fdx to auto gigE that we had to
- remove this from our ifcfg-ethX files
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
I don't see those options listed:
root@zaxen01:[~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
you're not looking in the right place, look in TFM: /usr/share/doc/initscripts-*/sysconfig.txt _______________________________________________
Interesting. It *almost* looks like a plain text (i.e. instructions / manual) file to me and there's 2 lines with the option you specified:
ETHTOOL_OPTS=... Any device-specific options supported by ethtool. For example, if you wanted to force 100Mb full duplex: ETHTOOL_OPTS="speed 100 duplex full autoneg off"
The NIC is connected @ 1GB, as per the switch interface, but I can't seem to verify it on CentOS directly, and I would prefer not to change this file since it shows as 100MB (if that 2nd line with ETHTOOL_OPTS is an actual configuration option, and not just comment) but there's no mention of Flow Control in that file. I don't know what impact this could have, if any at all.
Rudi Ahlers wrote:
On Thu, Jul 7, 2011 at 3:22 PM, R - elists lists07@abbacomm.net wrote:
rudi
when migrating some rackmount HP servers running Centos4 from hard coded 100 meg fdx to auto gigE that we had to
- remove this from our ifcfg-ethX files
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
<snip> A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
mark
A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
mark
in this case though, it is gigE so dont use 100
it was used cause cisco typically should be hard set for speed and duplex so you never have to guess or worry
realistically speed auto duplex auto
gigE is supposed to be smarter
rudi,
you can play with ethtool without having stuff in the ifcfg file
it isnt absolutely necessary to have that stuff in an ifcfg- interface file
those things in the line above are for "boot time" or when up'n the interface and wanting it always the same
- rh
R - elists wrote:
A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
in this case though, it is gigE so dont use 100
<snip> Right, but the point I was making is that when I started playing with ethtool last year, I found that the paramenter autoneg [on|off] *had* to be the first parameter, otherwise it gagged trying to set other options. Clearly, it's a dumb program, that just reads the list, and performs them one at a time, in order of appearance (read, perform, pop parm, repeat)
mark
On 7/7/2011 9:24 AM, R - elists wrote:
A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
markin this case though, it is gigE so dont use 100
it was used cause cisco typically should be hard set for speed and duplex so you never have to guess or worry
Errr, perhaps you mean because old cisco stuff didn't negotiate reliably? You always have to worry when you have hard set options because it won't work when someone changes the equipment at the other end.
Les Mikesell wrote:
On 7/7/2011 9:24 AM, R - elists wrote:
A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
in this case though, it is gigE so dont use 100
it was used cause cisco typically should be hard set for speed and duplex so you never have to guess or worry
Errr, perhaps you mean because old cisco stuff didn't negotiate reliably? You always have to worry when you have hard set options because it won't work when someone changes the equipment at the other end.
I thought we were talking about NIC's and the use of ethtool, not Cisco switches. And don't get me started on the networking turkey, er, team here.
mark
On 7/7/2011 10:15 AM, m.roth@5-cent.us wrote:
A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
in this case though, it is gigE so dont use 100
it was used cause cisco typically should be hard set for speed and duplex so you never have to guess or worry
Errr, perhaps you mean because old cisco stuff didn't negotiate reliably? You always have to worry when you have hard set options because it won't work when someone changes the equipment at the other end.
I thought we were talking about NIC's and the use of ethtool, not Cisco switches. And don't get me started on the networking turkey, er, team here.
Old Cisco switches - and Cisco's advice about how to work around their problems - are just the main reason that anyone would ever have turned off auto-negotiate. And it is a big problem if you only turn if off at one end which is what you end up with as you start to change equipment, because the other end will always get it wrong. These days, if a device doesn't negotiate properly you should probably just replace it.
On 07/07/2011 17:30, Les Mikesell wrote:
Old Cisco switches - and Cisco's advice about how to work around their problems - are just the main reason that anyone would ever have turned off auto-negotiate. And it is a big problem if you only turn if off at one end which is what you end up with as you start to change equipment, because the other end will always get it wrong. These days, if a device doesn't negotiate properly you should probably just replace it.
The problem is not the auto-negotiation iteself, but the fact that if one side hard codes its speed to 100-Full Duplex then the other side cannot auto-negotiate to 100-Full Duplex. It also needs to be hard-coded to 100-Full duplex - The auto-negotiation is not a "I'll do what you're set to" type protocol, but a "let's see what's best for us" protocol.
There was actually never any problem with auto-negotiation itself - it did exactly what it said on the box, just that it didn't work if either end turned it off and hard coded it's speed.
Having seen my fair share of performance problems, if you don't have console access to both interfaces then agree on the speed and duplex and hard code it - saves a lot of faffing about and almost always works a treat.
On 7/8/2011 5:50 PM, Giles Coochey wrote:
On 07/07/2011 17:30, Les Mikesell wrote:
Old Cisco switches - and Cisco's advice about how to work around their problems - are just the main reason that anyone would ever have turned off auto-negotiate. And it is a big problem if you only turn if off at one end which is what you end up with as you start to change equipment, because the other end will always get it wrong. These days, if a device doesn't negotiate properly you should probably just replace it.
The problem is not the auto-negotiation iteself, but the fact that if one side hard codes its speed to 100-Full Duplex then the other side cannot auto-negotiate to 100-Full Duplex. It also needs to be hard-coded to 100-Full duplex - The auto-negotiation is not a "I'll do what you're set to" type protocol, but a "let's see what's best for us" protocol.
There was actually never any problem with auto-negotiation itself - it did exactly what it said on the box, just that it didn't work if either end turned it off and hard coded it's speed.
Yes, if it hurts, don't do it.
Having seen my fair share of performance problems, if you don't have console access to both interfaces then agree on the speed and duplex and hard code it - saves a lot of faffing about and almost always works a treat.
Turning off negotiation pretty much guarantees problems if anything changes at the other end or you use an unmanaged switch. And the gigabit spec requires auto-negotiation.
On 09/07/2011 01:06, Les Mikesell wrote:
Turning off negotiation pretty much guarantees problems if anything changes at the other end or you use an unmanaged switch. And the gigabit spec requires auto-negotiation.
Let me make it clear - auto-negotiation only works if auto-negotiation is configured on both sides. It does not work if one side hard codes the speed and duplex. Both sides have to be set for it to negotiate. Agreeing on speed and duplex ensures that it will work.
If something is going to change on the remote end without you knowing, or your provider is using an unmanaged switch then it's time to change provider :-) - they obviously are cheapskates and don't have any change management control on their systems.
Gigabit is different.
Giles Coochey wrote:
On 09/07/2011 01:06, Les Mikesell wrote:
Turning off negotiation pretty much guarantees problems if anything changes at the other end or you use an unmanaged switch. And the gigabit spec requires auto-negotiation.
Let me make it clear - auto-negotiation only works if auto-negotiation is configured on both sides. It does not work if one side hard codes the speed and duplex. Both sides have to be set for it to negotiate. Agreeing on speed and duplex ensures that it will work.
If something is going to change on the remote end without you knowing, or your provider is using an unmanaged switch then it's time to change provider :-) - they obviously are cheapskates and don't have any change management control on their systems.
Gigabit is different.
My reading of the spec is that when a port is configured for 1GbE over 1000BASE-T (copper), "disabling" auto-negotiation disables the advertising of the auto-negotiation for 10BASE-T and 100BASE-T, but auto-negotiation is still advertised and operational for 1GbE. Auto-negotiation cannot be disabled for 1000BASE-X (optical fiber).
Deviation from the spec would mean such kit is Ethernet-like. An ability to "set" auto-negotiation one way in the user interface while leaving the hardware in a different - standards conforming - state is possible.
On 10/07/2011 10:22, Charles Polisher wrote:
Gigabit is different. My reading of the spec is that when a port is configured for 1GbE over 1000BASE-T (copper), "disabling" auto-negotiation disables the advertising of the auto-negotiation for 10BASE-T and 100BASE-T, but auto-negotiation is still advertised and operational for 1GbE. Auto-negotiation cannot be disabled for 1000BASE-X (optical fiber).
Deviation from the spec would mean such kit is Ethernet-like. An ability to "set" auto-negotiation one way in the user interface while leaving the hardware in a different - standards conforming - state is possible.
Fiber is not a CSMA/CD medium, it's a Point to Point medium - Duplex is meaningless.
I've been referring to the Spec of 10/100 ports. For Gigabit ports 1000Base-T, auto-negotiation is mandatory.
Quoting from Wikipedia [http://en.wikipedia.org/wiki/Autonegotiation]
Interoperability problems
The first version of the autonegotiation specification, IEEE 802.3u, was open to different interpretations. Although most manufacturers implemented this standard in one way, some others, including network giant Cisco, implemented it in a different way. Autonegotiation between devices that implemented it differently failed. This led many network administrators to not depend on autonegotiation and instead manually set the speed and duplex mode of each network interface card. Even Cisco recommended its customers not to use autonegotiation. However, the use of manually set configuration often led to duplex mismatches, in particular when two connected devices are:
* One manually set to half duplex and one manually set to full duplex * One set to autonegotiation and one manually set to full duplex * Both sides manually set to full duplex where one side still expects an autonegotiating link partner and the other side has autonegotiation completely disabled (the side that expects an autonegotiating link partner will fall back to half duplex because it does not detect a partner capable of full duplex)^[/citation needed/]
Duplex mismatch problems are difficult to diagnose because the network is apparently working, and simple programs used for network tests such as ping report a valid connection; however, the network is much slower than expected.
The debatable portions of the autonegotiation specifications were eliminated by the 1998 release of 802.3. This was later followed by the release of IEEE 802.3ab in 1999. The new standard specified that gigabit Ethernet over copper wiring requires autonegotiation. Currently, all network equipment manufacturers—including Cisco^[3] —recommend to use autonegotiation on all access ports. Cisco also recommends that you check back with them yearly for any potential changes in their recommendation as this has caused much confusion over the years. ^[4]
In some large installations that have had to deal with negotiation issues, network staff may believe that "autonegotiation doesn't work", and consider turning it off a best-practice. This should be avoided - once autonegotiation is turned off, it will not work by definition, creating a self-enforcing problem.
On 10/07/2011 12:57, John R Pierce wrote:
On 07/10/11 1:46 AM, Giles Coochey wrote:
Fiber is not a CSMA/CD medium, it's a Point to Point medium - Duplex is meaningless.
so is twisted pair.
ha... ha... of course, interesting.
I guess what I was trying to say is that for fiber connections duplex has no meaning, was there ever a fiber 'hub' where multiple point to point connections 'shared' a medium? (in a virtual sense)
On Sunday, July 10, 2011 07:05:16 AM Giles Coochey wrote:
I guess what I was trying to say is that for fiber connections duplex has no meaning, was there ever a fiber 'hub' where multiple point to point connections 'shared' a medium? (in a virtual sense)
You can make/get passive fiber hubs that act like old thinnet. They aren't common, and I don't recall if any GigE ones exist.
You can actually make these if you have a fusion splicer with enough manual controllability.
So, yes, you can have a true CSMA/CD fiber hub, where the hub is a totally passive fiber device.
On 07/11/11 10:17 AM, Lamar Owen wrote:
On Sunday, July 10, 2011 07:05:16 AM Giles Coochey wrote:
I guess what I was trying to say is that for fiber connections duplex has no meaning, was there ever a fiber 'hub' where multiple point to point connections 'shared' a medium? (in a virtual sense)
You can make/get passive fiber hubs that act like old thinnet. They aren't common, and I don't recall if any GigE ones exist.
You can actually make these if you have a fusion splicer with enough manual controllability.
So, yes, you can have a true CSMA/CD fiber hub, where the hub is a totally passive fiber device.
it was my understanding that 'hubs' were only supported on 10baseT and 100baseT ethernets, and of course, have to be half duplex, and all devices have to be the same speed. the so-called dual speed 10/100 hubs were really a 10baseT hub and a 100baseT hub with a switch between them (and were really really awful). GigE mandates switching and full duplex. Anyone using fiber for 10 or 100baseF is stuck in the last century.
On Monday, July 11, 2011 01:47:09 PM John R Pierce wrote:
it was my understanding that 'hubs' were only supported on 10baseT and 100baseT ethernets,
....
GigE mandates switching and full duplex. Anyone using fiber for 10 or 100baseF is stuck in the last century.
Or stuck with >550m multimode fiber links (we have a few of those; longest is over 1km, and only multimode ran to that location).
The 10Mb/s standard is (heading from IEEE 802.3-2008): 16. Fiber optic passive star and medium attachment unit, type 10BASE-FP
802.3-2008 clause 41 provides for a half-duplex 1000Mb/s repeater, but it is noted that no maintenance changes are being accepted since May 2007 for that clause.
And then, of course, there are EPON and GPON, but that's passive full duplex using WDM technologies with the passive splitters and combiners (typical fiber to the premises gear). GBIC's and SFP's are available for media converters and switches; if a GigE NIC has an SFP slot it might be possible to support those technologies at the workstation or server.
No, switching isn't mandated; while the 802.3 standard is pretty dense, it is now freely available from IEEE for anyone to download and look at.
On 7/9/11 12:18 PM, Giles Coochey wrote:
On 09/07/2011 01:06, Les Mikesell wrote:
Turning off negotiation pretty much guarantees problems if anything changes at the other end or you use an unmanaged switch. And the gigabit spec requires auto-negotiation.
Let me make it clear - auto-negotiation only works if auto-negotiation is configured on both sides.
Errr, auto-negotiation is normally the default. So it to be more clear, it works unless you break it by changing it only on one side.
It does not work if one side hard codes the speed and duplex. Both sides have to be set for it to negotiate. Agreeing on speed and duplex ensures that it will work.
That means both sides have to know about each other, whereas one side is networking equipment and the other is often host equipment, managed by different sets of people. And there is no need for them to agree/disagree or waste time thinking about it at all. The defaults should work.
If something is going to change on the remote end without you knowing, or your provider is using an unmanaged switch then it's time to change provider :-) - they obviously are cheapskates and don't have any change management control on their systems.
I'm not talking about 'providers', I'm talking about the people who set up network equipment vs. people who manage hosts. If the people managing the network equipment say negotiation needs to be off they are wrong, even if they claim to be the authority in the debate.
Gigabit is different.
No, the default of auto-negotiating works there too.
On Sun, 2011-07-10 at 22:08 -0500, Les Mikesell wrote:
On 7/9/11 12:18 PM, Giles Coochey wrote:
Gigabit is different.
No, the default of auto-negotiating works there too.
In 1000BASE-T, autonegotiation is required, according to http://en.wikipedia.org/wiki/Gigabit_Ethernet#1000BASE-T
Which, in turn, refers to (click through without username required) http://standards.ieee.org/getieee802/download/802.3-2008_section2.pdf that states (in section 28D.5, part a) that Auto-negotiation in 1000BASE-T is required...
-I
On Friday, July 08, 2011 07:06:06 PM Les Mikesell wrote:
And the gigabit spec requires auto-negotiation.
According to clause 37.1.4.4 "User Configuration with Auto-Negotiation" of IEEE 802.3-2008, auto-negotiation with 1000Base-X is optional, but encouraged.
In contrast, for 1000Base-T, we find the following information in clause 40.5.1: "40.5.1 Support for Auto-Negotiation All 1000BASE-T PHYs shall provide support for Auto-Negotiation (Clause 28) and shall be capable of operating as MASTER or SLAVE."
'Shall' being legal language for 'absolutely without a doubt must' in this case. Overwise the PHY wouldn't be able to determine which PHY is the clock MASTER.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Les Mikesell Sent: Thursday, July 07, 2011 7:37 AM To: centos@centos.org Subject: Re: [CentOS] how to enable Flow Control on CentOS?
On 7/7/2011 9:24 AM, R - elists wrote:
A minor detail - I think you *must* have autoneg set first - it applies them one after the other, in my experience.
ETHTOOL_OPTS="autoneg off speed 100 duplex full"
markin this case though, it is gigE so dont use 100
it was used cause cisco typically should be hard set for speed and duplex so you never have to guess or worry
Errr, perhaps you mean because old cisco stuff didn't negotiate reliably? You always have to worry when you have hard set options because it won't work when someone changes the equipment at the other end.
-- Les Mikesell
exactly,
this is in the datacenters though, so we didnt care...
we always hard set old cisco gear and whatever is connected to it for "our" internal needs...
for clients, some like auto, mainly cause they are using a BSD flavor or whatever equipment or dont care
- rh
On 7/7/11, Rudi Ahlers Rudi@softdux.com wrote:
Yes, I figured that much, but I don't know what to look for.
Some of the servers are Intel and other are SuperMicro, all using onboard NIC's. The Dell Windows servers connected to the same switch doesn't have this issue though
I had a similar experience but on Windows machines. Connecting to a DLink DES or DGS 1024, on two of the machines, having autoneg = similar sympton as if the network cable is broken. Setting the NICs to 100Mbps only fixed the problem. The common thing here seems to be DLink switch and some other factor that effectively causes autoneg to fail without fallback.
Emmanuel Noobadmin wrote:
On 7/7/11, Rudi Ahlers Rudi@softdux.com wrote:
Yes, I figured that much, but I don't know what to look for.
Some of the servers are Intel and other are SuperMicro, all using onboard NIC's. The Dell Windows servers connected to the same switch doesn't have this issue though
I had a similar experience but on Windows machines. Connecting to a DLink DES or DGS 1024, on two of the machines, having autoneg = similar sympton as if the network cable is broken. Setting the NICs to 100Mbps only fixed the problem. The common thing here seems to be DLink switch and some other factor that effectively causes autoneg to fail without fallback.
Another quirky thing we ran into was having to change some cables. The newer cables allowed gigabit, and the older ones did not.
mark
-
Charles Polisher -
Digimer -
Emmanuel Noobadmin -
Giles Coochey -
Ian Forde -
John R Pierce -
Lamar Owen -
Les Mikesell -
m.roth＠5-cent.us -
Nicolas Thierry-Mieg -
R - elists -
Rudi Ahlers