-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi list,
I have a *very* strange problem, unfortunately it's kind of a show stopper regarding the deployment of the machine. :(
I have two Intel Gigabit Ethernet NICs on board (Supermicro-based Server), quoting lspci (full output see at the end of the email):
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03) 0f:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
While the first device (seen as eth0) does *not* allow to set a different MTU (tried with 1576 and 1546, both values I could use; I am forced to use values >1500 here due to protocol stuff), the second works without a problem.
dmesg shows:
eth0: Unsupported MTU setting ADDRCONF(NETDEV_UP): eth0: link is not ready eth1: changing MTU from 1500 to 1576 ADDRCONF(NETDEV_UP): eth1: link is not ready 802.1Q VLAN Support v1.8 Ben Greear greearb@candelatech.com All bugs added by David S. Miller davem@redhat.com ADDRCONF(NETDEV_UP): eth0.543: link is not ready ADDRCONF(NETDEV_UP): eth0.820: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ADDRCONF(NETDEV_CHANGE): eth0.543: link becomes ready ADDRCONF(NETDEV_CHANGE): eth0.820: link becomes ready e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
The MTU is set as usual in the ifcfg-ethN files as 'MTU=1576', e.g.
I tried with kernel 2.6.18-194.8.1.el5.028stab070.2 (OpenVZ) as well as the vanilla CentOS kernels 2.6.18-194.8.1.el5 and 2.6.18-194.11.1.el5 -- same problem.
Has anybody an idea what goes wrong here?
Thanks,
Timo
- ---
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03) Subsystem: Super Micro Computer Inc Unknown device 108c Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 90 Region 0: Memory at d0200000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 2000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Address: 00000000fee01000 Data: 405a Capabilities: [e0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag- Device: Latency L0s <512ns, L1 <64us Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ Device: MaxPayload 128 bytes, MaxReadReq 512 bytes Link: Supported Speed 2.5Gb/s, Width x1, ASPM unknown, Port 0 Link: Latency L0s <128ns, L1 <64us Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch- Link: Speed 2.5Gb/s, Width x1 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number xy-xx-xx-ff-ff-48-30-00
0f:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller Subsystem: Super Micro Computer Inc Unknown device 109a Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 106 Region 0: Memory at d0300000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Address: 00000000fee01000 Data: 406a Capabilities: [e0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag- Device: Latency L0s <512ns, L1 <64us Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ Device: MaxPayload 128 bytes, MaxReadReq 512 bytes Link: Supported Speed 2.5Gb/s, Width x1, ASPM unknown, Port 0 Link: Latency L0s <128ns, L1 <64us Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch- Link: Speed 2.5Gb/s, Width x1 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number xx-xx-xx-ff-ff-48-30-00
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
thus Timo Schoeler spake:
Hi list,
Little followup,
I have a *very* strange problem, unfortunately it's kind of a show stopper regarding the deployment of the machine. :(
I have two Intel Gigabit Ethernet NICs on board (Supermicro-based Server), quoting lspci (full output see at the end of the email):
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03) 0f:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
I tried with Intel's most recent driver, it shows the same behaviour:
[root@bla ~]# ip link set dev eth0 mtu 1576 SIOCSIFMTU: Invalid argument [root@bla ~]# ip link set dev eth1 mtu 1576 [root@bla ~]# ip link show dev eth1 6: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1576 qdisc pfifo_fast qlen 1000 link/ether 00:30:48:xx:xx:xy brd ff:ff:ff:ff:ff:ff
Timo
While the first device (seen as eth0) does *not* allow to set a different MTU (tried with 1576 and 1546, both values I could use; I am forced to use values >1500 here due to protocol stuff), the second works without a problem.
dmesg shows:
eth0: Unsupported MTU setting ADDRCONF(NETDEV_UP): eth0: link is not ready eth1: changing MTU from 1500 to 1576 ADDRCONF(NETDEV_UP): eth1: link is not ready 802.1Q VLAN Support v1.8 Ben Greear greearb@candelatech.com All bugs added by David S. Miller davem@redhat.com ADDRCONF(NETDEV_UP): eth0.543: link is not ready ADDRCONF(NETDEV_UP): eth0.820: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ADDRCONF(NETDEV_CHANGE): eth0.543: link becomes ready ADDRCONF(NETDEV_CHANGE): eth0.820: link becomes ready e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
The MTU is set as usual in the ifcfg-ethN files as 'MTU=1576', e.g.
I tried with kernel 2.6.18-194.8.1.el5.028stab070.2 (OpenVZ) as well as the vanilla CentOS kernels 2.6.18-194.8.1.el5 and 2.6.18-194.11.1.el5 -- same problem.
Has anybody an idea what goes wrong here?
Thanks,
Timo
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03) Subsystem: Super Micro Computer Inc Unknown device 108c Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 90 Region 0: Memory at d0200000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 2000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Address: 00000000fee01000 Data: 405a Capabilities: [e0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag- Device: Latency L0s <512ns, L1 <64us Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ Device: MaxPayload 128 bytes, MaxReadReq 512 bytes Link: Supported Speed 2.5Gb/s, Width x1, ASPM unknown, Port 0 Link: Latency L0s <128ns, L1 <64us Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch- Link: Speed 2.5Gb/s, Width x1 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number xy-xx-xx-ff-ff-48-30-00
0f:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller Subsystem: Super Micro Computer Inc Unknown device 109a Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 106 Region 0: Memory at d0300000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Address: 00000000fee01000 Data: 406a Capabilities: [e0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag- Device: Latency L0s <512ns, L1 <64us Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ Device: MaxPayload 128 bytes, MaxReadReq 512 bytes Link: Supported Speed 2.5Gb/s, Width x1, ASPM unknown, Port 0 Link: Latency L0s <128ns, L1 <64us Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch- Link: Speed 2.5Gb/s, Width x1 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number xx-xx-xx-ff-ff-48-30-00
On Fri, 2010-08-20 at 09:36 +0200, Timo Schoeler wrote:
I tried with Intel's most recent driver, it shows the same behaviour:
[root@bla ~]# ip link set dev eth0 mtu 1576 SIOCSIFMTU: Invalid argument [root@bla ~]# ip link set dev eth1 mtu 1576 [root@bla ~]# ip link show dev eth1 6: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1576 qdisc pfifo_fast qlen 1000 link/ether 00:30:48:xx:xx:xy brd ff:ff:ff:ff:ff:ff
Try with "ethtool"
While the first device (seen as eth0) does *not* allow to set a different MTU (tried with 1576 and 1546, both values I could use; I am forced to use values >1500 here due to protocol stuff), the second works without a problem.
Some e1000 cards do not even have support for jumbo frames. Just maybe if yours is that case you may not be able to raise the mtu value no higher than 1500.
John
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
thus JohnS spake:
On Fri, 2010-08-20 at 09:36 +0200, Timo Schoeler wrote:
I tried with Intel's most recent driver, it shows the same behaviour:
[root@bla ~]# ip link set dev eth0 mtu 1576 SIOCSIFMTU: Invalid argument [root@bla ~]# ip link set dev eth1 mtu 1576 [root@bla ~]# ip link show dev eth1 6: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1576 qdisc pfifo_fast qlen 1000 link/ether 00:30:48:xx:xx:xy brd ff:ff:ff:ff:ff:ff
Try with "ethtool"
Doesn't work either.
While the first device (seen as eth0) does *not* allow to set a different MTU (tried with 1576 and 1546, both values I could use; I am forced to use values >1500 here due to protocol stuff), the second works without a problem.
Some e1000 cards do not even have support for jumbo frames. Just maybe if yours is that case you may not be able to raise the mtu value no higher than 1500.
I saw this:
82573(V/L/E) TX Unit Hang Messages ---------------------------------- Several adapters with the 82573 chipset display "TX unit hang" messages during normal operation with the e1000e driver. The issue appears both with TSO enabled and disabled, and is caused by a power management function that is enabled in the EEPROM. Early releases of the chipsets to vendors had the EEPROM bit that enabled the feature. After the issue was discovered newer adapters were released with the feature disabled in the EEPROM.
If you encounter the problem in an adapter, and the chipset is an 82573-based one, you can verify that your adapter needs the fix by using ethtool:
# ethtool -e eth0 Offset Values ------ ------ 0x0000 00 12 34 56 fe dc 30 0d 46 f7 f4 00 ff ff ff ff 0x0010 ff ff ff ff 6b 02 8c 10 d9 15 8c 10 86 80 de 83 ^^ The value at offset 0x001e (de) has bit 0 unset. This enables the problematic power saving feature. In this case, the EEPROM needs to read "df" at offset 0x001e.
A one-time EEPROM fix is available as a shell script. This script will verify that the adapter is applicable to the fix and if the fix is needed or not. If the fix is required, it applies the change to the EEPROM and updates the checksum. The user must reboot the system after applying the fix if changes were made to the EEPROM.
(from http://downloadmirror.intel.com/15817/eng/README.txt)
However, the card is okay, I didn't need to fix it.
I also had a look at the driver's source code, what still puzzles me after reading this is that the other card on board *works*:
(...)
case e1000_82573: case e1000_82574: case e1000_82583: /* Disable ASPM L0s due to hardware errata */ e1000e_disable_aspm(adapter->pdev, PCIE_LINK_STATE_L0S);
if (pdev->device == E1000_DEV_ID_82573L) { adapter->flags |= FLAG_HAS_JUMBO_FRAMES; adapter->max_hw_frame_size = DEFAULT_JUMBO; } break; default: break; }
return 0; }
(...)
static struct e1000_info e1000_82573_info = { .mac = e1000_82573, .flags = FLAG_HAS_HW_VLAN_FILTER | FLAG_HAS_WOL | FLAG_APME_IN_CTRL3 | FLAG_RX_CSUM_ENABLED | FLAG_HAS_SMART_POWER_DOWN | FLAG_HAS_AMT | FLAG_HAS_SWSM_ON_LOAD, .pba = 20, .max_hw_frame_size = ETH_FRAME_LEN + ETH_FCS_LEN, .init_ops = e1000_init_function_pointers_82571, .get_variants = e1000_get_variants_82571, };
(...)
/* 82573 Errata 17 */ if (((adapter->hw.mac.type == e1000_82573) || (adapter->hw.mac.type == e1000_82574)) && (max_frame > ETH_FRAME_LEN + ETH_FCS_LEN)) { adapter->flags2 |= FLAG2_DISABLE_ASPM_L1; e1000e_disable_aspm(adapter->pdev, PCIE_LINK_STATE_L1); }
(...)
John
Timo
On Fri, 2010-08-20 at 10:17 +0200, Timo Schoeler wrote:
--- I read that also. Will they work without using the Intel Driver? As in using the kernel driver in the kernel. Seems to me it should because the e1000 is a fairly popular card.
Since you can't set the mtu you may benefit from setting the tcp values in sysctl.conf.
John
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
thus JohnS spake:
On Fri, 2010-08-20 at 10:17 +0200, Timo Schoeler wrote:
I read that also. Will they work without using the Intel Driver? As in using the kernel driver in the kernel. Seems to me it should because the e1000 is a fairly popular card.
Sure, first I tried with the CentOS driver.
Since you can't set the mtu you may benefit from setting the tcp values in sysctl.conf.
Nope, unfortunately I'm forced to have that MTU. However, we just swapped the switch port using the Switch's management so that the NIC that's able to do >1500 MTU get's the traffic in question.
So, I have a solution, but it doesn't solve the problem (regarding the NIC itself)...
John
Timo
On 08/20/2010 12:15 AM, Timo Schoeler wrote:
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03)
http://download.intel.com/design/network/specupdt/82573.pdf 17 ASPM/Jumbo Frames Disabled Due to Early Receive Threshold Overrun Buffer Status: Intel does not plan to resolve this erratum in the 82573 Gigabit Ethernet Controller. Jumbo frames is not supported in 82573E/V & is supported with the workaround above in 82573L.
You'll have to use different hardware. The chipset you've got has a bug, and jumbo frame support is disabled as a result.
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03)
http://download.intel.com/design/network/specupdt/82573.pdf 17 ASPM/Jumbo Frames Disabled Due to Early Receive Threshold Overrun Buffer Status: Intel does not plan to resolve this erratum in the 82573 Gigabit Ethernet Controller. Jumbo frames is not supported in 82573E/V& is supported with the workaround above in 82573L.
Interestingly, I somewhere got the same PDF (from the intel site) that required a password -- and didn't investigate further on your link. Now, out of curiosity, I did...
You'll have to use different hardware. The chipset you've got has a bug, and jumbo frame support is disabled as a result.
...and yes, you're right.
I'm amused about PeeCee hardware (sorry, only half of a pun intended)... There's those two NICs on board of a *server* grade machine, a 82573E and a 82573L. One of them is just *broken* (see above).
Cheers,
Timo
On 08/23/2010 09:22 AM, Timo Schoeler wrote:
I'm amused about PeeCee hardware (sorry, only half of a pun intended)... There's those two NICs on board of a *server* grade machine, a 82573E and a 82573L. One of them is just *broken* (see above).
Actually, both of them are broken. One of them has a workaround available for its brokenness.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
thus Gordon Messmer spake:
On 08/23/2010 09:22 AM, Timo Schoeler wrote:
I'm amused about PeeCee hardware (sorry, only half of a pun intended)... There's those two NICs on board of a *server* grade machine, a 82573E and a 82573L. One of them is just *broken* (see above).
Hi,
Actually, both of them are broken. One of them has a workaround available for its brokenness.
yep, I read the driver's source (and hey, they do comment their code! ;)...
For me, it's dead silicon. I know, even CPUs may have hundreds of errata, but shipping and selling crappy NIC chipsets is something that collides with my universe.
YMMV, tho.
Timo
-
Gordon Messmer -
JohnS -
Timo Schoeler