[CentOS] CentOS5 with Dell Broadcom iSCSI Offload, does it work ?

Fri Sep 9 19:30:31 UTC 2011

----- "Steve Walsh" <steve at nerdvana.net.au> escreveu:

> De: "Steve Walsh" <steve at nerdvana.net.au>
> Para: centos at centos.org
> Enviadas: Sexta-feira, 9 de Setembro de 2011 9:17:50 (GMT-0300) Auto-Detected
> Assunto: Re: [CentOS] CentOS5 with Dell Broadcom iSCSI Offload, does it work ?
>
> On 09/09/2011 10:05 PM, Drew Weaver wrote:
> > Hi,
> >
> >   I could be wrong here but don't you go into the Broadcom NIC
> > configuration while the server is booting and add the iSCSI target in
> > there and then it should appear as "Just Another Volume (TM)" to the
> > operating system?
> >
> > I've never tried it but I assume thats how the 'offloading' works.
> 
> close, but not. TCP Offloading (obviously) offloads the TCP protocol 
> processing from the server CPU onto the server adapter. This preserves
> valuable CPU cycles for applications processing and improves overall 
> server performance and network efficiency, as the heavy lifting for 
> iSCSI is done on the card, by the card.
> 
> <snip>
> >     After finding multiples answers to this question via google, but
> > without making it work on my servers. Has anybody iSCSI Offload 
> > working on a Dell Server with Broadcom NICs ?
> 
> Depending on how the Broadcom is in the server (ie - Physical Card vs
> onboard mezzanine card) depends on if you enable it in BIOS, or switch
> into the firmware when prompted at boot time to enable it.

Hi,

  Well... I looked into the NICs config to make sure it's on. On BIOS I can
see the 4 onboard NICs. On these I can chage from "no boot" / PXE / iSCSI boot. And 
the BIOS "detect" that this NICs are TOE & ISOE "capable". I had PXE on the first one 
and "no boot" on the other 3. From the NICs firmware menu I can see all the 8 NICs 
(including the 4 onboard), I can change the same options, no option to enable/disable iSCSI
Offload, only iSCSI boot.

  I'm trying the instructions from:

   - http://www.open-iscsi.org/docs/README
   - http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/pdf/Online_Storage_Reconfiguration_Guide/Red_Hat_Enterprise_Linux-5-Online_Storage_Reconfiguration_Guide-en-US.pdf
   - http://webpages.marshall.edu/~wolfe21/papers/rhel5-iscsi-HOWTO.pdf

  All of these say almost the same, and I can configure open-iscsi to access the EMC SAN using the two 
broadcom NICs using dm-multipath. But when I tryed to use the second MAC on the NIC (to activate
iSCSI Offload) then I can't connect on the SAN. I can discovery it using the second MAC, but can't connect.

  Using "default" iface, it will conect using eth1 or eth2:

[root at vHst-153-NEAD03 ~]# iscsiadm -m discoverydb -t st -p 10.252.1.1:3260 -I default --discover -P 1
Target: iqn.1992-04.com.emc:cx.ckm00091500140.a2
	Portal: 10.252.1.1:3260,1
		Iface Name: default
Target: iqn.1992-04.com.emc:cx.ckm00091500140.b2
	Portal: 10.252.1.2:3260,2
		Iface Name: default
Target: iqn.1992-04.com.emc:cx.ckm00091500140.a3
	Portal: 10.252.2.1:3260,3
		Iface Name: default
Target: iqn.1992-04.com.emc:cx.ckm00091500140.b3
	Portal: 10.252.2.2:3260,4
		Iface Name: default

[root at vHst-153-NEAD03 ~]# iscsiadm -m node -I default -lLogging in to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.a3, portal: 10.252.2.1,3260]
Logging in to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.b2, portal: 10.252.1.2,3260]
Logging in to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.b3, portal: 10.252.2.2,3260]
Logging in to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.a2, portal: 10.252.1.1,3260]
Login to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.a3, portal: 10.252.2.1,3260] successful.
Login to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.b2, portal: 10.252.1.2,3260] successful.
Login to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.b3, portal: 10.252.2.2,3260] successful.
Login to [iface: default, target: iqn.1992-04.com.emc:cx.ckm00091500140.a2, portal: 10.252.1.1,3260] successful.

[root at vHst-153-NEAD03 ~]# iscsiadm -m session
tcp: [5] 10.252.2.1:3260,3 iqn.1992-04.com.emc:cx.ckm00091500140.a3
tcp: [6] 10.252.1.2:3260,2 iqn.1992-04.com.emc:cx.ckm00091500140.b2
tcp: [7] 10.252.2.2:3260,4 iqn.1992-04.com.emc:cx.ckm00091500140.b3
tcp: [8] 10.252.1.1:3260,1 iqn.1992-04.com.emc:cx.ckm00091500140.a2

[root at vHst-153-NEAD03 ~]# iscsiadm -m session -P 1
Target: iqn.1992-04.com.emc:cx.ckm00091500140.a3
	Current Portal: 10.252.2.1:3260,3
	Persistent Portal: 10.252.2.1:3260,3
		**********
		Interface:
		**********
		Iface Name: default
		Iface Transport: tcp
		Iface Initiatorname: iqn.1994-05.com.redhat:vHst-153-NEAD03
		Iface IPaddress: 10.252.2.153
		Iface HWaddress: <empty>
		Iface Netdev: <empty>
		SID: 5
		iSCSI Connection State: LOGGED IN
		iSCSI Session State: LOGGED_IN
		Internal iscsid Session State: NO CHANGE
<snip>

  But, if I try the same with the eth1 offload, I get:

[root at vHst-153-NEAD03 ~]# ifconfig eth1
eth1      Link encap:Ethernet  Endereço de HW BC:30:5B:FD:F7:E7  
          inet end.: 10.252.1.153  Bcast:10.252.1.255  Masc:255.255.255.0
          endereço inet6: fe80::be30:5bff:fefd:f7e7/64 Escopo:Link
          UP BROADCASTRUNNING MULTICAST  MTU:9000  Métrica:1
          RX packets:735 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          colisões:0 txqueuelen:1000 
          RX bytes:47040 (45.9 KiB)  TX bytes:620 (620.0 b)
          IRQ:26 Memória:d2000000-d2012800 

[root at vHst-153-NEAD03 ~]# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
bnx2i.bc:30:5b:fd:f7:e8 bnx2i,bc:30:5b:fd:f7:e8,10.252.1.53,<empty>,<empty>
bnx2i.bc:30:5b:fd:f7:ea bnx2i,bc:30:5b:fd:f7:ea,10.252.2.53,<empty>,<empty>

[root at vHst-153-NEAD03 ~]# iscsiadm -m iface -I bnx2i.bc:30:5b:fd:f7:e8
# BEGIN RECORD 2.0-872
iface.iscsi_ifacename = bnx2i.bc:30:5b:fd:f7:e8
iface.net_ifacename = <empty>
iface.ipaddress = 10.252.1.53
iface.hwaddress = bc:30:5b:fd:f7:e8
iface.transport_name = bnx2i
iface.initiatorname = <empty>
# END RECORD

[root at vHst-153-NEAD03 ~]# iscsiadm -m discoverydb -t st -p 10.252.1.1:3260 -I bnx2i.bc:30:5b:fd:f7:e8 --discover -P 1
Target: iqn.1992-04.com.emc:cx.ckm00091500140.a2
	Portal: 10.252.1.1:3260,1
		Iface Name: bnx2i.bc:30:5b:fd:f7:e8
Target: iqn.1992-04.com.emc:cx.ckm00091500140.b2
	Portal: 10.252.1.2:3260,2
		Iface Name: bnx2i.bc:30:5b:fd:f7:e8
Target: iqn.1992-04.com.emc:cx.ckm00091500140.a3
	Portal: 10.252.2.1:3260,3
		Iface Name: bnx2i.bc:30:5b:fd:f7:e8
Target: iqn.1992-04.com.emc:cx.ckm00091500140.b3
	Portal: 10.252.2.2:3260,4
		Iface Name: bnx2i.bc:30:5b:fd:f7:e8

[root at vHst-153-NEAD03 ~]# iscsiadm -m node -I bnx2i.bc:30:5b:fd:f7:e8 -l
Logging in to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.a3, portal: 10.252.2.1,3260]
Logging in to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.b2, portal: 10.252.1.2,3260]
Logging in to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.b3, portal: 10.252.2.2,3260]
Logging in to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.a2, portal: 10.252.1.1,3260]
iscsiadm: Could not login to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.a3, portal: 10.252.2.1,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.b2, portal: 10.252.1.2,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.b3, portal: 10.252.2.2,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: bnx2i.bc:30:5b:fd:f7:e8, target: iqn.1992-04.com.emc:cx.ckm00091500140.a2, portal: 10.252.1.1,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals

   Well, has anybody make this to work ? What I'm missing ? 

   Thanks in advance,

            Antonio.

-- 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Antonio S. Martins Jr. - Support Analyst | "Only The Shadow Knows     |
| Universidade Estadual de Maringá - Brasil|   what evil lurks in the   |
| NPD - Núcleo de Processamento de Dados   |       Heart of Men!"       |
| E-Mail: asmartins at uem.br / shadow at uem.br | !!! Linux User: 52392 !!!  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     "Real Programmers don’t need comments — the code is obvious."