Anyone have experience with any? I've been having a real hard time finding info on any cards that actually support this under linux. (most of the cards work but I don't see drivers that actually offload the TCP stack)
I have seen some comments where kernel developers don't like the idea as well.
I'm running CentOS 4.6 64-bit on a dual proc quad core system with 8GB memory.
I have a situation where I'm trying to make the most of my hardware here, a web application that serves up hundreds of tiny requests per second(each taking typically sub 100ms to complete).
With my load balancer in it's "normal" mode I can pump about 1000 requests a second through the system, before exhausting the TCP stack on the server(all ~65k sockets in use). CPU usage tops out at about 75%. The "normal" mode terminates idle connections forcefully after 25 seconds.
If I tell my load balancer to terminate idle connections *immediately* instead of waiting 25 seconds, I get similar CPU usage but transactions per second drop by about 30-33%.
During my tests I've also gotten 4 kernel panics, the last two I have captured and they are virtually identical, with the exception of one time it panic's on the java process, another time it panics on the swapper process. Both times the load balancer was configured to terminate connections faster than 25 seconds. I've done several searches and can't find anything remotely related to the below panic.
Kernel BUG at tcp_output:943 invalid operand: 0000 [1] SMP CPU 2 Modules linked in: dell_rbu md5 ipv6 autofs4 i2c_dev i2c_core nfs lockd nfs_acl sunrpc dm_mirror dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp bnx2 ext3 jbd ata_piix libata megaraid_sas sd_mod scsi_mod Pid: 10645, comm: java Not tainted 2.6.9-67.ELsmp RIP: 0010:[<ffffffff802e1a12>] <ffffffff802e1a12>{tcp_retransmit_skb+639} RSP: 0000:0000010006963e98 EFLAGS: 00010202 RAX: 00000101687cd9c0 RBX: 00000102266d2a80 RCX: 00000101687cd9c0 RDX: 00000101dec72e80 RSI: 0000000000000350 RDI: 0000000000000010 RBP: 000001015c0f7700 R08: 0000000000000008 R09: 0000000000000100 R10: 0000000000000000 R11: 0000000000000008 R12: 000001007b64ccb0 R13: 00000000000005b4 R14: 000001007b64ccb0 R15: 000001007b64c980 FS: 0000000048fe0960(005b) GS:ffffffff804f2e80(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000002adc82e028 CR3: 000000000694e000 CR4: 00000000000006e0 Process java (pid: 10645, threadinfo 0000010055cba000, task 000001005633c7f0) Stack: 0000010006963ec8 0000020000102a93 000001007b64ccb0 0000000000000001 000001007b64c980 0000000000000008 000001007b64ccb0 000001007b64ccb0 0000002c273fc4d0 ffffffff802e3f6d Call Trace:<IRQ> <ffffffff802e3f6d>{tcp_write_timer+1059} <ffffffff802e3b4a>{tcp_write_timer+0} <ffffffff80140945>{run_timer_softirq+356} <ffffffff8013cff0>{__do_softirq+88} <ffffffff8013d099>{do_softirq+49} <ffffffff80110bf5>{apic_timer_interrupt+133} <EOI>
Code: 0f 0b 48 c1 34 80 ff ff ff ff af 03 48 8b 43 10 4c 63 f6 ff RIP <ffffffff802e1a12>{tcp_retransmit_skb+639} RSP <0000010006963e98> <0>Kernel panic - not syncing: Oops
From what I can tell the Broadcom chips in this server support TOE
but the driver in linux does not implement it. I'd like to find a card that really has TOE in linux, if for nothing else to compare how it performs vs what I have now. With the goal of pumping out the highest number of transactions per second possible with the hardware/power utilization.
thanks
nate