Hello CentOS !
Iam trying to configure a Dual Core 1.6Ghz AMD E-350 machine to be able NIC (eth0) interrupts the most efficient way possible so that I can be able to handle the large number of random source packets per second. Currently running CentOS 6.4 2.6.32-358.6.2.el6.x86_64, i made some load testing with a random sourced syn flood to the NIC of this machine and found out according to top all the CPU interrupts are being sent to CPU0, thus saturating it and keeping the second core 100% idle because of this the amount of PPS handled is low (50k /sec). So.. below the output of /proc/interrupts, and also the output of the SMP affinity file which already contains "3" which I expect it tells the kernel to load balance the interrupts between the two cores and its not happening.
Questions: a.) Could anyody please help me with a solution so that iam able to balance equally the interrupts on both cores? b.) Is there anyway to actually assign more interrupts to this IRQ (25) or is that something that the kernel discretly takes care of?
Thanks
Alex
[root@node2 ~]# cat /proc/irq/25/smp_affinity 3
Btw I also tried setting this value to "2" and "1" to get to choose which core it handles, and that did work but again the problem is I can only use one core at a time and not balance!
[root@node2 ~]# cat /proc/interrupts CPU0 CPU1 0: 229 9838 IO-APIC-edge timer 1: 0 2 IO-APIC-edge i8042 8: 0 1 IO-APIC-edge rtc0 9: 0 0 IO-APIC-fasteoi acpi 12: 0 4 IO-APIC-edge i8042 14: 0 0 IO-APIC-edge pata_atiixp 15: 0 0 IO-APIC-edge pata_atiixp 16: 4 650 IO-APIC-fasteoi snd_hda_intel 17: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, ehc i_hcd:usb3 18: 0 4 IO-APIC-fasteoi ohci_hcd:usb4, ohci_hcd:usb5, ohc i_hcd:usb6, ohci_hcd:usb7 19: 7 2754 IO-APIC-fasteoi ahci 24: 0 31 PCI-MSI-edge snd_hda_intel 25: 1 876343 PCI-MSI-edge eth0
On 5/20/2013 9:53 PM, Alex Flex wrote:
a.) Could anyody please help me with a solution so that iam able to balance equally the interrupts on both cores? b.) Is there anyway to actually assign more interrupts to this IRQ (25) or is that something that the kernel discretly takes care of?
afaik, IRQs from a single device have to be handled serially, you can't have more than one active IRQ at the same time.
so there's really no point in distributing them across multiple CPU cores. let the other cores handle user space and other sorts of IOs.
SYN flooding is a pathological condition thats not going to happen under regular workloads, how does your system handle actual network traffic when it approaches network saturation ?
John,
We run a reverse proxy so our CPU need for that is very small, plus we get syn flooded often.. which is why we have the need to be able to load between cores, i think it is justified.
Thanks Alex
On 05/20/2013 10:48 PM, John R Pierce wrote:
On 5/20/2013 9:53 PM, Alex Flex wrote:
a.) Could anyody please help me with a solution so that iam able to balance equally the interrupts on both cores? b.) Is there anyway to actually assign more interrupts to this IRQ (25) or is that something that the kernel discretly takes care of?
afaik, IRQs from a single device have to be handled serially, you can't have more than one active IRQ at the same time.
so there's really no point in distributing them across multiple CPU cores. let the other cores handle user space and other sorts of IOs.
SYN flooding is a pathological condition thats not going to happen under regular workloads, how does your system handle actual network traffic when it approaches network saturation ?
On 5/20/2013 10:24 PM, Alex Flex wrote:
We run a reverse proxy so our CPU need for that is very small, plus we get syn flooded often.. which is why we have the need to be able to load between cores, i think it is justified.
IRQ handlers are not reentrant, as the hardware can't deal with it.
you can only have one CPU thread at a time in the IRQ handler for a specific device, so you might as well just handle one device with one CPU. if you have several ethernet adapters, then I'd hope the IRQs could be distributed
one device interrupting both cores would cause a lot of trouble with hardware contention, or it would need to be all spinlocked which would make it worse, you'd be CPU bound in TWO cores in spinlocks half the time.
but again, a Server adapter like a Intel Pro1000 or whatever, something like... http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2
will do much better than some $5 realtek junk.
Thanks John!,
I understand now. I think there are some NICs that have different queues which are connected to the kernel in the form of different IRQs, and as thus can be load balanced nicely. Ive seen that on 10GigE NICs.. but iam wondering if by any chance you know if the one suggested can do such, or the name of the feature Iam looking for so I can browse myself?
Alex
On 20 May 2013 23:45, John R Pierce pierce@hogranch.com wrote:
On 5/20/2013 10:24 PM, Alex Flex wrote:
We run a reverse proxy so our CPU need for that is very small, plus we get syn flooded often.. which is why we have the need to be able to load between cores, i think it is justified.
IRQ handlers are not reentrant, as the hardware can't deal with it.
you can only have one CPU thread at a time in the IRQ handler for a specific device, so you might as well just handle one device with one CPU. if you have several ethernet adapters, then I'd hope the IRQs could be distributed
one device interrupting both cores would cause a lot of trouble with hardware contention, or it would need to be all spinlocked which would make it worse, you'd be CPU bound in TWO cores in spinlocks half the time.
but again, a Server adapter like a Intel Pro1000 or whatever, something like... http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2
will do much better than some $5 realtek junk.
-- john r pierce 37N 122W somewhere on the middle of the left coast
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thanks John,
You have been most helpful.
Do you happen to have any suggestion to the following:
In my lab test with two identical Core I5 machines I created a syn flood on a gigabit switch: In the originating machine I was able to get 800k pps using one core only and that core was used at 75% with top reporting most of the consumption was at SYS, and SI almost nothing. At the recieving side the core assigned to the IRQ of the NIC was maxed with 100% SI (syn cookies was disabled).
Why is clearly so significantly expensive for the recieving side rather than the sending side to process a syn flood if they have identical hardware? Oh and btw the recieving side could only do 400k pps.
Thanks! Alex
On 05/20/2013 11:45 PM, John R Pierce wrote:
On 5/20/2013 10:24 PM, Alex Flex wrote:
We run a reverse proxy so our CPU need for that is very small, plus we get syn flooded often.. which is why we have the need to be able to load between cores, i think it is justified.
IRQ handlers are not reentrant, as the hardware can't deal with it.
you can only have one CPU thread at a time in the IRQ handler for a specific device, so you might as well just handle one device with one CPU. if you have several ethernet adapters, then I'd hope the IRQs could be distributed
one device interrupting both cores would cause a lot of trouble with hardware contention, or it would need to be all spinlocked which would make it worse, you'd be CPU bound in TWO cores in spinlocks half the time.
but again, a Server adapter like a Intel Pro1000 or whatever, something like... http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2
will do much better than some $5 realtek junk.
Am 21.05.2013 20:05, schrieb Alex Flex:
Why is clearly so significantly expensive for the recieving side rather than the sending side to process a syn flood if they have identical hardware?
Sending is fundamentally less work than receiving.
The sender just puts whatever it wants to send on the line. The receiver has to recognize it, analyze it, find out whether it is the intended recipient, match it with what it received before, keep state, etc.
In the case of a SYN flood, the sender exacerbates this on purpose, by reducing its own workload (exploiting the fact that it doesn't really want to communicate) and maliciously increasing the receiver's workload (forcing it to maintain enormous numbers of half-open connections).
So it's really quite unsurprising that a SYN flood puts less load on the sender than on the receiver.
John,
On 05/20/2013 10:48 PM, John R Pierce wrote:
afaik, IRQs from a single device have to be handled serially, you can't have more than one active IRQ at the same time.
so there's really no point in distributing them across multiple CPU cores. let the other cores handle user space and other sorts of IOs.
SYN flooding is a pathological condition thats not going to happen under regular workloads, how does your system handle actual network traffic when it approaches network saturation ?
In summary, its very easy to hit a low number of packets/second flood and have the server start looking legitimate ones. I find it hard to believe that one cannot benefit from load balancing the interrupts?
Alex
On 5/20/2013 10:39 PM, Alex Flex wrote:
In summary, its very easy to hit a low number of packets/second flood and have the server start looking legitimate ones. I find it hard to believe that one cannot benefit from load balancing the interrupts?
how many network adapters do you have?
you can load balance different network adapters.
Hello again John, I only have one network adapter. GigE.
Perhaps anybody else can give me another suggestion in order to achieve this?
Thanks Alex
On 05/20/2013 11:38 PM, John R Pierce wrote:
On 5/20/2013 10:39 PM, Alex Flex wrote:
In summary, its very easy to hit a low number of packets/second flood and have the server start looking legitimate ones. I find it hard to believe that one cannot benefit from load balancing the interrupts?
how many network adapters do you have?
you can load balance different network adapters.
-
account Flexzor -
Alex Flex
-
John R Pierce -
Tilman Schmidt