topic HP DL145 G3 ethernet interrupt sharing in Operating System - Linux

HP DL145 G3 ethernet interrupt sharing

Benny Lyne Amorsen — Tue, 28 Oct 2008 20:52:05 GMT

We have a DL145 G3 with 2 dual-core Opterons and two NC364T quad-ethernet.

The interrupts are unfortunately shared:

32: 145724432 3 1771329 9121 IO-APIC-fasteoi eth6, eth7, eth8, eth9
35: 0 439014411 115 30272 IO-APIC-fasteoi eth2, eth3, eth4, eth5
36: 0 16 152 722287939 IO-APIC-fasteoi eth0, eth1

The largest problem is that eth0 and eth1 share interrupts, since they are busiest. This means that the 4th core gets severely overloaded, while the 3rd core is almost completely idle. I was hoping to reduce the overload by switching to quad-core CPU's, but as it is, that would just lead to 5 idle cores instead of 1.

Is there a HP DL server which can provide unique interrupts to at least the two built-in ethernets and each of the ports on a quad-ethernet card? Would a different NIC solve the problem? Do I need to pass any options to the Linux kernel (hopefully not, interrupt routing seems to be deep magic these days.)

Re: HP DL145 G3 ethernet interrupt sharing

Matti_Kurkela — Wed, 29 Oct 2008 14:24:32 GMT

Which Linux distribution are you running?

RHEL has the "irqbalance" process, which is supposed to auto-tune IRQ settings so that all processors/cores get used equally. I think irqbalance is available in other distributions too, but perhaps it is not installed by default.

How fast are those counters increasing on average, i.e. what was the server's uptime at the time you got those values?

MK

Re: HP DL145 G3 ethernet interrupt sharing

Benny Lyne Amorsen — Wed, 29 Oct 2008 17:14:36 GMT

I am using irqbalanced, but that doesn't help when interrupts are shared. You can't spread the load from a shared interrupt between two different processors.

Hence my request for a server which can provide unique interrupts...

Re: HP DL145 G3 ethernet interrupt sharing

Benny Lyne Amorsen — Mon, 10 Nov 2008 11:24:48 GMT

It turns out that the DL160G5 can do it with the NC364T, at least for one card. I'll have to remove the SAS controller to test with 10 ports.

17: 15 10 6398 1380147367 IO-APIC-fasteoi eth0
18: 19 3237571 20 353767456 IO-APIC-fasteoi uhci_hcd:usb4, eth1
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3, ata_piix
23: 11 10 11 10 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb2
48: 26 1635373469 28 28 IO-APIC-fasteoi eth2
52: 27950 7711 4983 920 IO-APIC-fasteoi ioc0
58: 80 52635075 6 193398 IO-APIC-fasteoi eth3
62: 6 159171 17 326447 IO-APIC-fasteoi eth5
66: 10 323184 5 162442 IO-APIC-fasteoi eth4

eth0 and eth1 share interrupts with USB, so I should probably turn off USB.