Cannot balance network interrupts

[Harryy04]

I am trying to balance network interrupts across CPUs on a multi core server. But somehow, I always see two CPUs having much more interrupts than other CPUs configured. My server is HP Proliant DL360 G7, with 2 physical CPU/6 cores/24 logical CPUs. My NIC is Broadcom NetXtreme II BCM5709 1000Base-T, which has 8 message queues. 

My traffic is mostly udp messages on port 53.

I tried to bind each of the message queues to a CPU using smp_affinity, spreaded across physical CPUs and cores. I disabled irqbalance service.

My OS is Red Hat Enterprise Linux Server release 6.3 (Santiago). Kernel is:   2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux.

The default installation put all interrupts on CPU0. So it is overloaded when I push traffic.

adaptive-rx/tx are off and I could not enable them using ethtool.

I could see the CPUs I configured are receiving interrupts, but two of them (CPU3 and CPU7 -- this could change if I reboot/reset smp_affinity on the same CPUs) are receiving much more. Does anyone have any idea what could be wrong?

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23
59: 306 373 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-0
60: 842630 0 0 63841516 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-1
61: 153 0 0 0 0 704 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-2
62: 170903 0 0 0 0 0 0 7571156 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-3
63: 1968 0 0 0 0 0 0 0 2892 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-4
64: 518 0 0 0 0 0 0 0 0 0 2071 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-5
65: 1235 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2657 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-6
66: 104 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3588 0 0 0 0 0 IR-PCI-MSI-edge eth2-7

Thanks

Harry