Proliant DL360p, PCIe cards and NUMA nodes

[Stefan Puiu]

Hi,

We have an app running on an HP DL360p processing lots of network packets. The system has 2 CPUs, each being its own NUMA node, and runs Ubuntu Server 12.04. We've connected one HP 560SFP+ 10G adapter to slot 1; that card gets assigned to NUMA node 0, according to /sys/class/net/eth4/device/numa_node. I guess that's down to how that PCIe slot is connected - probably it's wired to node 0. However, after installing another NIC of the same model in slot 2, I found it was also getting assigned to node 0. Apparently none of the slots are assigned to node 1.

Is this by design? This being down to hardware wiring, I would expect this not to be configurable, right? Or is there a way to map a PCIe adapter to the other NUMA node?

Is this documented in the server specs? At least the datasheet doesn't specify anything about this...

Thanks in advance,

Stefan.

[Torsten. · Accepted solution]

The single riser card in the DL360p Gen8 is I/O controlled by the first CPU, but in the DL380p Gen8 each riser card is controlled by a different CPU, AFAIR.

[Stefan Puiu]

Torsten,

I think you are indeed right. I looked again at the quickspecs (http://h20195.www2.hp.com/v2/GetDocument.aspx?docname=c04128242&doctype=quickspecs&doclang=EN_US&searchquery=&cc=us&lc=en) and in the "Expansion slots" section it says: "Activated when installed with processor 1". I guess I was looking for too fancy keywords like NUMA, etc :).

I had tried to reply earlier, but had trouble logging in to the site most of the week...