NIC enumeration issues

[John Cagle]

This topic is to discuss problems with various Linux kernels and the way they number the NICs (eth0, eth1, eth2, etc.).

[John Cagle]

A customer has reported an issue with SystemImager on the BL480c, where NIC1 is enabled for PXE by default, and SystemImager boots, but then it discovers NIC3 as "eth0". This breaks the SystemImager installation because (1) NIC3's MAC address is different from NIC1, and (2) NIC3 may not be connected to the PXE network. Here is the solution I proposed: This is not a simple issue to explain, but, in the end, it is a Linux kernel issue. Server hardware designers have always relied on PCI (Bus/Device/Function) as the mechanism to order the NICs. Current Linux kernels use various methods to enumerate the NICs, some of them breadth-first and some depth-first. Neither method is “correct� according to the way modern servers are designed (at least those with multiple NICs). RHEL4 Update 5 will contain a workaround, and you can get a test kernel here: http://people.redhat.com/~jbaron/rhel4/ The workaround will discover the NICs in the proper order (NIC1, NIC2, NIC3, NIC4) by sorting them using their PCI Bus, Device, and Function numbers. So basically, you have three options – 1) Enable PXE only on NIC3, then this will match “eth0� as enumerated by the SystemImager kernel. 2) Modify the SystemImager kernel to name NIC1 as “eth0�, perhaps using a script that can detect the PXE MAC address and then use that to dynamically determine which NIC is NIC1. 3) Use RHEL4 Update 5’s kernel for SystemImager.

[Frank Sorenson]

We saw this issue as well, essentially with all HP hardware (and only HP hardware so far). We've found that using "ksdevice=MAC_ADDR" on the kernel boot line solves the problem for us. For example: KERNEL vmlinuz-rhel5-64 APPEND initrd=initrd-rhel5-64.img ks=nfs:w.x.y.z:/vol/ks.cfg ksdevice=00:1a:4b:dd:cd:d8 ip=w.x.y.z netmask=255.255.255.128 gateway=w.x.y.z dns=w.x.y.z Not sure if this will help SystemImager, but it works well for standard kickstart.

[John Cagle]

Thanks for the tip! I'm sure it will help someone else. Also, RHEL4 Update 5 has the fixes for HP BladeSystem products, so "eth0" should now correspond to NIC1 (the default PXE NIC) as long as you use an Update 5 kernel. By the way, this issue is not HP-specific. It happens on other brands too.

[violetlight]

Has anyone seen a fix for SLES10 or SLES10SP1? I'll chase this with Novell too. Thanks Simon.

[John Cagle]

We shared this patch with Novell, but they chose to not include it in their kernels. If you feel that it's necessary in SLES kernels, please send your feedback to Novell.