Order of dual quad cards in DL380 G5 with RHEL 3

[Hartgers]

Problem:

I have many DL380 G5 servers. Each one is installed with 2 "HP NC364T PCIe 4Pt Gigabit Server Adptr" network cards. Usually, when I install Redhat Ent 3, the bottom card in PCI-X slot 4 get port assignment eth0-eth3 and the top card in slot 5 get ports eth4-eth7. With some of the servers, the order is inversed. So, the top quad card in slot 5 get numbers eth0-eth3.

One might think this is not a problem, as all the ports do work, but the documentation that the cabling on the back of the server should always be the same for every server, without exception.

Question: How can I force the port assignment of the network cards to start the numbering on slot 4 with eth0 - eth3 in the hardware or when installing Red-Hat Ent 3?

I have tried sofar to:

- Swap the names of the eth0-eth7 in redhat-config-network. e.g. eth6 -> eth2 and eth2 -> eth6, no result

- Look for an order assignment in the BIOS, nothing found.

- Look at the IRQ in the BIOS, but these seem to be random

- Physically swap the two network cards in the DL380 G5, without result.

[Steven E. Protter]

Shalom,

Question: How can I force the port assignment of the network cards to start the numbering on slot 4 with eth0 - eth3 in the hardware or when installing Red-Hat Ent 3?

You really can not.

You can clean the system and start from scratch.

Use the BIOS to disable the built in NIC cards, do a clean install with only the NIC card you want to be eth0 in it. You might get eth0. You however might get eth2 because even with the onboard cards disabled, they may still be visible to the OS.

I have taken a few tries at this over the years and the results have been frustrating.

SEP

[Hartgers]

Toda Raba for the quick response. I am going to trust you and try the full reinstall.

Unfortunately, it is an automated install that does not allow for temporary hardware removal.
The internal NICS are never used and luckily always get assigned eth8 and eth9.

On the VMware community ( http://communities.vmware.com/thread/138420 ) there is someone with the exact same hardware, but not the same problem. However, one of the replies there seems to suggest that the port assignment is also influenced by active cabling. I am not sure if this applies, but, VMware does appear to have / have had a Red Hat core. Therefor, I will try to disconnect all cabling on the second quad card and see if it has an effect after a reinstall.

[Steven E. Protter]

Bevekasha,

Sadly you are stuck with the results. The only way to manipulate the process is a clean install of the OS minus part of the hardware.

That never worked for me.

So you need to adjust to the reality of how RHEL detects and assigns eth numbers.

Sorry I could not help further.

SEP

[Matti_Kurkela · Accepted solution]

This is RHEL 3, so it uses Linux 2.4 kernel. I seem to recall that the device detection order standardization happened in 2.5 development kernel series.

With 2.6 kernels, the device detection works by walking through the PCI bus tree (see "lspci" listing). When a NIC is detected, a driver is loaded and it takes the first free eth* device by default. If the same driver can control more cards, it allocates eth* devices for them too.

In Linux 2.4.* and older kernels, each driver implemented the device detection separately and could do it in any order. Some NIC drivers used PCI bus tree order, others started allocating NIC names by selecting the lowest-numbered MAC address first. I guess the latter is the case for your NICs.

So, a hardware solution for new systems:

When building up the system, check the MAC addresses of the NIC cards. (Quad NICs usually have four consecutive MAC addresses.)
Always put the card with numerically lower MAC addresses to slot 4, and the other (with higher MAC address numbers) to slot 5.

A software solution for existing systems:

Edit the /etc/sysconfig/network-scripts/ifcfg-eth* files. Add to *each* file a line with HWADDR=. You must add this to all NICs that need to be swapped: if only one of the pair to be swapped has this configuration line, there is a name conflict and renaming does not happen. Reboot to verify the port assignments.

Note: this strategy is the default in RHEL 4 and above. When a NIC is detected for the first time, its MAC address is written as a HWADDR line in its /etc/sysconfig/network-scripts/ifcfg-eth* file. This makes sure that once a NIC is configured using a particular name, that name stays with the NIC even if other NICs are removed or the NIC order is changed.

This causes much head-scratching for a sysadmin who is not aware of how the system works (been there, done that).

Note 2: the HWADDR configuration variable is different from the MACADDR variable. HWADDR means "this configuration file is meant for a network port with this MAC, rename NICs if necessary to make it so", while MACADDR means "set a custom MAC address for this NIC". You should never use these two variables together.

MK

[Hartgers]

I have looked at five DL380 G5 servers and one of the five does not have the lowest MAC addresses showing up first. I am therefore not sure if it is true that the NIC with the lower MACs will always receive the lower eth* numbers during a Red Hat Ent 3 install.

The issue is also that DL380 G5 servers that were purchased earlier have never shown this behaviour. They have always been assembled by HP and I do not believe they would always have paid attention to the NIC order on the riser card.

I still have another five servers in another location that have all installed with the correct NIC order. To install Red Hat and COTS, I always use the same procedure to install, with a kickstart file and self-written scripts.

I will also check the version of the BIOS. Possibly, a newer version introduced an unwanted feature. The BIOS I see on the five servers that "misbehave" is P56 11/02/2008 . I could not find that particular BIOS on the HP website however.

(I Googled for "Systems ROMPaq Firmware Upgrade for HP ProLiant DL380 G5 (P56) Servers")

Thanks for the HWADDR hint, it provides a good workaround and actually works very well. The only problem is that someone that is unaware might have a problem at a reinstall or after the replacement of a broken NIC. I will continue testing for now and look for a permanent and repeatable solution.

[Matti_Kurkela]

What's the name of the Linux driver module for your NICs? (If you are not sure, please post the "lsmod" listing from one of your servers.)

As I said, in 2.4 kernel series each driver implemented the hardware detection separately. Most drivers used either the PCI bus order or the MAC order, but the driver you're using might be one of the few that used a non-obvious algorithm.

If the driver module is one of the standard Linux NIC drivers, I might be able to take a peek at the source and find out how its detection algorithm works.

MK

[Hartgers]

It is an e1000 driver source rpm that I use. It has been modified to meet our special needs. I have attached the source rpm to this thread.

[Matti_Kurkela]

Looks like the e1000_probe() function just uses the pci device structure it is given, and does not do any clever multi-card probing on its own.

That would result in a PCI bus-based detection order, as far as I understand.

At this point, I'm almost stumped. I'll examine a few of our DL380 G5s at work, maybe it rings a bell. Unfortunately about all our G5s are running RHEL 4 or 5, not 3, so I probably don't have a chance to test the behaviour of a RHEL 3 on this platform.

Hmm... There is a HP customer advisory about a similar problem on Windows. Perhaps there is something at the hardware/firmware level that causes this?

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c00712319〈=en&cc=us&taskId=101&prodSeriesId=1121516&prodTypeId=15351

Have you checked the BIOS versions? Perhaps there is a commonality (= a particular BIOS version that causes a different slot detection order).

MK

[Hartgers]

Well spotted article on the W2003 NIC card issue.

My idea is also that the BIOS has something to do with it. Unfortunately, I am not allowed to upgrade the BIOS/firmware on any of the G5 servers that we have. It would have been nice to test at least one of them with the latest version that came out some weeks ago.

Although I cannot find in the release notes/bug fixes that there has been a PCI order issue, it may still have been solved. The reason is also that the BIOS I mentioned previously, cannot be found in the official release list.

What I hope to do one day is to test the newer BIOS, and until then use workarounds. Thanks a lot.

JH

[Steven E. Protter]

Shalom,

BIOS is the most likely path to a fix.

You should try IMO to budget for a sandbox server to test this out.

SEP

[Hartgers]

I have reached an agreement and updated the firmware to P56 05/18/2009.

Although a quick reinstall did not solve the issue, I have some new hope.

The Enhancements mentioned:

"Added support for an Advanced ROM Based Setup (RBSU) Option to disable PCI Bus Padding. By default, the ProLiant System ROM will pad the number of PCI buses assigned to prevent PCI bus numbers for existing devices from hanging when expansion cards are added to the system. This may prevent an operating system from attempting to re-enumerate existing devices when an expansion card is initially installed. Disabling this option may work around issues with certain expansion cards that do not properly handle the padding of PCI busses."

Also, when looking for "re-enumerate PCI devices" on google I found some people that were able to resolve some ethernet ordering issues by adding lines in the kickstart files on RHEL5.3.

First I will have a go at disabling the PCI bus padding in the BIOS.

[Hartgers]

Someone has spotted for me that the old G5 has a completely different riser card. The riser card in the G5 that we have, has two full length PCI-Express and one shorter PCI-Express slot. I was under the impression that the third slot (PCI3) was a PCI-X slot, but it is not.

I moved the upper card (PCI5) to the shorter PCI-Express slot. It should be sufficient to still get the full speed of the card in this slot.

I am lucky, now the order of the ports is as I like it.

Thanks to all for your input.