DL585G6 NICs enumerating differently on (supposedly) identical servers

[Chris Warne_1]

Hi

Earlier this year we built several DL585G6 servers with two PCI dual-port NIC cards in addition to the in-built NICs.
In every case, the on-board NICs were enumerated before the PCI NICs.

We have now built two more servers, supposedly identical, but both enumerate the PCI NIC ports before the on-board ones!

All the hardware is, on the face of it, identical, as are all the firmware revs and the server console configuration.
Although the part numbers are the same, there are probably many months between the build dates of the first lot of servers and second, so could there be a subtle difference between the two? This was the explanation I was given by an HP contact, although in retrospect it may have just been an educated guess!

I know I could re-assign the ports at the OS level, but this will mask the differences in the servers rather than solve the underlying problem, and having absolute consistency across all servers is a major issue for the company.

Has anyone experienced this before? Is there any way around it or is what I’ve been told correct; the "same" hardware can be subtly different.

The exact hardware list of each server is as follows :

574409-B21 HP DL585R06 CTO Chassis
539848-L21 HP O8431 DL585G6 FIO Kit
458492-B21 HP NC382T PCIe Dp Gigabit Adptr
458492-B21 HP NC382T PCIe Dp Gigabit Adptr
AH627A HP U320e SCSI Bus Adapter
462864-B21 HP P410/512 BBWC Smart Array

Any thoughts greatfully received!

Chris

[SMR]

Hi!

You forgot to mention the OS version & update/sp.

I believe the job of discovering devices on the PCI bus is entirely driven by the OS, take a look at this old document.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01430330〈=en&cc=us&taskId=101&prodSeriesId=428936

So that makes me think the hardware would hardly have such a drastic change in design between revisions to create a different enumeration on the same parts over time.

[Chris Warne_1]

The OS is RHEL 4.8 .
But the build is identical across all the servers - they are effectively clones. Aside from nodename and IP ionformation there is no difference at all. This is why I'm sure it's a hardware issue, or at least related to the hardware. I know that there's 100% no difference in the composition or configuration from an O/S point of view.

[rick jones]

Linux is infamous for quasi-random device enumeration - even with identical hardware. The generally suggested workarounds involve setting-up udev rules or the like to hardwire NIC enumeration.

[Chris Warne_1]

Thanks for the replies.
After some further invesatigation it seems the hardware is NOT identical, hence the problem! (HP have changed the chipset in the PCI NICs, despite the part number being the same.)

So it looks like we'll have to hardcode the MACs, which is a pain as it means implementing a "software" change.

Rick - the simplest method would be to add the MAC addresses to the ethn-ifg files, but are you aware of any issues this can have? All our interfaces are bonded, run VLANs and clustering, and bonding especially can be a bit fussy!

Chris

[rick jones]

I've done the ethN-cfg file bit before and it was sufficient to my needs at the time - can't say I was doing bonding though. I have vague recollections of perhaps needing to iterate it once or twice.

I don't know that the 4.8 bits have a very sophisticated udev, but I believe that in later revs it is possible to do the interface enumeration using PCI addressing - that could have the nice property of allowing the one udev file to be shared across identically configured systems, and preserving instance naming in the face of hardware replacement. It makes the naming a bit more like HP-UX (what I consider a good benchmark in this area) which bases instance enumeration on hardware path and card type not MAC addresses.

[Chris Warne_1]

Well, Redhat have now assured me it'll work with 4.8, so we'll see!

Anyway, thanks for the help, and I'll close this now as the initial issue has been resolved, or at least I now know why it's happening!

Chris