topic Re: Issues with 32 bit RHEL 5.4 kickstart on HP blades in Operating System - Linux

Issues with 32 bit RHEL 5.4 kickstart on HP blades

David Knierim — Tue, 02 Feb 2010 18:40:39 GMT

I have been having intermittent problems doing network kickstarts to BL460 G1 and BL460 G6 blades. I have submitted a bugzilla report at Red Hat on the issue (https://bugzilla.redhat.com/show_bug.cgi?id=547746), but I have not heard anything from Red Hat in over a month.

I thought I would ask if others have seen the same issue.

What I am seeing is the kickstart fails in several places. In all cases, the problem is the network is not working. The two screens it stops at are "Error Unable to retrieve
http://169.254.104.4//TPD/rhel_5.4_i386/images/stage2.img" or "Configure
TCP/IP". I occasionally also see an error that a downloaded rpm file is corrupt.

The problem is intermittent. I would estimate that it happens about 10% of the time (If I kickstart 10 blades, one will show the problem), but if I continually do kickstarts, this problem will eventually happen to every blade.

I have not tested for this issue on rack mount servers.

If I run the same test with the same kickstart server and the same blades, but use RHEL 5.4 64 bit, the problem does not happen.

I am using 3020 switches with IOS version 12.2(50)SE3. I have upgraded the firmware on the blades using version 1.70 of the "HP BladeSystems Firmware Deployment Tool".

Has anyone else seen this issue? Can anyone offer an ideas on how to get it resolved?

Thanks,
David

Re: Issues with 32 bit RHEL 5.4 kickstart on HP blades

Matti_Kurkela — Wed, 03 Feb 2010 11:04:36 GMT

Those error messages alone would make me strongly suspect network congestion. But when it appears only with the 32-bit OS, it might be a driver problem instead.

Do you have any network statistics available, or have you otherwise made certain that there was no temporary network congestion when you were installing the 32-bit blades?

The BL460 G1 and G6 are both designed to be 64-bit systems from the beginning, and I'd guess the overwhelming majority of them are now being installed with 64-bit OSs, so it is possible that a bug that only gets triggered in 32-bit code might have escaped notice.

I'm afraid you'll need network traces to get to the bottom of this.

We don't (yet) use 3020 switches at our site, so I don't know much about them. But I'd expect them to have the port monitoring functionality, where all the traffic of one switch port can be mirrored to another port.

For example, configure a 3020 to mirror all the traffic of a 32-bit blade to a 64-bit blade on the same enclosure. On the 64-bit blade, use tcpdump to capture e.g. the full content of all DHCP traffic ("tcpdump -s 0 -w trace.dat [any other options]"). Make sure the blades' clocks are reasonably in sync before starting the trace: it may help in correlating the error messages with the trace.

Use Ethereal/Wireshark to analyze the trace files: it allows you to easily find any corrupted packets in the trace.

If you see DHCP-related errors ("Configure TCP/IP" or "Error unable to retrieve http://") while the trace shows the DHCP server in fact did send a valid DHCP response to the 32-bit blade, you'll have a pretty strong proof that something between the 32-bit OS and the 3020 switch (inclusive) is losing or corrupting packets.

If the trace indicates the DHCP responses really are missing or corrupted when the errors happen, then it would be a network or DHCP server problem. If the DHCP server and the Kickstart server are two separate hosts, take a hard look at the common components on the network route from them to the blade(s).

MK

Re: Issues with 32 bit RHEL 5.4 kickstart on HP blades

Steven E. Protter — Wed, 03 Feb 2010 21:47:13 GMT

Shalom,

This is normally not a real IP address.

http://169.254.104.4/

Is this a symptom of the problem or is that the same address you get with 64 bit Linux.

The blade is certified with 32 bit and 64 bit RHEL. That does not mean something did not get missed, but I'm reluctant to agree that this is a problem with how HP did the hardware.

It might be the switches. Make sure they comply with support matrix from Cisco on this subject.

You could just go 64 bit for a quick fix. Most 32 bit RH apps will run just fine on 64 bit RHEL.

SEP