topic Re: rx7640 11.23 Boot failure in Operating System - HP-UX

rx7640 11.23 Boot failure

Pupil_1 — Tue, 04 Sep 2007 17:49:55 GMT

Team,

I've 2 rx7640s running 11.23, each having 3 vpars (total of 6 vpars on 2 different frames) with no local disk, booting from an XP12K. The problem is that the reboot of each the 4 nodes that I've checked (vpar1 and 2 in frame1 and vpar1 and 2 in frame2) won't boot on a reboot command. I'll have to wait for about 5 mins to find the boot device. I get the error path not found and after waiting for close to 5 mins, it boot normal.
I've not been able to reboot the 3rd vpar on both the frames is because I've not got the downtime for them, but suspect the same issue with them as well. Wait for close to 5 mins and then issue a vparboot -p , it comes back like a beauty !!

syslog.log, fcmsuitl, dmesg and cstm looks fine. Any ideas?

Re: rx7640 11.23 Boot failure

Steven E. Protter — Tue, 04 Sep 2007 18:28:31 GMT

Shalom,

This issue happens before any of those log files are open, because they are opened after boot.

You are going to need to look to the console for messages.

SEP

Re: rx7640 11.23 Boot failure

Pupil_1 — Tue, 04 Sep 2007 19:05:18 GMT

Missed the .txt file in the org post.

Re: rx7640 11.23 Boot failure

Torsten. — Wed, 05 Sep 2007 00:24:34 GMT

Some things to check:

- servers firmware
- FC HBA firmware
- OS patches (e.g. boot loader patches)
- vPars version

All of them are involved in the boot process.

Re: rx7640 11.23 Boot failure

Pupil_1 — Wed, 05 Sep 2007 09:44:11 GMT

I've 14 rx7640s, almost all of them vpar'ed, with the sept 06 patch level. All have been built with the same golden image and have the same firmware and patches.

Could this be a faulty HBA? cstm and syslog does not give any indication of that.

Thanks.

Re: rx7640 11.23 Boot failure

Torsten. — Wed, 05 Sep 2007 10:59:18 GMT

Hmm, all you have is this message:

"Load of 0/0/14/1/0/4/0.51.93.0.0.0.2 failed: Not Found"

But your fcmsutil and ioscan is showing

0/0/12/1/0/4/0
1/0/12/1/0/4/0

There is no 0/0/14 path.

Did you check your vpar config?

Regarding firmware: check the HBA firmware with this:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3365410&swItem=ux-49434-1&prodNameId=3365414&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

(display the version with "fcmsutil /dev/fcd2 vpd")

Re: rx7640 11.23 Boot failure

Pupil_1 — Wed, 05 Sep 2007 12:40:02 GMT

Torsten,

Thanks for the link, I'm checking that !!

About the path, my mistake...copy paste err... The CL from mp is better.

Anything specific with the vpar that you are suggesting to check?

Re: rx7640 11.23 Boot failure

Torsten. — Wed, 05 Sep 2007 13:01:45 GMT

Looks like your current EFI driver/firmware version for the FC HBAs is
driver 1.05, firmware 4.00.070
but the current version is
1.09/4.00.90

I would consider to update this.

Re: rx7640 11.23 Boot failure

Pupil_1 — Thu, 06 Sep 2007 13:58:24 GMT

The EFI driver/firmware was tried and did NOT fix the problem. The new one that was tried was 1.09/4.00.90.

HP will be bringing in the SAN packet sniffer to analyse the SAN Frames in the after noon.

Will update once I've details.

Thanks

Re: rx7640 11.23 Boot failure

Pupil_1 — Fri, 07 Sep 2007 13:07:07 GMT

No finding on this problem yet, the fabric analyser is still not able to give a conclusion !

We might want to try the same HBA on another 7640 and boot from it.

Thanks

Re: rx7640 11.23 Boot failure

Tim Nelson — Fri, 07 Sep 2007 13:16:58 GMT

My non-constructive comment is this is the type of potential issue that is very difficult to troubleshoot when booting from SAN devices in lieu of local disk.

I would be interested in the resolution.
Please be sure to follow up.

Tim

Re: rx7640 11.23 Boot failure

Torsten. — Fri, 07 Sep 2007 13:30:14 GMT

IMHO the OS is not related, because you get the "not found" message from EFI.
From this point is must be firmware or hardware related.

I would check the switch and disk array firmware + logs. Maybe you even find a high error rate related to the FC connection. In this case it could be a bad cable or transceiver.
I'm sure you have an alternate path to the boot disk.
Try to swap primary and alternate path in your vPar config.

Re: rx7640 11.23 Boot failure

Pupil_1 — Fri, 07 Sep 2007 13:43:14 GMT

We have 2 paths to the XP. Tried both the paths, did not work. Same error.

Ran a separate FC to the director switch port, no luck either !

Thanks.

Re: rx7640 11.23 Boot failure

Pupil_1 — Fri, 07 Sep 2007 13:59:42 GMT

I thought that I've already said this, but looks like i missed it..

The switch port if giving port fault errors and too many of them when the box comes up for the first time, but on the next attempt, no errors on the port !!

Thanks.

Re: rx7640 11.23 Boot failure

Tim Nelson — Fri, 07 Sep 2007 15:10:19 GMT

Something else worth reading.

http://docs.hp.com/en/7408/HPUXSANBootWP_final_v2_update.pdf

Re: rx7640 11.23 Boot failure

Tim Nelson — Fri, 07 Sep 2007 15:14:19 GMT

Here is another doc with known FW issues for booting from SAN.

http://docs.hp.com/en/J6374-90053/ch01s05.html

Re: rx7640 11.23 Boot failure

likid0 — Mon, 10 Sep 2007 04:12:37 GMT

The problems i have had in the past, booting from san, with vpars is that when you get a single change on the san net infrastructure(switch port for example), the hdw path to the disk changes and it wont boot again, getting the :not found error when triying to boot,if something has changed in your san that could well be the problem, to fix it I boot from ignite,find the new hdw path and export,import vg00, change boot paths, etc.

hope it helps

Re: rx7640 11.23 Boot failure

Pupil_1 — Mon, 10 Sep 2007 09:11:08 GMT

Thanks for the info Daniel.

I'm aware that nothing on the SAN has changed. There are close to 14 other vpars thru the same SAN switch that does not have this problem !! I cannot have 6 of them reinstalled without getting to the root of the problem though.

Again, the paths are visible, I'm able to see the efi structure in cxtxdxs1 when I walk through it manually from the efi shell when I've the box to come up in npar mode.

Thanks

Re: rx7640 11.23 Boot failure

likid0 — Tue, 11 Sep 2007 02:33:31 GMT

Ok, if you get to see the disks with the correct path in npar/efi shell, then thats not the problem for sure.

Re: rx7640 11.23 Boot failure

Pupil_1 — Tue, 11 Sep 2007 09:13:43 GMT

We've managed to figure out that the person who built these boxes had messed up lot of things. The setboot was not set correctly on 4 of the vpars and once this was fixed, all the errors has gone away. It certainly does not explain why would a machine boot after 5 mins and certainly does not explain why would the port fault on the SAN switch occur because of an incorrect Boot path setting !

Anyways, we have almost all of them sorted out. Thanks for all those who responded. N'joy you penny !

Must say, had one HBA failure on the same node yesterday night and there are still some mess that needs to be cleaned up.