rx7640 11.23 Boot failure

[Pupil_1]

Team,

I've 2 rx7640s running 11.23, each having 3 vpars (total of 6 vpars on 2 different frames) with no local disk, booting from an XP12K. The problem is that the reboot of each the 4 nodes that I've checked (vpar1 and 2 in frame1 and vpar1 and 2 in frame2) won't boot on a reboot command. I'll have to wait for about 5 mins to find the boot device. I get the error path not found and after waiting for close to 5 mins, it boot normal.
I've not been able to reboot the 3rd vpar on both the frames is because I've not got the downtime for them, but suspect the same issue with them as well. Wait for close to 5 mins and then issue a vparboot -p , it comes back like a beauty !!

syslog.log, fcmsuitl, dmesg and cstm looks fine. Any ideas?

[Steven E. Protter]

Shalom,

This issue happens before any of those log files are open, because they are opened after boot.

You are going to need to look to the console for messages.

SEP

[Pupil_1]

Missed the .txt file in the org post.

[Torsten.]

Some things to check:

- servers firmware
- FC HBA firmware
- OS patches (e.g. boot loader patches)
- vPars version

All of them are involved in the boot process.

[Pupil_1]

I've 14 rx7640s, almost all of them vpar'ed, with the sept 06 patch level. All have been built with the same golden image and have the same firmware and patches.

Could this be a faulty HBA? cstm and syslog does not give any indication of that.

Thanks.

[Torsten.]

Hmm, all you have is this message:

"Load of 0/0/14/1/0/4/0.51.93.0.0.0.2 failed: Not Found"

But your fcmsutil and ioscan is showing

0/0/12/1/0/4/0
1/0/12/1/0/4/0

There is no 0/0/14 path.

Did you check your vpar config?

Regarding firmware: check the HBA firmware with this:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3365410&swItem=ux-49434-1&prodNameId=3365414&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

(display the version with "fcmsutil /dev/fcd2 vpd")

[Pupil_1]

Torsten,

Thanks for the link, I'm checking that !!

About the path, my mistake...copy paste err... The CL from mp is better.

Anything specific with the vpar that you are suggesting to check?

[Torsten.]

Looks like your current EFI driver/firmware version for the FC HBAs is
driver 1.05, firmware 4.00.070
but the current version is
1.09/4.00.90

I would consider to update this.

[Pupil_1]

The EFI driver/firmware was tried and did NOT fix the problem. The new one that was tried was 1.09/4.00.90.

HP will be bringing in the SAN packet sniffer to analyse the SAN Frames in the after noon.

Will update once I've details.

Thanks

[Pupil_1]

No finding on this problem yet, the fabric analyser is still not able to give a conclusion !

We might want to try the same HBA on another 7640 and boot from it.

Thanks

[Tim Nelson]

My non-constructive comment is this is the type of potential issue that is very difficult to troubleshoot when booting from SAN devices in lieu of local disk.

I would be interested in the resolution.
Please be sure to follow up.

Tim

[Torsten. · Accepted solution]

IMHO the OS is not related, because you get the "not found" message from EFI.
From this point is must be firmware or hardware related.

I would check the switch and disk array firmware + logs. Maybe you even find a high error rate related to the FC connection. In this case it could be a bad cable or transceiver.
I'm sure you have an alternate path to the boot disk.
Try to swap primary and alternate path in your vPar config.

[Pupil_1]

We have 2 paths to the XP. Tried both the paths, did not work. Same error.

Ran a separate FC to the director switch port, no luck either !

Thanks.

[Pupil_1]

I thought that I've already said this, but looks like i missed it..

The switch port if giving port fault errors and too many of them when the box comes up for the first time, but on the next attempt, no errors on the port !!

Thanks.

[Tim Nelson]

Something else worth reading.

http://docs.hp.com/en/7408/HPUXSANBootWP_final_v2_update.pdf

[Tim Nelson]

Here is another doc with known FW issues for booting from SAN.

http://docs.hp.com/en/J6374-90053/ch01s05.html

[likid0]

The problems i have had in the past, booting from san, with vpars is that when you get a single change on the san net infrastructure(switch port for example), the hdw path to the disk changes and it wont boot again, getting the :not found error when triying to boot,if something has changed in your san that could well be the problem, to fix it I boot from ignite,find the new hdw path and export,import vg00, change boot paths, etc.


hope it helps

[Pupil_1]

Thanks for the info Daniel.

I'm aware that nothing on the SAN has changed. There are close to 14 other vpars thru the same SAN switch that does not have this problem !! I cannot have 6 of them reinstalled without getting to the root of the problem though.

Again, the paths are visible, I'm able to see the efi structure in cxtxdxs1 when I walk through it manually from the efi shell when I've the box to come up in npar mode.

Thanks

[likid0]

Ok, if you get to see the disks with the correct path in npar/efi shell, then thats not the problem for sure.

[Pupil_1]

We've managed to figure out that the person who built these boxes had messed up lot of things. The setboot was not set correctly on 4 of the vpars and once this was fixed, all the errors has gone away. It certainly does not explain why would a machine boot after 5 mins and certainly does not explain why would the port fault on the SAN switch occur because of an incorrect Boot path setting !

Anyways, we have almost all of them sorted out. Thanks for all those who responded. N'joy you penny !

Must say, had one HBA failure on the same node yesterday night and there are still some mess that needs to be cleaned up.

[Pupil_1]

Thanks for sharing your thoughts on this. It's much appreciated !!