HPUX VxVM boot from SAN hangs

[Jason W. Neiss]

Hi, all;

I'm not certain if this is a VxVM issue, an XP1024 SAN issue, or an HPUX issue, but I figured I'd start here and see where it takes me.

I recently took over administration of an MC/Serviceguard cluster that manages an XP1024 SAN. The nodes are configured to boot from the XP1024. However, during the boot process, one of the nodes hangs:

alloc_pdc_pages: Relocating PDC from 0xffff800000 to 0x7fa00000.
gate64: sysvec_vaddr = 0xc0002000 for 2 pages
NOTICE: autofs_link(): File system was registered at index 3.
NOTICE: cachefs_link(): File system was registered at index 5.
NOTICE: nfs3_link(): File system was registered at index 6.
td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/8/0/0
td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/9/0/0

System Console is on the Built-In Serial Interface
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 2, minor is 0x1; start = 0, size = 8388608
Starting the STREAMS daemons-phase 1
Starting vxconfigd in boot mode (pre_init_rc).
NOTICE: vxvm:vxdmp: added disk array DISKS, datype = Disk

NOTICE: vxvm:vxdmp: added disk array 43654, datype = XP1024

{and off it goes to never-never land}

Everything appears, to my eye, to be normal. There is another node that does not (to my knowledge) has this issue. I can get into the GSP, issue a reset, and return to this spot in the boot process. If I try to boot to single-user mode, it hangs at the same spot. I can boot to MMB mode (ISL: hpux -vm), but since I am not very familiar with VxVM, it doesn't do me much good.

The fiber connections from the server to the Brocade and the Brocade to the XP have been tested good, according to my predecessor. The person who originally set this up left the company in a major huff not long ago, so he's no longer a resource. The previous admin noted that when he had this problem, he was able to bypass the hang within ten seconds or so by hitting ESC repeatedly (or maybe it was BREAK, he's not sure). Banging on ESC or banging on BREAK or banging on CTRL-C does nothing for me. He also suggested a firmware update for the Brocade, but that doesn't seem to make a lot of sense to me--since the machine obviously boots on occasion, and the other node does not--again, as far as I know--have this problem).

I'm stumped. I'm not necessarily looking for a fix (though that would be spectacular), I'm mostly looking for either ideas of a direction to take or ways to bypass the hang to get the system booted again. I'm running a production back-end system for a web-based service on a single node, and that makes me very, very nervous.

Thanks;
Jason

[Chauhan Amit]

Hi Jason,

All the messages logged are normal messages , there is no error message at all. From the Sequence seen above , what I can observe is that it doesn't seems to be HP-UX or VxVM issue .
Check LUN for Data Integrity. It can be a issue with FC Card.

-Amit

[Chauhan Amit]

I would suggest you to log a call with HP Response centre.

[Zinky]

Jason.. did this node ever successfully booted off the XP1024 LUN(s) in the past? Have you checked tha LUN(s) presented as boot disks to said node if :

(a) It is presented properly
(b) It is not the same LUN presented to another node?

Also, hav eyou tried booting to LVM maintenance mode?

[Kent Ostby]

Jason -- Assuming its NOT a vxvm issue, I would mount up this disk as a data disk and check the /dev/console, /dev/syscon, /dev/systty and make sure they match the disk that is bootable.

Also, if this is an IA box, then I'd make sure the console settings were correct.

See:

http://docs.hp.com/en/5991-0792/ch04s01.html#cihgcffj

for details

[Kent Ostby]

Another thing to try is to boot a recovery shell and see if that will boot all the way up.

From there you could also access /dev and the syscon, systty, and console files.

But I'd check the EFI console settings first.

[Jason W. Neiss]

Kent: The machine is a PARISC L-Class, and the console settings are correct for proper booting. It's done so in the past; it's simply hanging during the boot process.

Nelson: Yes, this node has booted from the XP LUNs in the past. The rootvol is VxVM, so LVM maintenance mode won't do me much good (unless I can boot to LVM maintenance mode, start VxVM,and switch to init 3 from there...hmmm, that's a {slim} possibility). As I said, it'll do VxVM maintenance mode boot, but I don't know what I would do from there.

Amit: Yes, I think a support call is in order.

Thanks;
Jason

[Jason W. Neiss]

Update:

While I haven't resolved the problem, I found a workaround. By booting into VxVM MMB mode, I was able to enable and mount the volumes, and switch to init 3, thus bringing the system back online (not a fix, but a perfectly acceptable workaround). My process was as follows:

ISL> hpux -vm
# vxconfigd
# vxvol -g rootdg start varvol
# vxvol -g rootdg start usrvol
# vxvol -g rootdg start optvol
# vxvol -g rootdg start homevol
(needed repair; I used vxrecover -s -g rootdg homevol)
# vxvol -g datadg start vol1
# vxvol -g datadg start vol2
# vxvol -g datadg start vol3
# vxvol -g datadg start vol3
# fsck {various and sundry since they weren't clean due to the crash}
# mount -a
# init 3

I'm back in business!
Thanks;
Jason