11.31.System Boot Hang

[Paul F Rose]

The system is a rx6600, newly installed with HPUX 11.31 March 2008 EOE DVD. The system is attached to an EMC CX700 array with Powerpath installed.
The system was working fine with 6 LUNs on the array. I added 4 new LUNs to the array. When I ran 'ioscan -fnC disk', the process hung, then I lost my connection to the host.
So I connected to the console, and discovered that the system was rebooting - except it is now hung during the boot process.

Here is the console listing:

Booting kernel...

Loaded ACPI revision 2.0 tables.
Using /stand/ext_ioconfig

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 31137 MB 31137 MB 31137 MB
Kernel : 31137 MB 31137 MB 31137 MB
User : 29873 MB 26483 MB 26587 MB
-------------------------------------------------------------------------

Starting ktracer 0 1
Installing Socket Protocol families AF_INET and AF_INET6
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 11.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 12.
NOTICE: cachefsc_install: cachefs File system was registered at index 14.
NOTICE: USB device attached. Identification String:
Devices/Device/USB/Standard/hp/Multibay/0_a1
<2.1.3.10.1008.294.161>
NOTICE: USB device attached. Identification String:
Devices/MassStorage-SCSI/USB/BulkOnly/hp/Multibay/0_a1
<2.310.3.150.1008.294.161>
NOTICE: USB device attached. Identification String:
Devices/ScsiControllerAdaptor/USB/BulkOnly/hp/Multibay
<2.1000.3.150.1008.294>
Boot device's HP-UX HW path is: 0.4.1.0.53846d1e4c739bd.0

System Console is on the Built-In Serial Interface
iether0: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at
hardware path 0/2/1/0/6/0
iether3: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path
0/4/2/0
iether5: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at
hardware path 0/6/1/0/6/0
iether1: INITIALIZING HP A7012-60601 PCI/PCI-X 1000Base-T Dual-port Adapter at hardware pa
th 0/3/1/0
iether2: INITIALIZING HP A7012-60601 PCI/PCI-X 1000Base-T Dual-port Adapter at hardware pa
th 0/3/1/1
iether4: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path
0/4/2/1
AF_INET socket/streams output daemon running, pid 45
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216
Checking root file system.
log replay in progress
replay complete - marking super-block as CLEAN
Root check done.
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 33522780 Kbytes, lockable: 25174364 Kbytes, available: 28648620 Kbytes

/sbin/ioinitrc:
/dev/vg00/lvol1:log replay in progress
/dev/vg00/lvol1:replay complete - marking super-block as CLEAN
/sbin/krs_sysinit:
* The module 'rng' has been loaded.

insf: Installing special files for sdisk instance 46 address 0/2/1/0/4/0.10.18.0.0.0.6
insf: Installing special files for sdisk instance 47 address 0/2/1/0/4/0.10.18.0.0.0.7
insf: Installing special files for sdisk instance 48 address 0/2/1/0/4/0.10.18.0.0.1.0
insf: Installing special files for sdisk instance 49 address 0/2/1/0/4/0.10.18.0.0.1.1
insf: Installing special files for sdisk instance 50 address 0/6/1/0/4/0.9.22.0.0.0.6
insf: Installing special files for sdisk instance 51 address 0/6/1/0/4/0.9.22.0.0.0.7
insf: Installing special files for sdisk instance 52 address 0/6/1/0/4/0.9.22.0.0.1.0
insf: Installing special files for sdisk instance 53 address 0/6/1/0/4/0.9.22.0.0.1.1
insf: Installing special files for ipmi instance 0 address 250/0


I have another identical system which shares the same array LUNs (both are in the same storage group) which boots and works fine. The only difference I am aware of is that the system which is working was factory installed whereas I have just finished installing the problem system from the DVD. I had verified that both systems had the exact same software bundles installed, but I hadn't checked down to the individual patch levels.

[SKR_1]

Is it hanging at the same level always?

Remove the fibre cable from the HBA and then try to boot the server.

Thanks
SKR

[Paul F Rose]

I was able to boot the system into LVM maintenance, get all the VGs activated, and ran insf successfully. When I rebooted, however, the boot process hung at the exact same place.

So I booted into LVM maintenance mode, and tried to bring the system up to run level 2 but the init process hung up part way through, locking the console. Fortunately, the init 2 had enabled the network so I was able to get in via telnet, manually start the sw agent and then remove Powerpath (which required a reboot).

The system then booted cleanly.

So I guess I can close this thread.

[Paul F Rose]

See above.

[Tim Nelson]

Paul,

I know you have already closed this thread but I think I have experienced this in the past.

There is a device file that powerpath maintains during boot. File is in /etc/ and called power(somthing) or pp(somthing). If this file becomes corrupt, which it can, then when powerpath starts during boot it hangs.

By deleting this file and rebooting, powerpath will fix itself and or can be fixed with a simple powermt config.

Sorry that I cannot remember the name of the file.

[Paul F Rose]

Tim,

Thanks for replying. I believe the file you were talking about is /etc/powermt.custom. It turns out I had exactly the same kind of panic on the other server attached to the same storage group, which also resulted in a hung boot process. Deleting /etc/powermt.custom and rebooting didn't solve the problem, but deleting that file and running 'ioinit -i' did.

[Paul F Rose]

I re-opened the thread so I could give Tim some points.