System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Help please! System crash and won't boot

 
Tim Medford
Valued Contributor

Help please! System crash and won't boot

0/5/1/0.0x50001fe1501eaf29.0x0 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x0 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4001000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4002000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4003000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4004000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4005000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4006000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4001000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4007000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4002000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4003000000000000 eslpt
0/5/1/0.0x50001fe1501eaf29.0x4008000000000000 eslpt
0/5/1/0.4 fcd_fcp
0/5/1/0.4.28.255.0 fcd_vbus
0/5/1/0.4.28.255.0.0 tgt
0/5/1/0.4.28.255.0.0.0 sctl
0/5/1/0.4.28.0.0 fcd_vbus
0/5/1/0.4.28.0.0.0 tgt
0/5/1/0.4.28.0.0.0.1 sdisk
0/5/1/0.4.28.0.0.0.2 sdisk
0/5/1/0.4.28.0.0.0.3 sdisk
0/5/1/0.4.28.0.0.0.4 sdisk
0/5/1/0.4.28.0.0.0.5 sdisk
0/5/1/0.4.28.0.0.0.6 sdisk
0/5/1/0.4.28.0.0.0.7 sdisk
0/5/1/0.4.28.0.0.1 tgt
0/5/1/0.4.28.0.0.1.0 sdisk
0/5/1/0.0x50001fe1501eaf2b.0x4004000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4005000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4006000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4007000000000000 eslpt
0/5/1/0.0x50001fe1501eaf2b.0x4008000000000000 eslpt
0/5/1/0.4.26.255.0 fcd_vbus
0/5/1/0.4.26.255.0.0 tgt
0/5/1/0.4.26.255.0.0.0 sctl
0/5/1/0.4.26.0.0 fcd_vbus
0/5/1/0.4.26.0.0.0 tgt
0/5/1/0.4.26.0.0.0.1 sdisk
0/5/1/0.4.26.0.0.0.2 sdisk
0/5/1/0.4.26.0.0.0.3 sdisk
0/5/1/0.4.26.0.0.0.4 sdisk
0/5/1/0.4.26.0.0.0.5 sdisk
0/5/1/0.4.26.0.0.0.6 sdisk
0/5/1/0.4.26.0.0.0.7 sdisk
0/5/1/0.4.26.0.0.1 tgt
0/5/1/0.4.26.0.0.1.0 sdisk
255/1 mass_storage
255/1/0 usb_ms_scsi
64000/0xfa00/0x0 esctl
64000/0xfa00/0x1 esctl
64000/0xfa00/0x2 esdisk
64000/0xfa00/0x3 estape
64000/0xfa00/0x4 estape
64000/0xfa00/0x5 eschgr
64000/0xfa00/0x7 estape
64000/0xfa00/0x8 estape
64000/0xfa00/0x20 esdisk
64000/0xfa00/0x23 esdisk
64000/0xfa00/0x24 esdisk
64000/0xfa00/0x25 esdisk
64000/0xfa00/0x26 esdisk
64000/0xfa00/0x27 esdisk
64000/0xfa00/0x28 esdisk
64000/0xfa00/0x29 esdisk
Boot device's HP-UX HW path is: 0/6/0/0/0/0/1/0/0/0.0x0.0x4000000000000000

System Console is on the Built-In Serial Interface
iether0: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at har0
iether1: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at har1
AF_INET socket/streams output daemon running, pid 51
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 1, minor is 0x2; start = 2349952, siz4
VFS_MOUNTROOT error information:
0:(hfs) 22
1:(nfs) 223
9:(vxfs) 28
10:(nfs3) 223
11:(nfs4) 223
12:(autofs) 251
15:(cifs) 223
17:(dnlc_neg_fs) 251

Message buffer contents after system crash:

panic: VFS_MOUNTROOT: RDWR: all VFS_MOUNTROOTs failed: NEED DRIVERS ?????
Stack Trace:
IP Function Name
0xe000000002123e40 vfs_mountroot+0x700
0xe000000001785b90 im_preinitrc+0x450
0xe000000002111aa0 DoCalllist+0x390
End of Stack Trace

linkstamp: Thu Mar 28 11:05:14 PDT 2013
_release_version: @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Calling function e00000000188bda0 for Shutdown State 1 type 0x2
Calling function e0000107ed1a0140 for Shutdown State 9 type 0x2
Calling function e0000107ed1a0140 for Shutdown State 9 type 0x2
i 0 pfn 0x1 pages 0x3def
i 1 pfn 0x3e24 pages 0x2e
i 2 pfn 0x3fc0 pages 0x1d
i 3 pfn 0x10000 pages 0xaffff
i 4 pfn 0x1004000 pages 0xbeff
i 5 pfn 0x100ff00 pages 0xb5
i 6 pfn 0x100ffc0 pages 0x20
i 7 pfn 0x100ffe8 pages 0x16
*** Not enough CPUS for a compressed dump ***

*** A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

*** The dump will be a SELECTIVE dump with
compression OFF and concurrency ON: 626 of 49123 megabytes.
*** To change this dump type, press any key within 10 seconds.
*** Proceeding with selective dump, with compression off and concurrency on.

13 REPLIES
Patrick Wallek
Honored Contributor

Re: Help please! System crash and won't boot

What type of disk are you trying to boot from?  Is it fibre attached?  What model server?  If Itanium, go to the EFI shell, then do a 'reconnect -r' and 'map -r' to generate a new device map.

 

Make sure the device you are trying to boot from exists.  If you have multiple paths to a device, make sure all paths are still active, and try booting from a different path.

 

I ran into a situation recently where an Itanium server would start to boot, but get to the point of activating its swap device and then panic.  We finally tried booting from a different path to that device and it worked.  The problem was that the primary boot path wasn't quite set up correctly at the fibre level.  It could see enough to start booting the system, but not enough to activate the VG.

 

You might also try booting into single-user mode or LVM maintenance mode and see if you have any better luck.

 

If all else fails, you may have to restore your system from an Ignite backup.

Tim Medford
Valued Contributor

Re: Help please! System crash and won't boot

Hi Patrick - I should have included more info in my message, sorry...in a bit of my own panic.

 

It is an rx3600 IA64 server, booting from an internal P400 raid drive.  And wouldn't you know it I have an igniteUX tape for every server in the shop, except this one.

 

I will try your suggestions and let you know.  Thanks.

Robert_Jewell
Honored Contributor

Re: Help please! System crash and won't boot

This usually indicates a problem with mounting the root filesystem.  Was anything done prior to this reboot that could cause an issue with accessing the disk or root filesystem? 

 

Couple of things to try:

 

 - Easiest would be to try to boot into LVM maintenance mode (from ISL>/HPUX> prompt:  hpux -lm

       If that works, then see about checking your VG00 config and manually running fsck on /dev/vg00/lvol3

 

 - More likely, you will need to boot into a recovery shell from installation media.  Look to recover your bootlif

 

 

-Bob

----------------
Was this helpful? Like this post by giving me a thumbs up below!
Torsten.
Acclaimed Contributor

Re: Help please! System crash and won't boot

I would first try to boot into lvm maintenance mode and check the settings:

# lnlvboot -v

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Patrick Wallek
Honored Contributor

Re: Help please! System crash and won't boot

When the system is coming up, can you go into the RAID card configuration and verify that your RAID volume is in a good state?

 

 

Torsten.
Acclaimed Contributor

Re: Help please! System crash and won't boot

Good idea.

During boot to EFI when the system is showing

Press <F8> to run the Option ROM Configuration for Arrays Utility

then press ESC 8.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Tim Medford
Valued Contributor

Re: Help please! System crash and won't boot

We were attempting to setup APA and had moved the network cables around.  The system essentially hung up on me and I couldn't get it to respond.  I ended up issuing a RESET command from the iLO interface.  When it came back up from there, it started to panic and create crash dumps.  After it does the dump, it just says rebooting, and then that cycle continues.  

 

I have tried getting into maintenance mode in LVM, but I'm not exactly sure I'm doing it correctly.  If I interrupt the boot process and go to an EFI shell, is that how to get to the ISL prompt?

 

Patrick Wallek
Honored Contributor

Re: Help please! System crash and won't boot

See this thread on how to boot into single-user or LVM maint mode.

 

http://h30499.www3.hp.com/t5/General/How-to-boot-HP-UX-on-Itanium-to-lvm-maintenance-mode/td-p/3697297

 


EDIT -- I actually added the thread URL...

Tim Medford
Valued Contributor

Re: Help please! System crash and won't boot

Nothing seems to be working.  I have attempted to boot into LVM maintenance mode and it won't even get that far.  Right now I am booting off installation media to attempt to get into a recovery shell.

 

Torsten.
Acclaimed Contributor

Re: Help please! System crash and won't boot

Could you please record the output and post?


However, you probably damaged the filesystem.

In this case a restore is the only way out.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Tim Medford
Valued Contributor

Re: Help please! System crash and won't boot

Thanks everyone for your help yesterday.  After 4.5 hours on the phone with HP level1 support we were unable to fix the problem.  We tried all of the above suggestions, and even a recovery shell from the installation media.  There was a bad superblock on the boot sector and fsck was not able to repair it.  I had to do a cold install from media and then recover from data protector backups.  That was 12 hours of my life I will never get back.

chris huys_4
Honored Contributor

Re: Help please! System crash and won't boot

A ignite network server, were a ignite network image of this server would have been stored on,  would have saved you a lot of time. 

 

And its to late offcourse, but my actionplan would have been, to present a new 50 gbyte lun of what looks like a eva diskarray, to be presented to this host, cold install on this lun a core-os from cd/dvd/ignite server , and try to import the vg00 with the problem internal p400 lun this way as a datavolumegroup on the just installed core-os vg00, and then try to repair it.

 

At least then you would see what really was the problem.

Tim Medford
Valued Contributor

Re: Help please! System crash and won't boot

Thanks for the advice again.  It was really the perfect-storm scenario here, I got caught with my pants down.  I normally have ignite backups and even alternate boot disks for all of our servers, even the development ones.  

 

We are in the middle of a big migration from hpux 11.23 and 11.31.  This server had just been built a couple weeks ago and was temporarily being used as our data protector cell manager during the migration.

 

It was a good reminder of why I always have at least 2 alternate ways to boot my systems in addition to the primary boot path.