HPVM rebooting continuously

[majhar_mujawar]

Dear Team, 

I have assigned a LUN to hpvm then restarted it after that VM is not starting up it is rebooting again and again after DUMP, with below error messages,

Starting: HPUX Primary

(C) Copyright 1999-2011 Hewlett-Packard Development Company, L.P.
All rights reserved

HP-UX Boot Loader for IPF -- Revision 2.044

Press Any Key to interrupt Autoboot
\EFI\HPUX\AUTO ==> boot vmunix
Seconds left till autoboot - 10 9
- - - - - - - - - - - - Live Console - - - - - - - - - - - -
8 7 6 5 4 3 2 1 0
AUTOBOOTING...> System Memory = 10234 MB
loading section 0
........................................................................ (complete)
loading section 1
................. (complete)
loading symbol table
loading System Directory (boot.sys) to MFS
.....
loading MFSFILES directory (bootfs) to MFS
..............................
Launching /stand/vmunix
SIZE: Text:36605K + Data:8426K + BSS:26981K = Total:72013K

Console is on Serial COM1
Booting kernel...

Disabling 8259s
Loaded ACPI revision 2.0 tables.
Using /stand/ext_ioconfig

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 9734 MB 9734 MB 9734 MB
Kernel : 9734 MB 9734 MB 9734 MB
User : 8833 MB 7830 MB 7861 MB
-------------------------------------------------------------------------

ktracer is off until requested.
CPU00 failed to synchronize its interval timer with CPU01 within 45 Ticks
Installing Socket Protocol families AF_INET and AF_INET6
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 10.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 11.
NOTICE: cachefsc_install: cachefs File system was registered at index 13.
Boot device's HP-UX HW path is: 0/0/0/0.0x0.0x0
igssn0: INITIALIZING HP IGSSN PCI AVIO LAN Adapter at hardware path 0/0/1/0
igssn1: INITIALIZING HP IGSSN PCI AVIO LAN Adapter at hardware path 0/0/2/0

System Console is on the Built-In Serial Interface
afinet_prelink: module installed
AF_INET socket/streams output daemon running, pid 47
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 61472768
The device file /dev/rroot does not exist or is not correct.
Automatic checks of the root file system not performed.
vxfs: WARNING: msgcnt 1 mesg 096: V-2-96: vx_setfsflags - /dev/root file system fullfsck flag set - vx_ierror
vxfs: WARNING: msgcnt 2 mesg 017: V-2-17: vx_dirlook - / file system inode 241 marked bad incore
?vxfs msgcnt 2 offset 0x00000000 0 0 0 0
?vxfs msgcnt 2 offset 0x00000010 0 0 0 0
?vxfs msgcnt 2 offset 0x00000020 0 0 0 0
?vxfs msgcnt 2 offset 0x00000030 0 22b1f92c 0 0
?vxfs msgcnt 2 offset 0x00000040 0 0 0 0
?vxfs msgcnt 2 offset 0x00000050 0 0 0 0
?vxfs msgcnt 2 offset 0x00000060 0 e 0 0
?vxfs msgcnt 2 offset 0x00000070 0 0 0 0
?vxfs msgcnt 2 offset 0x00000080 0 0 0 0
?vxfs msgcnt 2 offset 0x00000090 0 0 0 0
?vxfs msgcnt 2 offset 0x000000a0 0 0 0 0
?vxfs msgcnt 2 offset 0x000000b0 0 0 0 0
?vxfs msgcnt 2 offset 0x000000c0 0 0 0 0
?vxfs msgcnt 2 offset 0x000000d0 0 0
vxfs: WARNING: msgcnt 3 mesg 096: V-2-96: vx_setfsflags - /dev/root file system fullfsck flag set - vx_ierror

Stored message buffer up to system crash:

MFS is defined: base= 0xe000000102297000 size= 45804 KB
Loaded ACPI revision 2.0 tables.


MCA recovery subsystem disabled, not supported on this platform.
montecito_proc_features: PROC_GET_FEATURES returned 0xfffffffffffffff8
Using /stand/ext_ioconfig

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 9734 MB 9734 MB 9734 MB
Kernel : 9734 MB 9734 MB 9734 MB
User : 8833 MB 7830 MB 7861 MB
-------------------------------------------------------------------------

ktracer is off until requested.
montecito_proc_features: PROC_GET_FEATURES returned 0xfffffffffffffff8
CPU00 failed to synchronize its interval timer with CPU01 within 45 Ticks
Installing Socket Protocol families AF_INET and AF_INET6
64000/0xfa00 esvroot
Kernel EVM initialized
64000/0x0 mass_storage
64000/0x0/0x0 usb_ms_scsi
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 10.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 11.
NOTICE: cachefsc_install: cachefs File system was registered at index 13.
btlan_load() Loaded Successfully
0 cec_gen
0/0 gh2p
gvsd: Initialized HPVM AVIO Stor Adapter at hardware path 0/0/0/0
0/0/0/0 gvsd
0/0/1/0 igssn
0/0/2/0 igssn
0/0/3/0 legacyio
0/0/3/0/1 asio0
0/1 gh2p
0/2 gh2p
0/3 gh2p
0/4 gh2p
0/5 gh2p
0/6 gh2p
0/7 gh2p
0/0/0/0.0x0 estp
120 processor
121 processor
250 pdh
250/0 ipmi
250/1 acpi_node
0/0/0/0.0x1 estp
0/0/0/0.0x2 estp
0/0/0/0.0x1.0x0 eslpt
0/0/0/0.0x0.0x0 eslpt
0/0/0/0.1 tgt
0/0/0/0.1.0 sdisk
0/0/0/0.0 tgt
0/0/0/0.0.0 sdisk
0/0/0/0.0x2.0x0 eslpt
0/0/0/0.2 tgt
0/0/0/0.2.0 stape
255/1 mass_storage
255/1/0 usb_ms_scsi
64000/0xfa00/0x0 esdisk
64000/0xfa00/0x9 esdisk
64000/0xfa00/0xa estape
Boot device's HP-UX HW path is: 0/0/0/0.0x0.0x0
igssn0: INITIALIZING HP IGSSN PCI AVIO LAN Adapter at hardware path 0/0/1/0
igssn1: INITIALIZING HP IGSSN PCI AVIO LAN Adapter at hardware path 0/0/2/0

System Console is on the Built-In Serial Interface
afinet_prelink: module installed
AF_INET socket/streams output daemon running, pid 47
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 61472768
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 1, minor is 0x0; start = 2349920, size = 30736384
vxfs: WARNING: msgcnt 1 mesg 096: V-2-96: vx_setfsflags - /dev/root file system fullfsck flag set - vx_ierror
vxfs: WARNING: msgcnt 2 mesg 017: V-2-17: vx_dirlook - / file system inode 241 marked bad incore
?vxfs msgcnt 2 offset 0x00000000 0 0 0 0
?vxfs msgcnt 2 offset 0x00000010 0 0 0 0
?vxfs msgcnt 2 offset 0x00000020 0 0 0 0
?vxfs msgcnt 2 offset 0x00000030 0 22b1f92c 0 0
?vxfs msgcnt 2 offset 0x00000040 0 0 0 0
?vxfs msgcnt 2 offset 0x00000050 0 0 0 0
?vxfs msgcnt 2 offset 0x00000060 0 e 0 0
?vxfs msgcnt 2 offset 0x00000070 0 0 0 0
?vxfs msgcnt 2 offset 0x00000080 0 0 0 0
?vxfs msgcnt 2 offset 0x00000090 0 0 0 0
?vxfs msgcnt 2 offset 0x000000a0 0 0 0 0
?vxfs msgcnt 2 offset 0x000000b0 0 0 0 0
?vxfs msgcnt 2 offset 0x000000c0 0 0 0 0
?vxfs msgcnt 2 offset 0x000000d0 0 0
vxfs: WARNING: msgcnt 3 mesg 096: V-2-96: vx_setfsflags - /dev/root file system fullfsck flag set - vx_ierror
VFS_MOUNTROOT error information:
0:(hfs) 22
1:(nfs) 223
9:(vxfs) 28
10:(nfs3) 223
11:(nfs4) 223
12:(autofs) 251
15:(cifs) 223
17:(dnlc_neg_fs) 251

Message buffer contents after system crash:

panic: VFS_MOUNTROOT: RDWR: all VFS_MOUNTROOTs failed: NEED DRIVERS ?????
Stack Trace:
IP Function Name
0xe00000000209d120 vfs_mountroot+0x700
0xe000000001722190 im_preinitrc+0x450
0xe00000000208dee0 DoCalllist+0x390
End of Stack Trace

linkstamp: Mon Dec 19 20:01:39 IST 2011
_release_version: @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Calling function e000000001828240 for Shutdown State 1 type 0x2
i 0 pfn 0x40 pages 0xb0
i 1 pfn 0x54c pages 0x7fab4
i 2 pfn 0x100000 pages 0x1ffe12
i 3 pfn 0x2ffe7e pages 0x13e
*** Not enough CPUS for a compressed dump ***

*** A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

*** The dump will be a SELECTIVE dump with
compression OFF and concurrency ON: 1409 of 10235 megabytes.
*** To change this dump type, press any key within 10 seconds.
*** Proceeding with selective dump, with compression off and concurrency on.

Primary Dump Header Location :
Device details:
Major number: 0x1f Minor number: 0x0
Offset: 2349920.
*** The dump may be aborted at any time by pressing ESC.
*** Dumping: 0% complete (0 of 1409 MB) *** Dumping: 0% complete (1 of 1409 MB) *** Dumping: 100% complete (1409 of 1409 MB) *** Dumping: 100% complete (1409 of 1409 MB) *** Dumping: 100% complete (1409 of 1409 MB) *** Dumping: 100% complete (1409 of 1409 MB) *** Dumping: 100% complete (1409 of 1409 MB)
time: 11 seconds, Number of Dump units: 1

**** Dumping Guest Image ****

**** Done with dump (19956Kbytes) ****

I dont have ignite backup, is it possible to recover the VM.

Regards,

Majhar

[Bill Hassell]

Based on these messages:

xfs: WARNING: msgcnt 1 mesg 096: V-2-96: vx_setfsflags - /dev/root file system fullfsck flag set - vx_ierror
vxfs: WARNING: msgcnt 2 mesg 017: V-2-17: vx_dirlook - / file system inode 241 marked bad incore

It would appear that the disk has failed or has been corrupted. Without an Ignite backup, you'll have to reinstall the VM.

[majhar_mujawar]

Dear Sir,

Thanks for your suggestion,

i installed new OS on VM , is it possible to recover old VG,LV from disk, i dont have any backup.

Regards,

Majhar

[pyzze]

Hi,

According to the logs you have a corrupted root filesystem:

vxfs: WARNING: msgcnt 2 mesg 017: V-2-17: vx_dirlook - / file system inode 241 marked bad incore

This could be due to a hardware issue, but it also can be caused for example by presenting the related volume to more than one host by mistake.

If it is not a hardware issue, you may not have an alternative to recovery from backup.

Regards,

JA

[pyzze]

You may use the command:

vgscan -a -p -v

From the output, take note of the disks device files that belong to each VG.

Use vgimport to import the VGs, specifying the correct device files for each VG.

Regards,

JA

[majhar_mujawar]

Hi,
2 Hpvm having avio configured but I presented LUN using scsi ..it asked to take reboot...after rebooting 1 vm crashed...how should i avoid crash of another vm .

Thanks

[majhar_mujawar]

Dear,

Thanks for your reoly, i have a qyery regarding LUN presentation,

Previously attached LUN to vm were through AVIO_STOR but i presented using SCSI, is this the problem which caused VM to crash.

I have another VM to which i presented LUN using SCSI and it also asking for reboot but i havent rebboted it yet.

how should i avoid OS crash after reboot, by removing LUN after hpvmstop will help me to avoid this.

Regards,

Majhar