System not booting after power shutdown...help!!

[Tim Killinger]

Help!!

We decided to reboot our hpux11i system and as it came up several components failed to come up. We booted into single user mode and get:

/sbin/ioinitrc:
/sbin/ioinitrc[69]: /sbin/rm: not found
/dev/vg00/lvol1: LINK COUNT FILE I=4609
OWNER=root MODE=100600
/dev/vg00/lvol1: SIZE=67304 MTIME=Mar 16 18:37 2009 COUNT 2 SHOULD BE 1 (ADJUST
ED)
/dev/vg00/lvol1: LINK COUNT FILE I=4610
OWNER=root MODE=100600
/dev/vg00/lvol1: SIZE=24 MTIME=Mar 16 18:37 2009 COUNT 2 SHOULD BE 1 (ADJUSTED)
/dev/vg00/lvol1: LINK COUNT FILE I=4611
OWNER=root MODE=100600
/dev/vg00/lvol1: SIZE=67304 MTIME=Mar 16 18:16 2009 COUNT 2 SHOULD BE 1 (ADJUST
ED)
/dev/vg00/lvol1: LINK COUNT FILE I=4612
OWNER=root MODE=100600
/dev/vg00/lvol1: SIZE=24 MTIME=Mar 16 18:16 2009 COUNT 2 SHOULD BE 1 (ADJUSTED)
/dev/vg00/lvol1: FREE INODE COUNT WRONG IN SUPERBLK (FIXED)
/dev/vg00/lvol1: 10 DUP BLKS IN BIT MAPS
/dev/vg00/lvol1: BAD CYLINDER GROUPS (FIXED)
/dev/vg00/lvol1: 62 files, 0 icont, 6304 used, 55190 free (134 frags, 6882 block
s)
/sbin/krs_sysinit[24]: /sbin/rm: not found
/sbin/krs_sysinit:

INIT: Overriding default level with level 's'

INIT: SINGLE USER MODE

INIT: Running /sbin/sh

Then..... when we try to init 1...

# init 1



INIT: New run level: 1

/sbin/bcheckrc:

Checking for LVM volume groups and Activating (if any exist)

Volume group "/dev/vg00" has been successfully changed.

Activated volume group

Volume group "/dev/vgbck" has been successfully changed.

Activated volume group

Volume group "/dev/vgicgs" has been successfully changed.

Activated volume group

Volume group "/dev/vgrams" has been successfully changed.

Activated volume group

Volume group "/dev/vgapps" has been successfully changed.

Resynchronized volume group /dev/vg00

vxfs fsck: sanity check: root file system OK (mounted read/write)

Checking hfs file systems

/sbin/fsclean: /dev/vg00/lvol1 (mounted) ok

HFS file systems are OK, not running fsck

Checking vxfs file systems

/dev/vg00/lvol8 :

Resynchronized volume group /dev/vgbck

vxfs fsck: sanity check: /dev/vg00/lvol8 OK

/dev/vgbck/lvadmbck :

vxfs fsck: sanity check: /dev/vgbck/lvadmbck OK

/dev/vgicgs/lvdba :

vxfs fsck: sanity check: /dev/vgicgs/lvdba OK

/dev/vgicgs/lvicgs :

vxfs fsck: sanity check: /dev/vgicgs/lvicgs OK

/dev/vgrams/lvramsarch :

vxfs fsck: sanity check: /dev/vgrams/lvramsarch OK

/dev/vgicgs/lvoracle :

vxfs fsck: sanity check: /dev/vgicgs/lvoracle OK

/dev/vgrams/lvrams :

vxfs fsck: sanity check: /dev/vgrams/lvrams OK

/dev/vgicgs/lvpkg :

vxfs fsck: sanity check: /dev/vgicgs/lvpkg OK

/dev/vgicgs/lvdgn :

vxfs fsck: sanity check: /dev/vgicgs/lvdgn OK

/dev/vgbck/lvbck :

vxfs fsck: sanity check: /dev/vgbck/lvbck OK

/dev/vgicgs/lvmware :

vxfs fsck: sanity check: /dev/vgicgs/lvmware OK

/dev/vgapps/lvapps :

Resynchronized volume group /dev/vgicgs

vxfs fsck: sanity check: /dev/vgapps/lvapps OK

/dev/vgicgs/lvicgsarch :

vxfs fsck: sanity check: /dev/vgicgs/lvicgsarch OK

/dev/vg00/lvol3 :

vxfs fsck: sanity check: root file system OK (mounted read/write)

/dev/vg00/lvol4 :

Resynchronized volume group /dev/vgrams

vxfs fsck: sanity check: /dev/vg00/lvol4 OK

/dev/vg00/lvol5 :

Resynchronized volume group /dev/vgapps

vxfs fsck: sanity check: /dev/vg00/lvol5 OK

/dev/vg00/lvol6 :

vxfs fsck: sanity check: /dev/vg00/lvol6 OK

/dev/vgicgs/lvomnibackdb :

vxfs fsck: sanity check: /dev/vgicgs/lvomnibackdb OK

/dev/vg00/lvol7 :

vxfs fsck: sanity check: /dev/vg00/lvol7 OK



Cleaning /etc/ptmp...

(c)Copyright 1983-2000 Hewlett-Packard Co., All Rights Reserved.

(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California

(c)Copyright 1980, 1984, 1986 Novell, Inc.

(c)Copyright 1986-1992 Sun Microsystems, Inc.

(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology

(c)Copyright 1989-1993 The Open Software Foundation, Inc.

(c)Copyright 1986 Digital Equipment Corp.

(c)Copyright 1990 Motorola, Inc.

(c)Copyright 1990, 1991, 1992 Cornell University

(c)Copyright 1989-1991 The University of Maryland

(c)Copyright 1988 Carnegie Mellon University

(c)Copyright 1991-2000 Mentat Inc.

(c)Copyright 1996 Morning Star Technologies, Inc.

(c)Copyright 1996 Progressive Systems, Inc.

(c)Copyright 1991-2000 Isogon Corporation, All Rights Reserved.





RESTRICTED RIGHTS LEGEND

Use, duplication, or disclosure by the U.S. Government is subject to

restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in

Technical Data and Computer Software clause in DFARS 252.227-7013.



Hewlett-Packard Company

3000 Hanover Street

Palo Alto, CA 94304 U.S.A.



Rights for non-DOD U.S. Government Departments and Agencies are as set

forth in FAR 52.227-19(c)(1,2).

/sbin/sh: /sbin/rm: not found

ln: /dev/systty and /dev/syscon are identical



/sbin/auto_parms: DHCP access is disabled (see /etc/auto_parms.log)



/sbin/rc1.d/S093vxvm-reconfig[48]: /etc/vx/lib/vxcommon: not found

/sbin/rc1.d/S110kmbuild[276]: rm: not found



HP-UX Start-up in progress

__________________________



Configure system crash dumps .............. OK

VxVM device node check .................... OK

VxVM general startup ...................... OK

.......................................... FAIL *

Mount file systems ........................ FAIL *

Update kernel and loadable modules ........ N/A

Initialize loadable modules ............... N/A

Setting hostname .......................... FAIL *

Start Kernel Logging facility ............. N/A

Set privilege group ....................... N/A

Display date .............................. N/A

Copy processor logs to /var/tombstones .... N/A

Save system crash dump if needed .......... N/A

Start evp ................................. OK

Enable auxiliary swap space ............... FAIL *

Start syncer daemon ....................... N/A



* - An error has occurred !

* - Refer to the file /etc/rc.log for more information.







The system is ready.



INIT: Command is respawning too rapidly.

Will try again in 5 minutes.

Check for possible errors.

id:SMLK "/opt/SMI-S/cimom/hpSMISCIMOM >/dev/null 2>&1"



INIT: Command is respawning too rapidly.

Will try again in 5 minutes.

Check for possible errors.

id:cons "/usr/sbin/getty console console # system console"



INIT: Command is respawning too rapidly.

Will try again in 5 minutes.

Check for possible errors.

id:ems4 "/etc/opt/resmon/lbin/p_client"

[Johnson Punniyalingam]

Hi Tim,

Below error are from /etc/inittab,

you need to fix them, if you find the below error are not related to application use to can comment out them from /etc/inittab..

I advice you check them

id:SMLK "/opt/SMI-S/cimom/hpSMISCIMOM >/dev/null 2>&1"

id:cons "/usr/sbin/getty console console # system console"


id:ems4 "/etc/opt/resmon/lbin/p_client"

Thanks,
JOhnson

[Dennis Handly]

>System not booting after power shutdown

Was this a graceful shutdown or abrupt?
It seems you have fsck errors and you have lost important files in /sbin/. You are going to have to get this from a backup or ignite.

[Tim Killinger]

HP Software support solved it... essentialy copied the rm executabale from on location to another.