fsck errors on boot which resolve as boot up progresses but system has issues

[paul_xedos]

Sorry about long winded title - couldn't think of a better way to describe. 

Probably easiset thing is if I paste the console log in the hope that someone coud tell me how to resolve the initial errors which I think are the cause of issues ( too messy to go into but relate to kernel rebuild problems when changing tunables)

To give some background though - system has had re-build with various filesystems overlaid on top from backup ( not /, or /usr or /stand but everything else)

Console is on Serial COM1
Booting kernel...

Disabling 8259s
Loaded ACPI revision 2.0 tables.

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 15579 MB 15579 MB 15579 MB
Kernel : 15579 MB 15579 MB 15579 MB
User : 14685 MB 13019 MB 13070 MB
-------------------------------------------------------------------------

Starting ktracer 0 1
Installing Socket Protocol families AF_INET and AF_INET6
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 10.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 11.
NOTICE: cachefsc_install: cachefs File system was registered at index 13.
Boot device's HP-UX HW path is: 0.0.0.0.1.0
iether0: INITIALIZING HP PCI/PCI-X 1000Base-T at hardware path 0/0/1/0

System Console is on the Built-In Serial Interface
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
AF_INET socket/streams output daemon running, pid 33
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 4194304
Checking root file system.
fsck: invalid argument to "-F FStype" option
fsck: vxfs is not a valid file system type
usage: fsck [-F FStype] [-V] [-m] [special...]
or fsck [-F FStype] [-V] [-m] [-o specific_options] [special...]
Root check done.
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 16771616 Kbytes, lockable: 12229452 Kbytes, available: 13937684 Kbytes

INIT: /etc/inittab: Bad action field for id "db2i" -- ignored entry
/sbin/ioinitrc:
fsck: invalid argument to "-F FStype" option
fsck: vxfs is not a valid file system type
usage: fsck [-F FStype] [-V] [-m] [special...]
or fsck [-F FStype] [-V] [-m] [-o specific_options] [special...]
/sbin/krs_sysinit:
Performing mpt postload operation

Performing fcd postload operation...
Load status is BOOT
FCD driver is loaded during boot time
* The module 'rng' has been loaded.

insf: Installing special files for ipmi instance 0 address 250/0

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
/sbin/lvmrc[101]: test: argument expected
Error: Failed to detect the root VG scan on the system.
Volume group "/dev/vg00" is already active on this system.
Activated volume group
Volume group "/dev/vgdata" has been successfully changed.
Activated volume group
Volume group "/dev/vg01" has been successfully changed.
Resynchronized volume group /dev/vg00
Resynchronized volume group /dev/vgdata
Resynchronized volume group /dev/vg01
Checking vxfs file systems
fsck: invalid argument to "-F FStype" option
fsck: vxfs is not a valid file system type
usage: fsck [-F FStype] [-V] [-m] [special...]
or fsck [-F FStype] [-V] [-m] [-o specific_options] [special...]
fsck: invalid argument to "-F FStype" option
fsck: vxfs is not a valid file system type
usage: fsck [-F FStype] [-V] [-m] [special...]
or fsck [-F FStype] [-V] [-m] [-o specific_options] [special...]
fsck: invalid argument to "-F FStype" option

... etc

couldn't fix vxfs filesystems, manual intervention needed

(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993 The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying. Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.

/sbin/auto_parms: DHCP access is disabled (see /etc/auto_parms.log)

/sbin/auto_parms[24]: who: not found
/sbin/auto_parms[34]: ps: not found
/sbin/auto_parms[34]: grep: not found
/sbin/rc1.d/S093vxvm-reconfig[50]: /etc/vx/lib/vxcommon: not found
interpreter "/bin/sh" not found

...etc..

file link resolves to "/usr/bin/sh"
/sbin/rc[14]: /sbin/rc2.d/S999isisd: not found
/sbin/rc3.d/S200tps.rc[15]: /usr/bin/hostname: not found

HP-UX Start-up in progress
__________________________

Configure system crash dumps ............................................... OK
Removing old vxvm files .................................................... OK
........................................................................... OK
........................................................................... OK
........................................................................... FAIL *
Mount file systems ......................................................... FAIL *
Remounting Root File System ................................................ OK
Setting hostname ........................................................... OK
Start containment subsystem configuration .................................. OK
Set privilege group ........................................................ N/A
Display date ............................................................... N/A
Save system crash dump if needed ........................................... N/A
Start evp .................................................................. OK
Enable auxiliary swap space ................................................ OK
Start syncer daemon ........................................................ OK
Initializing livedump ...................................................... N/A
Start Utmp Daemon : manages User Accounting Database ....................... OK
Error: Cannot find "/usr/lbin/sw/control_utils". ........................... N/A
Configure IPv6 Upgrade ..................................................... FAIL *
Configure Loopback interfaces (lo0) ........................................ OK
Start IP Security (IPSec) .................................................. OK
Starting Event Management (EVM) (phase 1) .................................. OK
........................................................................... OK
........................................................................... OK
Continuing install jobs, configuring filesets .............................. OK
Configuring all unconfigured software filesets ............................. OK
Configuring PFIL ........................................................... N/A
Starting IPFilter .......................................................... N/A
ERROR: /opt/sec_mgmt/bastille/swLoopbackConfig.sh not present. Exiting. .... N/A
Recover editor crash files ................................................. OK
Clean UUCP ................................................................. OK
List and/or clear temporary files .......................................... OK
Clean up old log files ..................................................... OK
Start system message logging daemon ........................................ OK
Checking user database ..................................................... OK
Start pty allocator daemon ................................................. OK
Configuring OLA/R interface ................................................ N/A
Start network tracing and logging daemon ................................... OK
Configure HP igelan Gigabit Ethernet interfaces ............................ OK
Configure HP iether 100BT/Gigabit Ethernet interfaces ...................... OK
Configure HP gelan Gigabit Ethernet interfaces ............................. OK
Configure HP igssn Gigabit Ethernet interfaces ............................. FAIL *
Configure INTEL 100BASE-T interfaces ....................................... N/A
Configure HP 100BASE-T interfaces .......................................... N/A
Configure HP AUTO-PORT AGGREGATION interfaces .............................. FAIL *
Configure VLAN interfaces .................................................. OK
Configure LAN interfaces ................................................... OK
Configure LAN interfaces for IPv6 .......................................... OK
Starting Event Management (EVM) (phase 2) .................................. OK
Configure Auto-Port Aggregation(LAN Monitor) interfaces .................... FAIL *
Starting DHCPv6 Server daemon .............................................. N/A
Configuring DHCPv6 Interfaces .............................................. OK
Start name server daemon ................................................... N/A
Starting HP-UX Secure Shell ................................................ OK
Start NFS core subsystem ................................................... OK
Start NFS IPv6 subsystem ................................................... OK
Start NIS server subsystem ................................................. OK
Start ldap client daemon ................................................... N/A
Start NIS/LDAP server subsystem ............................................ N/A
Start NIS client subsystem ................................................. OK
Start lock manager subsystem ............................................... OK
Start NFS client subsystem ................................................. OK
Start AUTOFS subsystem ..................................................... OK
Finish containment subsystem configuration ................................. FAIL *
Start multicast routing daemon ............................................. N/A
OK ........................................................................... . i
........................................................................... OK
Start Internet services daemon ............................................. OK
Start dynamic routing daemon ............................................... N/A
Start ramd routing daemon .................................................. N/A
Start router discover protocol daemon ...................................... N/A
Configuring PPP Interface .................................................. OK
Configuring PPPoE .......................................................... N/A
Start RARP protocol daemon ................................................. N/A
Start HP Virtual Machine Monitor ........................................... FAIL *
Start remote system status daemon .......................................... N/A
Configuring HPVM Host AVIO Storage Software ................................ N/A
Start IPv6 router advertisement daemon ..................................... N/A
Starting sendmail [Done] Starting sm-client [Done] ......................... OK
........................................................................... N/A
........................................................................... N/A
Starting outbound connection daemons for DDFA software ..................... N/A
........................................................................... OK
Start SNMP Master Network Management daemon ................................ OK
Start OSPF MIB Network Management subAgent ................................. N/A
Start SNMP HP-UNIX Network Management subAgent ............................. OK
Start SNMP IPv6 Network Management subAgent ................................ OK
Start SNMP MIB-2 Network Management subAgent ............................... OK
Start Native Adapter Agent ................................................. N/A
Start SNMP Trap Dest Network Management subAgent ........................... OK
Start HA cluster SNMP subagent ............................................. OK
Start DCE daemons .......................................................... N/A
Start RPC daemon if needed ................................................. OK
Start CIM cimserver subsystem .............................................. OK
Initialize Instant Capacity ................................................ N/A
Starting X Font Server at TCP port 7000 .................................... N/A
Start vt daemon ............................................................ N/A
Start time synchronization ................................................. N/A
Start SNAplus2 ............................................................. N/A
Start accounting ........................................................... N/A
........................................................................... FAIL *
........................................................................... OK
........................................................................... N/A
Starting the password/group assist subsystem ............................... OK
Start print spooler ........................................................ N/A
Start clock daemon ......................................................... OK
Check Security Bulletin Compliance ......................................... N/A
Start diagnostic subsystem ................................................. OK
Start environment monitoring daemon ........................................ OK
........................................................................... OK
........................................................................... OK
Start auditing subsystem ................................................... N/A
........................................................................... OK
Start audio server daemon .................................................. N/A
Start USB hub daemon ....................................................... OK
SAM System administration configuration .................................... OK
........................................................................... OK
PA performance software is being started. .................................. OK
........................................................................... OK
Configure PRM -or- Configure and Enable PRM ................................ N/A
Start Workload Manager subsystem ........................................... N/A
Initialize Software Distributor agent daemon ............................... OK
........................................................................... N/A
Starting CIFS Client ....................................................... N/A

Starting Control Daemon .................................................... N/A

* - An error has occurred !
* - Refer to the file /etc/rc.log for more information.

The system is ready.

INIT: /etc/inittab: Bad action field for id "db2i" -- ignored entry

GenericSysName [HP Release B.11.31] (see /etc/issue)

[Patrick Wallek]

Your problem isn't with your file systems, it is with 'fsck' itself I think.

fsck: invalid argument to "-F FStype" option

fsck: vxfs is not a valid file system type
usage: fsck [-F FStype] [-V] [-m] [special...]
or fsck [-F FStype] [-V] [-m] [-o specific_options] [special...]

This means that fsck is trying to check vxfs file systems, but fsck thinks that vxfs isn't a valid file system type.

You say that you restored some data to VG00.  Did you restore anything for /sbin (which is what is used during boot-up)?

Post the results of this:

# cd /sbin

# ls -lR fs*

[Bill Hassell]

There are a *lot* of different errors...

fsck: invalid argument to "-F FStype" option
fsck: vxfs is not a valid file system type

INIT: /etc/inittab: Bad action field for id "db2i"

/sbin/lvmrc[101]: test: argument expected

/sbin/auto_parms[24]: who: not found
/sbin/auto_parms[34]: ps: not found
/sbin/auto_parms[34]: grep: not found
/sbin/rc1.d/S093vxvm-reconfig[50]: /etc/vx/lib/vxcommon: not found
interpreter "/bin/sh" not found

/sbin/rc3.d/S200tps.rc[15]: /usr/bin/hostname: not found

...many rc startup scripts have no name...

etc...

Your rebuild steps have severely messed up permissions, ownerships, too many to fix.

I don't think the system is salvageable at this point. Time to restore from your Ignite backup.

Or if you have a similar system, start by verifying the exact ownership and permission for each directory starting at /. With so many errors

[Ralf Seefeldt]

Hi,
you write that you did not restore / /usr and /stand.
So, exactly what did you do?
May be, you reinstalled the server from a backup, ignite or by reinstalling the basic OS.
To me, it looks like several products like vmfs support, auto port aggreagtion, some virtual machine stuff are missing on this server. Those software needs separate licensing or a higher than basic OS-version (like datacenter edition, ...)
If so, you have to reinstall that stuff.

Bye
Ralf

[Todd McDaniel_1]

correct me if Im wrong, but vxfs is provided by a patch. your restore may have blown away some definition for vxfs...

check to see if it is installed, if not patch it and try again.

or else you need to restore from ignite tape which may fix the issue.