HP-UX 11.31 boot not working correctly

[ah1kayyali]

dears,

we tried to install the latest QPK, and the swinstall show succesfull.

after reboot the OS is not booting, and unfotunately we don't have ignite backup...

when we try to boot the system now, we got the following

Configure system crash dumps ................................ OK
     Removing old vxvm files ..................................... OK
     Mount file systems .......................................... OK
     Remounting Root File System ................................. OK
     Setting hostname ............................................ OK
     Start containment subsystem configuration ................... FAIL *
     Set privilege group ......................................... N/A
     Display date ................................................ N/A
     Save system crash dump if needed ............................ N/A
     Start evp ................................................... OK
     Enable auxiliary swap space ................................. FAIL *
     Start syncer daemon ......................................... OK
     Initializing livedump ....................................... N/A
     Start Utmp Daemon : manages User Accounting Database ........ FAIL *
     Configure Loopback interfaces (lo0) ......................... OK
     Reconfiguring setboot event subscription .................... OK
     Starting Event Management (EVM) (phase 1) ................... FAIL *
     Start Essential Services monitoring daemon .................. OK
     Continuing install jobs, configuring filesets ............... OK
     Configuring all unconfigured software filesets .............. FAIL *
     Configuring PFIL ............................................ N/A
     Starting IPFilter ........................................... N/A
     Configuring Install Time Security ........................... N/A
     Recover editor crash files .................................. OK
     Clean UUCP .................................................. OK
     List and/or clear temporary files ........................... OK
     Clean up old log files ...................................... OK
     Updating nPartition run state ............................... OK
     Start system message logging daemon ......................... OK
     Checking user database ...................................... OK
     Start pty allocator daemon .................................. OK
     Configuring OLA/R interface ................................. OK
     Start dynamic P-states power savings ........................ N/A
     Start network tracing and logging daemon .................... OK
     Configure HP igelan Gigabit Ethernet interfaces ............. FAIL *
     Configure HP iether 100BT/Gigabit Ethernet interfaces ....... FAIL *
     Configure HP iexgbe 10 Gigabit Ethernet interfaces .......... FAIL *
     Configure HP gelan Gigabit Ethernet interfaces .............. FAIL *
     Configure HP igssn Gigabit Ethernet interfaces .............. FAIL *
     Configure INTEL 100BASE-T interfaces ........................ N/A
     Configure HP 100BASE-T interfaces ........................... N/A
     Configure HP AUTO-PORT AGGREGATION interfaces ............... FAIL *
     Configure VLAN interfaces ................................... OK
     Configure LAN interfaces .................................... OK
     Configure LAN interfaces for IPv6 ........................... OK
     Starting Event Management (EVM) (phase 2) ................... FAIL *
     Configure Auto-Port Aggregation(LAN Monitor) interfaces ..... OK
     Starting DHCPv6 Server daemon ............................... N/A
     Configuring DHCPv6 Interfaces ............................... OK
     Start name server daemon .................................... N/A
     Starting HP-UX Secure Shell ................................. FAIL *
     Start NFS core subsystem .................................... OK
     Start NFS IPv6 subsystem .................................... N/A
     Start enhanced NFS IPv6 subsystem ........................... N/A
     Start NIS server subsystem .................................. OK
     Start ldap client daemon .................................... N/A
     Start NIS/LDAP server subsystem ............................. N/A
     Start NIS client subsystem .................................. OK
     Start lock manager subsystem ................................ OK
     Start NFS client subsystem .................................. OK
     Start AUTOFS subsystem ...................................... OK
     Finish containment subsystem configuration .................. FAIL *
     Start multicast routing daemon .............................. N/A
     Start Internet services daemon .............................. FAIL *
     Start dynamic routing daemon ................................ N/A
     Start ramd routing daemon ................................... N/A
     Start router discover protocol daemon ....................... N/A
     Configuring PPP Interface ................................... OK
     Configuring PPPoE ........................................... N/A
     Start RARP protocol daemon .................................. N/A
     Start remote system status daemon ........................... N/A
     Start IPv6 router advertisement daemon ...................... N/A
     Starting sendmail [Done] Starting sm-client [Done] .......... OK
     Starting cfengine's cfservd daemon .......................... N/A
     Starting syslog-ng daemons .................................. N/A
     Starting outbound connection daemons for DDFA software ...... N/A
     Starting sfmdb PostgreSQL daemons ........................... OK
     Start SNMP Master Network Management daemon ................. OK
     Start OSPF MIB Network Management subAgent .................. N/A
     Start SNMP HP-UNIX Network Management subAgent .............. OK
     Start SNMP IPv6 Network Management subAgent ................. OK
     Start SNMP MIB-2 Network Management subAgent ................ OK
     Start Native Adapter Agent .................................. N/A
     Start SNMP Trap Dest Network Management subAgent ............ OK
     Start DCE daemons ........................................... N/A
     Start RPC daemon if needed .................................. N/A
     Start CIM cimserver subsystem ............................... FAIL *
     Initialize Instant Capacity ................................. FAIL *
     Starting X Font Server at TCP port 7000 ..................... N/A
     Start vt daemon ............................................. N/A
     Start time synchronization .................................. OK
     Start accounting ............................................ N/A
     Install/Load XF86 DLKM Helper Modules ....................... N/A
     Starting the password/group assist subsystem ................ N/A
     Start print spooler ......................................... N/A
     Start clock daemon .......................................... N/A
     Start oserrlog daemon ....................................... FAIL *
     Check Security Bulletin Compliance .......................... N/A
     Start diagnostic subsystem .................................. FAIL *
     Start environment monitoring daemon ......................... OK
     Start auditing subsystem .................................... N/A
     Start audio server daemon ................................... N/A
     Start USB hub daemon ........................................ OK
     SAM System administration configuration ..................... OK
     Configure PRM -or- Configure and Enable PRM ................. N/A
     Initialize Software Distributor agent daemon ................ FAIL *
     Starting the System Management HomePage server .............. OK
     Starting the gWLM Agent ..................................... N/A
     Starting CIFS Client ........................................ N/A
     Configure HP RAID SA interfaces ............................. N/A
     Performing any needed DRD cleanup ........................... OK
     Starting Event Monitoring Service ........................... OK
     Start EMS SNMP subagent ..................................... OK
     Start interrupt balance daemon .............................. FAIL *
     Configuring Ultra320 SCSI Mass Storage interfaces ........... N/A
     Starting the Winbind Daemon ................................. N/A
     Configuring HP SerialSCSI SASD Mass Storage interfaces ...... OK
     Configuring HPVM AVIO Mass Storage interfaces ............... OK
     Starting the NetBackup client daemons ....................... OK
     Configuring HP Fibre Channel FCD Mass Storage interfaces .... OK
     Start NFS server subsystem .................................. OK
     Start X print server(s) ..................................... N/A
     Starting HP-UX Apache-based Web Server ...................... N/A
     Starting HP-UX Tomcat-based Servlet Engine .................. N/A
     Starting HP-UX Webmin-based Admin ........................... N/A
     Starting the HPUX Webproxy subsystem ........................ N/A
     Starting HP-UX XML Web Server Tools ......................... OK
     Start kwdbd ................................................. N/A
     Validating HP Virtual Machine Configuration ................. N/A
     Start LVM daemon ............................................ OK
      ............................................................

and the system will be hang on this stage, it appears with ping on the network, but it is not responding to ssh or telnet.

googling since tuesday bdid not help,

any ideas

[Dennis Handly]

Can you look at /etc/rc.log & rc.log.old?

[ah1kayyali]

#
#
#
# cat /etc/rc.log
Old /etc/rc.log moved to /etc/rc.log.old

**************************************************
HP-UX Start-up in progress
Sat Dec 22 08:58:50 2012
**************************************************

Configure system crash dumps
Output from "/sbin/rc1.d/S080crashconf start":
----------------------------
EXIT CODE: 0

Removing old vxvm files
Output from "/sbin/rc1.d/S090sw_clean_vxvm start":
----------------------------

Mount file systems
Output from "/sbin/rc1.d/S100localmount start":
----------------------------
checking quotas

Remounting Root File System
Output from "/sbin/rc1.d/S101vxfs_remount_root start":
----------------------------

Setting hostname
Output from "/sbin/rc1.d/S320hostname start":
----------------------------

Start containment subsystem configuration
Output from "/sbin/rc1.d/S330sec_init start":
----------------------------
/sbin/rc1.d/S330sec_init[49]: setfilexsec:  not found.
ERROR CODE 127
/sbin/rc1.d/S330sec_init[82]: setfilexsec:  not found.
ERROR CODE 127
"/sbin/rc1.d/S330sec_init start" FAILED

Set privilege group
Output from "/sbin/rc1.d/S400set_prvgrp start":
----------------------------
"/sbin/rc1.d/S400set_prvgrp start" SKIPPED

Display date
Output from "/sbin/rc1.d/S420set_date start":
----------------------------
"/sbin/rc1.d/S420set_date start" SKIPPED

Save system crash dump if needed
Output from "/sbin/rc1.d/S440savecrash start":
----------------------------
savecrash directory not set;  defaulting to: /var/adm/crash
savecrash: Dump previously saved, use -r to resave

EXIT CODE: 2 -  savecrash found no core dump to save
"/sbin/rc1.d/S440savecrash start" SKIPPED

Start evp
Output from "/sbin/rc1.d/S450evp.init start":
----------------------------

Enable auxiliary swap space
Output from "/sbin/rc1.d/S500swap_start start":
----------------------------
Enabling device paging on /dev/vg00/sswap.
Enabling device paging on /dev/vg00/sswap.
/usr/sbin/swapon: /dev/vg00/sswap is already enabled for paging.
Warning: swapon returned exit code: 1
"/sbin/rc1.d/S500swap_start start" FAILED

Start syncer daemon
Output from "/sbin/rc1.d/S520syncer start":
----------------------------
syncer started

Initializing livedump
Output from "/sbin/rc1.d/S540livedump start":
----------------------------
"/sbin/rc1.d/S540livedump start" SKIPPED

Start Utmp Daemon : manages User Accounting Database
Output from "/sbin/rc1.d/S600utmpd start":
----------------------------
/sbin/rc1.d/S600utmpd[42]: /usr/sbin/utmpd:  not found.
EXIT CODE: 127
"/sbin/rc1.d/S600utmpd start" FAILED

Configure Loopback interfaces (lo0)
Output from "/sbin/rc2.d/S008net.init start":
----------------------------
Boot time cleanup of /etc/ifconfig.muxids completed.
NOTE: /var/tmp/NETSTAT.TMP/ cleanup only occurs during boot time.
Message catalog can't be opened/accessed for language en_US.iso88591.
Language C will used.

Reconfiguring setboot event subscription
Output from "/sbin/rc2.d/S019setboot start":
----------------------------

Starting Event Management (EVM) (phase 1)
Output from "/sbin/rc2.d/S020evm start":
----------------------------
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/usr/sbin/evmstart[80]: 542 Abort(coredump)
evmstart: Daemon failed to start properly
EXIT CODE: 1
"/sbin/rc2.d/S020evm start" FAILED

Start Essential Services monitoring daemon
Output from "/sbin/rc2.d/S021esm start":
----------------------------

Continuing install jobs, configuring filesets
Output from "/sbin/rc2.d/S119swm.config start":
----------------------------

Configuring all unconfigured software filesets
Output from "/sbin/rc2.d/S120swconfig start":
----------------------------
       * Turning off all network based resolving services in
         '/etc/nsswitch.conf'
       * Starting temporary swagentd
ERROR:   The agent binary "/usr/lbin/swagent" cannot be executed.  No
         such file or directory (2).  Exiting.
ERROR:   Unable to start the temporary swagentd.
       * Restoring '/etc/nsswi#
#
#
#
#
#

[ah1kayyali]

Due to large file, check attached rc.log and rc.log.old

[Matti_Kurkela]

Your /etc/rc.log seems to have been interrupted or truncated somehow.

But it looks like your system is missing at least one critical system file, perhaps others.

Start containment subsystem configuration
Output from "/sbin/rc1.d/S330sec_init start":
----------------------------
/sbin/rc1.d/S330sec_init[49]: setfilexsec:  not found.
ERROR CODE 127

[...]

Start Utmp Daemon : manages User Accounting Database
Output from "/sbin/rc1.d/S600utmpd start":
----------------------------
/sbin/rc1.d/S600utmpd[42]: /usr/sbin/utmpd:  not found.
EXIT CODE: 127

[...]

Starting Event Management (EVM) (phase 1)
Output from "/sbin/rc2.d/S020evm start":
----------------------------
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/usr/sbin/evmstart[80]: 573 Abort(coredump)

[...]

       * Starting temporary swagentd
ERROR:   The agent binary "/usr/lbin/swagent" cannot be executed.  No
         such file or directory (2).  Exiting.
ERROR:   Unable to start the temporary swagentd.
       * Restoring '/etc/nsswitch.conf' to its original contents
"/sbin/rc2.d/S120swconfig start" FAILED

[...]

ERROR:  /usr/sbin/lanadmin is missing or not executable.
"/sbin/rc2.d/S305hpigelan start" FAILED

[...]

Output from "/sbin/rc2.d/S393secsh start":
----------------------------
/usr/lib/hpux64/uld.so: Unable to open '/usr/lib/hpux64/dld.so'.
/sbin/rc2.d/S393secsh[70]: 855 Abort(coredump)
EXIT CODE: 134

In fact, many or all of these errors might have a single cause: if /usr/lib/hpux64/dld.so is missing or unusable, many programs will stop working. Not all of them will even be able to output an accurate error message in this situation: some might report that the program binary is "not found" even though it actually exists. So I would concentrate on fixing the /usr/lib/hpux64/dld.so first. It might not solve all the problems of this system, but I think it might solve more than one of them at once.

As sshd and other network services have not been able to start properly, console access is obviously required to fix this.

You said you don't have an Ignite backup. Do you have *any* sort of backup that would include /usr/lib/hpux64/dld.so? Could you extract this file from the backup on some other host, and then use some removable media to transfer it to the system that has the problem? You'll probably need a tape or a CD-R disc; I'm not sure if USB storage support will work in this situation. Your backup is probably from the time before the patching, so replacing a patched file with an unpatched one is not ideal, but it might be good enough for the first step of the recovery.

Once you can get the system to the point that your normal backup software can run, restoring a known good backup would be a good idea. Obviously something went badly wrong in the patching. (The swinstall/swagentd logs at /var/adm/sw might have more details!)

By the way, according to rc.log.old, you have a huge number of files in /tmp. If you have installed applications that use /tmp like this, I would recommend cleaning /tmp and/or making sure all the application components are stopped before patching. Of course swinstall checks the available disk space before starting, but if an application fills up a filesystem (like /tmp) while swinstall is running, bad things might happen.

[Dennis Handly]

>uld.so: Unable to open '/usr/lib/hpux64/dld.so'.

As Matti said, this is the majority of your problems.

>Could you extract this file from the backup on some other host

You can get this from the latest linker & dld patch.

>Of course swinstall checks the available disk space before starting,

The amount of space used in /tmp and /var/tmp is < 2 MB, trivial.

But the biggest problem may be all of the recent files in lost+found:

NOTE:  Files in /lost+found:
total 24704
crw-rw-rw-   1 root       root        15 0x000000 Dec 18 14:35 1032
crw-------   1 root       root         7 0x000000 Dec 18 14:48 1044

...