Debugging startup process

[Vince Laurent]

We have a handful of servers that exhibit the same behavior. Not everything starts up (according to rc.log).  Here is an example:

(root@sapsnd)# tail /etc/rc.log
Start LVM daemon
Output from "/sbin/rc3.d/S900lvm start":
----------------------------
Starting up the lvmpud daemon
/usr/sbin/lvmpud

Starting SAP Services
Output from "/sbin/rc3.d/S900sapinit start":
----------------------------
start hostcontrol using profile /usr/sap/hostctrl/exe/host_profile

There are a few more things that should start but don't:

(root@sapsnd)# ls -al1 S9*
lrwxr-xr-x 1 root sys 21 Mar 18 2011 S900OVTrcSrv -> /sbin/init.d/OVTrcSrv
lrwxrwxrwx 1 bin bin 16 Mar 18 2011 S900lvm -> /sbin/init.d/lvm
lrwxr-x--- 1 root root 20 Apr 22 2013 S900sapinit -> /sbin/init.d/sapinit
lrwxr-xr-x 1 root sys 16 Mar 18 2011 S920sap -> /sbin/init.d/sap
lr-xr--r-- 1 bin bin 23 Mar 18 2011 S990dtlogin.rc -> /sbin/init.d/dtlogin.rc
lrwxr-xr-x 1 root sys 18 Mar 18 2011 S998amgrd -> /sbin/init.d/amgrd
lrwxr-xr-x 1 root sys 21 Feb 10 2012 S99prngd.rc -> /sbin/init.d/prngd.rc
lrwxr-xr-x 1 root sys 19 Feb 17 10:34 S99z0OVCtrl -> /sbin/init.d/OVCtrl

What is the best way to debug the process? 

Thanks

vince

p.s. running HP-UX 11.31

[Bill Hassell · Accepted solution]

The first place to start is with /etc/rc.config.d

This directory is unlike ANY other directory in HP-UX. EVERY file in this directory will be executed by every start script. That means that *NO* junk files or old files or different versions of files can be in that directory. Just before all the rc scripts are started, you may see all sorts of strange error messages due to bad files in that directory.

To debug, sort the files by time order, most recent at the bottom:

cd /etc/rc.config.d
ls -lrt

The files at the bottom were recently created or changed. They are the most suspect. If you want to keep old copies of the control files, make a directory called old and move all the old files to old. The directory protects these files from being executed.

Then for each failed rc script, run the failed script manually. I did not see any error messages in your post. Are you saying that the sapinit scipt is failing? What happens when you run it manually?

[Vince Laurent]

Thanks - baby steps through each item showed that one was waiting for user input. Fixed it so it doesn't ask anymore and all is happy.