Server bootup and stopped halfway through rc2.d

[Coolmar]

Hi,

We had to reboot a server yesterday and thought it booted up alright until we found various apps/services not started. I checked the /etc/rc.log and found that it only made it partway through rc2.d. I can't find anything that hints as to why it stopped. Any ideas?

Here is the /etc/rc.log (the tail end of it anyway):

Starting mail daemon
Output from "/sbin/rc2.d/S540sendmail start":
----------------------------
/etc/mail/aliases: 7 aliases, longest 9 bytes, 88 bytes total
sendmail

Starting outbound connection daemons for DDFA software
Output from "/sbin/rc2.d/S550ddfa start":
----------------------------
"/sbin/rc2.d/S550ddfa start" SKIPPED

Start SNMP Master Network Management daemon
Output from "/sbin/rc2.d/S560SnmpMaster start":
----------------------------

Start OSPF MIB Network Management subAgent
Output from "/sbin/rc2.d/S565OspfMib start":
----------------------------
"/sbin/rc2.d/S565OspfMib start" SKIPPED

Start SNMP HP-UNIX Network Management subAgent
Output from "/sbin/rc2.d/S565SnmpHpunix start":
----------------------------

Start SNMP MIB-2 Network Management subAgent
Output from "/sbin/rc2.d/S565SnmpMib2 start":
----------------------------

Start SNMP Trap Dest Network Management subAgent
Output from "/sbin/rc2.d/S565SnmpTrpDst start":
----------------------------

Start DCE daemons
Output from "/sbin/rc2.d/S570dce start":
----------------------------
Beginning DCE startup...
NOTE: No components of DCE are configured.

Finished DCE startup.
"/sbin/rc2.d/S570dce start" SKIPPED

Start SNMP PCI FDDI Network Management subAgent
Output from "/sbin/rc2.d/S576SnmpFddi4 start":
----------------------------

Start RPC daemon if needed
Output from "/sbin/rc2.d/S590Rpcd start":
----------------------------
starting the DCE RPC Daemon
/opt/dce/sbin/rpcd

Start CIM cimserver subsystem
Output from "/sbin/rc2.d/S600cimserver start":
----------------------------
CIM cimserver started

Start Isogon License Server Daemon
Output from "/sbin/rc2.d/S600iforls start":
----------------------------
"/sbin/rc2.d/S600iforls start" SKIPPED

Initialize capacity on demand
Output from "/sbin/rc2.d/S602icod start":
----------------------------

[Tim Nelson]

Looks stuck at ICOD, are you using capacity on demand ?

ps -ef|grep rc may tell you what is currently running from the boot. If ICOD or some other process that does not matter then kill it. The system will finish running the rest of the boot scripts.

You will still need to find out why ICOD is stuck.

I am not familiar with the processes associated with ICOD.

Someone else I am sure is.

[Rodney Hills]

I once put in a "read" command to input from a terminal into /etc/profile. Some of the rc startup scripts do run /etc/profile, thus halting the process (as it waits for terminal input when a terminal is not connected).

HTH

-- Rod Hills

[Blofeld]

You may want to try to change the start up file in rc.config.d to not start the ICOD process if you are not using it and see how far the boot process gets.

[Blofeld]

sorry not in rc.config.d/. May just mv it out of the rc2.d dir all together. Just remember where you put so you can put it back.

The script is just calling the command /usr/sbin/icod_stat -z

[Coolmar]

The ICOD seems to have started up ok. When I do a ps on ICOD I get:

ps -ef |grep icod
root 1330 1 0 Aug 30 ? 0:00 /usr/lbin/icodd

[Blofeld]

Sally
If you are sure you are not using the ICOD service, look to see if the file /var/adm/icod.log exists, try and stop it using the /sbin/init.d/icod stop. Then run the startup and see if you get any errors.

[Michael Jorgensen]

Sally,

As suggested it looks like the rc process has hung on the icod startup. It's interesting that you say it looks like this is running... now I don't know what icod is but if you want to ensure that your system starts up you'll have to see if it is indeed stuck on that init script.

Do a ps -ef | grep rc on your node and show us the output. You'll likely find that /sbin/rc2.d/S602icod start is still running. This will stop the next init scripts from starting because it does them sequentially.

What you'll find likely is that while a ps shows the icod process running it's not finished initialising properly or didn't exit properly so rc hasn't picked up on it and moved on.

Best bet is to kill the icod init script and watch as rc resumes where it was up to and finish booting properly. Then investigate what icod is, if it's important, is it running properly now and more importantly will this happen next time you boot?

[Coolmar]

Hi all,

I did do a grep on the iocd startup script and it was not in the running processes list. I rebooted again last night and it all came up fine...so I guess that previous reboot just stopped at rc2.d and I will just chalk it up to "whatever". lol!

Thanks for the help everyone,
Sally

[Michael Jorgensen]

When you did your ps was it grep'ing for "icod" or were you grepping for "rc", I guess it's possible it was hung on whatever the next init script was and if you only grep'd for "iocd" you wouldn't have seen what it was hung on.

[javasoup]

Hi All, I'm facing the this problem of extremely long bootup time for my server. Although it did bootup to HPUX successfully, it wasn't like this previously, i'm using a K580 running HPUX 11.11.

Currently it took 1hr 30mins to bootup. I've logged a case to investigate the problem but i would like to put it on forum to discuss with the experts.

Previously it took only 20mins, now i tried using single user mode and bootup to 3rd level, it took only 15mins.

I have simply no idea what is wrong with the normal bootup, could anyone kindly tell me how to look for the source of this long bootup problem?

I figured it that there might be some hardware monitoring process that needs a long time to process during the bootup before HPUX.

Anyone has any advise, please give me some tips, thanks!

Sincerely,
Edmund