Very slow bootup

[Coolmar]

I have a system that is suddenly taking very long (30 minutes) to boot up. Used to boot up fairly quickly and now it is taking too long. It seems to get hung up on the following:

Mail Daemon Status (ok)
SNMP HP-UNIX Network Management subAgent Status (ok)
Initialize Software Distributor agent daemon Status (ok)
Configure HP Dual Port 2Gbps/1Gbps Fiber Channel Universal PCI-X Adapter Status (ok)

The strange thing is that they eventually work and show as ok in the rc.log. I am thinking DNS issues but the /etc/resolv.conf and /etc/nsswitch are fine. Any ideas?

[Steven E. Protter]

I would do as follows:

check /etc/rc.log

Maybe there is a timeout happening.

Change config to NOT start sendmail, boot.

If that speeds it up at least you have a daemon to look at.

If networking isn't perfect, you mentioned the files sendmail will slow down a boot process considerably.

Also focus on network wide, what has changed.

SEP

[Rick Garland]

Were there community names changes recently? If you do not receive mail to the HPUX systems it is safe to turn off sendmail. (Will still send mail, just not receive it.) Is there something on the DNS server that is taking longer? Were there new additions, a new server put in place, new records?

Lots of possibilities. Hopefully I have provided some areas to direct your search.

Do check the rc.log. See if any abnormal messages in there.

[A. Clay Stephenson · Accepted solution]

You could be falling back to a secondary DNS server. Do some nslookups to test that. My best guess is that you have a mismatched speed/duplex setting between a NIC and its corresponding switch port. Surprisingly, mismatched duplexes will almost work well and wouldn't even be noticable in a telnet session, for example.

[Coolmar]

I got the following when I did a lanscan -x 0:

Current Config = 100 Full-Duplex AUTONEG

Doesn't say if auto negotiation is on or off...so not sure. I want it off though.

[A. Clay Stephenson]

You also need to know how the other end (the switch port) is set. Auto-negotiation requires that both ends be set to auto-negotiate. The bad news is that even then the negotiation can fail in some cases (and especially if the cable length is about 35-42m.

[Rick Garland]

Wasn't too long ago that HP NICs and CISCO switches had trouble with AUTONEG. Then you had to turn off the AUTONEG.

To turn off the AUTONEG use the 'lanadmin' command.

[Coolmar]

I turned off the AUTONEG and it is now set to "MANUAL". So hopefully that will solve the problem.

Thanks everyone

[Bill Hassell]

Even if you have a mismatch, it won't slow down the bootup process at all since there is virtually no traffic during that time. You can verify a duplaex mismatch with landiag. The statistic: FCS errors will be non-zero. As mentioned, you probably have a DNS issue. The resolv.conf and nsswitch.conf files probably haven't changed but it is likely the first DNS server has changed or has become unavailable. Every DNS failure in resolv.conf adds 20-30 seconds to every hostname lookup, and that can seriously slow down network-based programs, especially at startup.

Start by testing nslookup:

nslookup $(hostname)

The first line should verify the DNS server. If there is an error that the IP address of the DNS server is unknown, it means that the server doesn't know it's own name (a common mistake configuring DNS servers). It may not respond to the request and you'll see a message that says the rewuest timed out (after a long delay). To be sure, just ask nslookup to talk to a specific server:

nslookup hp.com 12.34.56.78

[Steven E. Protter]

AUTONEG/MANUAL is going to effect network transfer speed and possbily host resolution if what A.Clay says is true and you are failing over to a secondary DNS server.

You will find that network problems will cause slow bootup especially around the sendmail startup step.

To isolate the problem you might want to either time the steps in startup with a stopwatch or put write statements in the startup script that include the date command.

If the step or steps that are slow are isolated its quite possible to more accurately diangose the situation.

SEP

[Suraj Singh_1]

Also, you may boot into single user, and from there you may go to run-level 2 (rather than the default run-level 3), and see if there is any improvement. This way you should be able to isolate the problem causing daemon.

Regards,
Suraj

[Suraj Singh_1]

I guess you have a SAN environment.
You may also check the zoning (between storage & your server) issues.

[Fluite Kruid]

I had the same problem with my C3600. When connected to the LAN via an old 10Mb 3Com hub it would take about an hour to boot. I've now connected it to a newer dual-speed 3Com switch and it boots normally.

Strange thing is that while connected to the older hub, the hub would randomly lock-up, even if the C3600 was off. Removing the network cable unfroze the hub.

[javasoup]

Hi All, I'm facing the same problem of extremely long bootup time for my server. It wasn't like this previously, i'm using a K580 running HPUX 11.11.

Currently it took 1hr 30mins to bootup. I've logged a case to investigate the problem but i would like to put it on forum to discuss with the experts.

Previously it took only 20mins, now i tried using single user mode and bootup to 3rd level, it took only 15mins.

I have simply no idea what is wrong with the normal bootup, could anyone kindly tell me how to look for the source of this long bootup problem?

I figured it that there might be some hardware monitoring process that needs a long time to process during the bootup before HPUX.

Anyone has any advise, please give me some tips, thanks!

Sincerely,
Edmund

[Bill Hassell]

As mentioned in previous posts, you need to time each step in the bootup process. Find out how long it takes for selftests. A K580 is fairly fast for it's generation but it can take 15mins to test 8Gb of RAM. My philosophy for extensive RAM testing is that it should only be enabled when troubleshooting a known memory error. Production systems are always testing memory by using it and rebooting is non-productive time. So I enable fastboot to shorten the memory test time. You can always turn it back on (only) when there is a memory problem.

The rest of the bootup process is a set of scripts so you need to find where the OS spends (wastes) most of it's time. Probably the most common time-waster is NFS. If you're not using it, turn all of it off in the nfsconf file. Post the names of the tasks that take the longest time.