topic Re: Boot failure after QPK11 03/02 installation in Operating System - HP-UX

Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 16:17:29 GMT

Hi there...

THis afternoon I remotely installed qpk11 march 02, and the system failed to return. Upon getting to the console all that appears are two init errors and no prompt. I can hit ctrl-c to get a login prompt, but after login the systems doesn't know the hostname. I can run /sbin/init.d/hostname to fix that but there no entries in the routing table and the network card isnt' configured. I think there's a serious init problem. There's no /etc/rc.log either and syslog is no longer being updated. I don't see the usual list of subsystem startups during the init sequence either. Any ideas? I'm going to boot to single user and increase the init levels manually while I wait for the expert replies.

Ian

Re: Boot failure after QPK11 03/02 installation

Ted Ellis_2 — Mon, 21 Oct 2002 16:24:06 GMT

what are the errors that you see exactly?

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 16:30:35 GMT

The first is from the legato networker and is simply basename not found, and the other is named stating unknown hostname. Nothing conclusive just symptoms of a root cause which I have not yet identified.

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 16:31:27 GMT

More...

I just did a ps -ef | more and there are only 27 processes running. ie. Nothing started anywhere.

Re: Boot failure after QPK11 03/02 installation

Ashwani Kashyap — Mon, 21 Oct 2002 16:35:05 GMT

What run level are you in .
who -r should tell that .

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 16:51:42 GMT

who -r reveals run level 3. Which is what it has in /etc/inittab also.

Re: Boot failure after QPK11 03/02 installation

Ashwani Kashyap — Mon, 21 Oct 2002 16:58:57 GMT

I would say your original idea is better . Boot to single user and then individually turn on each run level . That way you know which run level your system has problems in .

Re: Boot failure after QPK11 03/02 installation

Ted Ellis_2 — Mon, 21 Oct 2002 16:59:38 GMT

what was the system patch level prior to this install? Curious if the system was relatively current, which would make the QPK relatively minor in most cases, or long overdue for patching, which would make a QPK pretty intense. Is everything on the system mounted ok? mount -a

what do you see with lanscan?

Was anything else changed prior to the patching?

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 17:06:14 GMT

Patch level was at 46 I think.. so 1999-2000 somewhere. Yes all filesystems mount fine.

Lanscan shows both interfaces UP.

I've only done one change since I inherited this system and I simply increased /usr. System got booted on oct 7 though. We have some junior admins who have a bad habit of rebooting to solve problems. I might send them on NT admin courses.

manual init results shortly...

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 17:33:16 GMT

Damn!! Everything works fine when I run the init sequences manually. When it finishes run level 3 I'll shutdown normally again. You never know. Any thoughts??

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Mon, 21 Oct 2002 18:31:18 GMT

transition to run level 3 doesn't show the subsystem startup list, but I'm not sure if that's normal or not for manual startup. The restart didn't help. I get the feeling that init is somehow skipping levels 1 and 2 and attempting 3 immediately upon reboot. Does that ring any bells for anyone.

Anyway.. I'm manually initting again and that's where I'm going to leave it over night.
Points tomorrow...

ian

Re: Boot failure after QPK11 03/02 installation

Ashwani Kashyap — Mon, 21 Oct 2002 19:40:50 GMT

Ian ,

Check whether your files in /sbin/rcx.d ( where x = 1,2,3,4 ) are linked correctly with the appropriate files in /sbin/init.d .

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Tue, 22 Oct 2002 08:09:48 GMT

There doesn't seem to be anything wrong with the sym links.

It is definitely skipping levels 1 and 2 and attempting 3. I looked back through old rc.logs and found that those original 2 networker/named errors pre-existed the patching.

I've attached my inittab incase you might spot any anomolies that i cant.

Re: Boot failure after QPK11 03/02 installation

George Petrides_1 — Tue, 22 Oct 2002 08:39:45 GMT

Weird startup problems can be sometimes caused by a script residing in the /etc/rc.config.d directory. The startup program (/sbin/rc) executes with a '.' all files in /etc/rc.config.d and if someone throws a script in that directory he/she can cause the machine to hang during startup.
Just my $0.02.
George

Re: Boot failure after QPK11 03/02 installation

George Petrides_1 — Tue, 22 Oct 2002 08:42:42 GMT

If these were pre-existing problems, it's one more reason to reboot the machine on a frequent basis. Sometimes, we live with systems problems that we do not know about and the machine is up for months or more than 1 year. When we apply patches and reboot problems come up that are not really related to the patches but since the coincide we assume they are related. I strongly believe is a good practice to reboot the server even once a month.
George

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Tue, 22 Oct 2002 08:50:53 GMT

Thanks george. I had rebooted oct 7. The errors were around then but didn't cause a failure to boot. I'll have a play with /etc/rc.config.

Re: Boot failure after QPK11 03/02 installation

Dave Unverhau_1 — Tue, 22 Oct 2002 10:16:08 GMT

Ian,

I didn't see this mentioned, but have you looked at /var/adm/syslog/syslog.log for startup errors, or var/adm/sw/swagent.log for errors during the QPK install session? Also, a check for unconfigured or corrupt software with 'swlist -l fileset -a state' command? It might provide some further insight.

Regards,

Dave

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Tue, 22 Oct 2002 11:38:08 GMT

Thanks dave. swlist shows all filesets configured and swagentd.log has no errors for this installation, nor does swagent.log.

Re: Boot failure after QPK11 03/02 installation

Ian Killer_1 — Tue, 22 Oct 2002 11:57:36 GMT

Unbelievable. It rebooted fine. Possibly down to a dodgey script in run level 3. Unlikely but that's the only change since it failed.

Thanks all for you help.