Boot failure after QPK11 03/02 installation

[Ian Killer_1]

Hi there...

THis afternoon I remotely installed qpk11 march 02, and the system failed to return. Upon getting to the console all that appears are two init errors and no prompt. I can hit ctrl-c to get a login prompt, but after login the systems doesn't know the hostname. I can run /sbin/init.d/hostname to fix that but there no entries in the routing table and the network card isnt' configured. I think there's a serious init problem. There's no /etc/rc.log either and syslog is no longer being updated. I don't see the usual list of subsystem startups during the init sequence either. Any ideas? I'm going to boot to single user and increase the init levels manually while I wait for the expert replies.

Ian

[Ted Ellis_2]

what are the errors that you see exactly?

[Ian Killer_1]

The first is from the legato networker and is simply basename not found, and the other is named stating unknown hostname. Nothing conclusive just symptoms of a root cause which I have not yet identified.

[Ian Killer_1]

More...

I just did a ps -ef | more and there are only 27 processes running. ie. Nothing started anywhere.

[Ashwani Kashyap]

What run level are you in .
who -r should tell that .

[Ian Killer_1]

who -r reveals run level 3. Which is what it has in /etc/inittab also.

[Ashwani Kashyap]

I would say your original idea is better . Boot to single user and then individually turn on each run level . That way you know which run level your system has problems in .

[Ted Ellis_2]

what was the system patch level prior to this install? Curious if the system was relatively current, which would make the QPK relatively minor in most cases, or long overdue for patching, which would make a QPK pretty intense. Is everything on the system mounted ok? mount -a

what do you see with lanscan?

Was anything else changed prior to the patching?

[Ian Killer_1]

Patch level was at 46 I think.. so 1999-2000 somewhere. Yes all filesystems mount fine.

Lanscan shows both interfaces UP.

I've only done one change since I inherited this system and I simply increased /usr. System got booted on oct 7 though. We have some junior admins who have a bad habit of rebooting to solve problems. I might send them on NT admin courses.

manual init results shortly...

[Ian Killer_1]

Damn!! Everything works fine when I run the init sequences manually. When it finishes run level 3 I'll shutdown normally again. You never know. Any thoughts??

[Ian Killer_1]

transition to run level 3 doesn't show the subsystem startup list, but I'm not sure if that's normal or not for manual startup. The restart didn't help. I get the feeling that init is somehow skipping levels 1 and 2 and attempting 3 immediately upon reboot. Does that ring any bells for anyone.

Anyway.. I'm manually initting again and that's where I'm going to leave it over night.
Points tomorrow...

ian

[Ashwani Kashyap]

Ian ,

Check whether your files in /sbin/rcx.d ( where x = 1,2,3,4 ) are linked correctly with the appropriate files in /sbin/init.d .

[Ian Killer_1]

There doesn't seem to be anything wrong with the sym links.

It is definitely skipping levels 1 and 2 and attempting 3. I looked back through old rc.logs and found that those original 2 networker/named errors pre-existed the patching.

I've attached my inittab incase you might spot any anomolies that i cant.

[George Petrides_1]

Weird startup problems can be sometimes caused by a script residing in the /etc/rc.config.d directory. The startup program (/sbin/rc) executes with a '.' all files in /etc/rc.config.d and if someone throws a script in that directory he/she can cause the machine to hang during startup.
Just my $0.02.
George

[George Petrides_1]

If these were pre-existing problems, it's one more reason to reboot the machine on a frequent basis. Sometimes, we live with systems problems that we do not know about and the machine is up for months or more than 1 year. When we apply patches and reboot problems come up that are not really related to the patches but since the coincide we assume they are related. I strongly believe is a good practice to reboot the server even once a month.
George

[Ian Killer_1]

Thanks george. I had rebooted oct 7. The errors were around then but didn't cause a failure to boot. I'll have a play with /etc/rc.config.

[Dave Unverhau_1]

Ian,

I didn't see this mentioned, but have you looked at /var/adm/syslog/syslog.log for startup errors, or var/adm/sw/swagent.log for errors during the QPK install session? Also, a check for unconfigured or corrupt software with 'swlist -l fileset -a state' command? It might provide some further insight.

Regards,

Dave

[Ian Killer_1]

Thanks dave. swlist shows all filesets configured and swagentd.log has no errors for this installation, nor does swagent.log.

[Ian Killer_1]

Unbelievable. It rebooted fine. Possibly down to a dodgey script in run level 3. Unlikely but that's the only change since it failed.

Thanks all for you help.