Startup Errors

[Pete Randall]

Morning folks!

We had a planned re-boot of our production system this morning. Something didn't look quite right when I came in, so I started checking. Looking at /etc/rc.log, the first thing I noticed as this:

Virtual Partitions Initialization
Output from "/sbin/rc1.d/S105vparinit start":
----------------------------
/sbin/rc1.d/S105vparinit[138]: 394 Memory fault(coredump)
Building a new kernel based on template file "/stand/system"
Generating module: krm...
Compiling /stand/build/conf.c...
Loading the kernel...
Generating kernel symbol table...
NOTE: /stand/vmunix now relocatable ... rebooting.

For the life of me, I can't remember whether that's normal behaviour with VPARS or not. I'll check some old rc.logs to see if I can find out.

However, continuing through rc.log, I see repeated occurences of this:

Output from "/sbin/rc1.d/S320hostname start":
----------------------------
/sbin/rc1.d/S320hostname[36]: !/sbin/sh: not found.

Line 36, and all the other lines mentioned in the failures, point at the second line in this sequence:

if [ -f /etc/rc.config ] ; then
. /etc/rc.config

or the second line in this sequence:

if [ -f /etc/rc.config.d/informix-fair ] ; then
. /etc/rc.config.d/informix-fair


In other words, the "dot space /etc/rc.config" sourcing line.

So far, the Support Center is drawing a blank - they're referring me to the "Addtional Resources" group, or something like that. I figured while I was waiting I might as well check with you folks.

Your thoughts, ideas, and suggestions are, as always, appreciated.


Pete

[Doug O'Leary]

Hey;

First, the coredump in the vpar init would have me *very* concerned.

Second, check all the files in /etc/rc.config.d for anything that shouldn't be there - like the core file from the vparinit. The /etc/rc.config file simply sources every file in /etc/rc.config.d. If the file isn't supposed to be there or has an error in it, *every* init script that sources the /etc/rc.config file will error out.

If you've ever tried to cat a core file, you can just imagine what will happen if you try to execute it...

HTH;

Doug

[Steven E. Protter]

Shalom,

Sounds like the whole vpar has become corrupt for some reason.

You've been referred to the back end engineers at the support center.

I've had a few cases referred there in the past and they are REALLY brilliant.

I think i managed to do this doing a vpar excercize at HP Technology forum but don't know what I did.

I've nothing to add and don't expect any points Pete.

Good Luck,

SEP

[Torsten.]

Hi Pete,

the message

"NOTE: /stand/vmunix now relocatable ... rebooting"

is curious, because the kernel must be relocatable from the beginning!

I remember this happens to me while updating an old vPar version 2.xx to a newer version some years ago.

Sorry, I don't remember the details at the moment, but somehow it works.

[Torsten.]

This is what I remember. I know you are an expert and able to handle this ;-)

The vPar software makes the kernel relocatable, so it can be placed *anywhere* in the memory. A "normal" system will load the kernel in the beginning of the RAM.

The vparinit will check on every boot, if the kernel is relocatable. If it isn't, it will build a new kernel. (This is what happend).

See the script (taken from an old version):

"
...
# Make kernel file relocatable if it isn't already. We do this even
# if we are not running in a VPAR as long as the VPAR product has
# been installed. This functionality can be disabled by changing
# the value of DO_RELOC to 0 in the /etc/rc.config.d/vparinit file.

...

"

Just try to simply reboot and tell us about the results.

Did you try to update?

[Patrick Wallek · Accepted solution]

That a heck of a way to start a Friday morning, Pete.

The thing I find curious is this "!/sbin/sh: not found.". Do any of the scripts in /etc/rc.config.d have the '!/sbin/sh' as the first line? It should be '#!/sbin/sh'.

[Pete Randall]

Thank you all for your help, your concern, your sympathy. Torsten, in particular, calmed my fears about the new kernel being built (no wonder it takes a while to boot!!)

Patrick, however, hit the nail on the head. We had a rc.config.d script that had a sh-bang in it - without the sh. In other words, just like Patrick said, "!/bin/sh" rather than "#!/sbin/sh".

The curiosity about this, however, is the fact that the config script has been untouched since last Sept 13th (or 13 Sept, for you enlightened European folks). I even restored the script for back in November: same date stamp, same error. Since that September date, this machine has been re-booted successfully 6 times. I though maybe patches affected the shell behaviour but the last patches were applied on the 9th of January and involved a (successful) reboot. The machine was also re-booted (successfully) on the 31st of January.

Between the end of January and today, the only things I can think of were:

1) I ran up cleanup to free some /var space

2) I swremoved some unused software (apache, tomcat, mozilla, etc.) to free some /opt space.

Does anyone care to take a swing at that nail?


Pete