11i boot: auto_parms[1589]: /tmp/sh##.###:

[Alan Turner]

Has anyone seen (and possibly resolved) this problem with booting on 11i?
Initially all goes well, then after several reboots, I get the following error msgs
just after the msg saying DHCP access is disabled:

/sbin/auto_parms[1589]: /tmp/sh76.113: cannot create
/sbin/auto_parms[8]: who: not found
/sbin/auto_parms[17]: ps: not found
/sbin/auto_parms[17]: grep: not found

Line 1589 of auto_parms is where it sources /sbin/set_parms.util, but I didn't
see anything like /tmp/$$. in there.
Lines 8 and 17 of auto_parms do not make sense, but lines 8 and 17 of the
CheckRunLevel function in auto_parms do use who, ps and grep to try and
find out if we're running from boot. The output in /etc/auto_parms.log suggest
this check fails, because the log says execution was stopped.
If I look at /tmp in single user mode before the real /tmp filesystem is mounted
I can see 128 files of the form sh.. They come in blocks
of 4, looks like they hold messages. I can see /tmp/sh76.113 (and .114, .115,
.116).
Apart from the "cannot create" / "not found" messages, my system seems to come up OK. Also, if I clear the /tmp directory in the root FS, the problem goes away for several boots.
I tried a call to HP a while ago - but not resolved so i thought I'd try the foru

[Chris Wilshaw]

The errors below indicate a number of problems

sbin/auto_parms[1589]: /tmp/sh76.113: cannot create
/sbin/auto_parms[8]: who: not found
/sbin/auto_parms[17]: ps: not found
/sbin/auto_parms[17]: grep: not found


The cannont create error could be for one of 2 likely reasons;

1) There is no free space

2) There is a permissions problem with either the /tmp directory, or a file/directory called sh76.113 beneath it.

The "not found" errors show that the script that is running is trying to pick up these commands from a location (probably /usr/bin), which is not mounted as a filesystem when the script runs.

[Tim Adamson_1]

You will find it is caused by the fact that /usr has not been mounted yet. I bet the messages occur prior to the start sequence where all the /sbin/rc.?/* scripts are run.

However, I don't have a solution for you. You will need to determine which of the scripts in /sbin/init.d is causing the problem.

I will need a little more thought on this problem.


Tim

[Tim Adamson_1]

Had another idea. Check to make sure that the files in /sbin/rc?.d are all sym links to files in /sbin/init.d.

Next go through the scripts in /sbin/init.d and look for those commands - who, grep, etc.

This way you should identify the scripts that are causing the problems.

It looks as though the system is sourcing the files in /sbin/init.d before executing them and because /usr is not mounted the who and grep fail as they live in /usr/bin (and most likely / is mounted as read only at this stage of the boot process hence the /tmp files can't be created).

Now more thought is required. Just thinking out loud.


Tim.

[Alan Turner]

Chris / Tim -thanks for your responses.
There's plenty of space on the root filesystem, which is where the /tmp/sh76.113 file exists (there's also loads of free space in the /tmp
filesystem). Permissions on /tmp/sh76.113 allow root to have write. As I have
/tmp/sh76.1 --> /tmp/sh76.116 in the /tmp directory, what I suspect is
happening is a crude "create workfile" loop, which tries to make a new file
/tmp/sh$$. and keeps incrementing to look for a file
which does not already exist - baling out at sh76.113 looks like a safetry
measure to stop an endless loop (interestingly, when I raised the HP call, the error file was /tmp/sh81.113). Possibly there's a subsequent "tidy up" of
/tmp, which hypothetically fails because by now the filesystem has mounted
on /tmp.
As for "who", "ps" and "grep" - yes, I would expect these to be in /usr/bin.
Given that auto_parms normally works, what I think has happened is that
/usr/bin ought to have been monted by this stage, but the earlier failure may
have stopped the mount occurring, or alternatively, may have stopped /usr/bin
being added to PATH.
I suspect all would work fine if I didn't have a separate filesystem for /tmp.
I do have an ls and tar archive of /tmp on the root filesystem, but I'm not sure they'd be helpful

[Kent Ostby]

Tim is right about the /usr not being mounted as the cause.

To find the script giving the error, go into /etc/rc.config.d and do the following:

ll -t | head -20

to get a list of the most recently changed files in this directory.

There should be NO scripts in this directory at all.

Starting at the top of the list, type:

. ./filename

Any file that kicks out messages when you do this should not be in this directory.

This directory is used for files that should be SOURCED not RUN so if a script was put in here, it needs to be moved.

[Alan Turner]

Tim (response #2) - the rc#.d scripts are symbolic links to init.d scripts, but
the who/ps/grep errors are from auto_parms, and are before the rc.utils script runs and draws the tabular boot display - so I need to hit CTRL-S at just the
right time or I can't see the error messages. The root FS is writable at the
stage of the errors, on the basis that if i dleete the files from the /tmp
directory, they get created and the errors stop (till slots 113 - 116 are used up
in the hypothetical temp file name schem

[Tim Adamson_1]

Hi Alan,

I might not have been clear before. Apologies if that is the case.

I am trying to picture where in the boot process this is occuring. Obviously the kernel is loaded. The auto_parms message normally occurs just before where the system lists everything as OK or FAIL or N/A as it runs through the /sbin/rc?.d directories.

If I am right in that assumption, then the filesystems have NOT been mounted, hence /usr is not mounted.

Also, / is mounted read-only first and then re-mounted later on. The message about not being able to create the /tmp/sh##.### file could be caused by the fact that root is still mounted as read only at this stage of the boot process.


Hope I have been clearer.


Tim

[Alan Turner]

Kent - I'm pretty sure that the who, ps and grep messages are coming from
the CheckRunLevel shell function in /sbin/auto_parms - line 8 of this
function does a "who", and line 17 of the function pipes "ps" through "grep"
and "awk", looking to see if auto_parms is running under /sbin/rc. I don't
get a "not found" for awk, but awk is in /sbin.
rc/config.d just holds files to be sourced - anyway, the problem goes away
if I remove the contents of /tmp on the root flesystem, but this isn't a nice
solution because I can't do it once the system has booted, as I can't umount
/tmp

[Alan Turner]

Tim
The error occurs just after the message

"/sbin/auto_parms: DHCP access is disabled (see /etc/uato_parms.log)"

i.e. just before the OK / N/A / FAIL tabular display (if I hit CTRL-S to see the
error messages, then press CTRL-Q, I get an error message from /sbin/rc
saying it couldn't read row and column information, and I lose the paged
version of the boot table - comes up as a scrolled list.

I believe the root FS has been remounted read/write at the time of the errors,
because new files are created if I clear out /tmp, also the DHCP access msg
refers to /etc/auto_parms.log, suggesting the /etc/auto_parms.log was written

I haven't been able to check if /usr is mounted at this stage, and I'm wary of
tweaking auto_parms to look, but I actually would expect /usr to be mounted
at this stage, otherwise why do I NOT get the "not found" messages from
auto_parms when I've cleared out /tmp (and, why would auto-parms have
"who", "ps" and "grep" in it if /usr is not meant to be mounted when
auto_parms runs, yet autopamrs is meant only to be run by /sbin/rc)

[Tim Adamson_1]

Interestingly, auto_parms is actually called by /sbin/rc.

Just before it executes auto_parms, it runs /sbin/rc.utils which sources the files in /etc/rc.config.d if I remember correctly.

So the stuff in there must be correct but I'm not convinced that is the problem.

This is an interesting thread!!!


Tim

[Tim Adamson_1]

Will have to come back to this thread. Wish I could replicate it on a local system and wish I had more time to investigate. Unfortunately I have been called away.

Maybe someone will provide the solution.


Tim

[Alan Turner]

Tim

I'm not convinced that rc.utils sources the rc.config.d files - vi didn't find the
string "config" in rc.utils. rc.utils doesn't actually "do" anything - it defines a
set of shell functions, which could be used by other scripts. It's possible
that /sbin/rc will run source rc.utils as a general practice before running
other scripts.

I'm intrigued by what's trying to create /tmp/sh##.113 (as I've had the problem
with ## = 76 and ## = 81). from the line number, it's something in or under
set_parms.util, but I haven't followed this through. I believe it's this that's the
real trigger for the problem. Once the create error has occurred, further
errors could be spurious, e.g. perhaps set_parms.util, or something it calls,
ends up mounting /usr so that who, ps and grep will work and the failure
stops the mount working.

As a matter of interest, have you tried booting 11i with /tmp as a separate filesystem? [I haven't noticed any problems under 10.20)

[Kent Ostby]

From the /sbin/rc script:

if [ -f /etc/rc.config ]; then
. /etc/rc.config

[Kent Ostby]

Alan --

1) Have you taken a look at:

/etc/auto_parms.log
/etc/rc.log
/tmp/dhcpclient.log

Those may provide more details.

2) Do this command:

ll /sbin/rc2.d/S* | head

and see what scripts are below the S100 level when the filesystems are mounted.

If you see something new or especially something NOT LINKED then this could be the cause.

3) Another thing to check is to do a "mount -a" which will verify the number of the fields in /etc/fstab.

Best regards,

Kent M. Ostby

[Alan Turner]

I hadn't realised /sbin/rc was a script!!!!
The problems start after the . /sbin/auto_parms
so /etc/rc.config runs OK.

I've had a look at /sbin/set_parms.util, and
since the statement adding /usr/bin to the
PATH is the first non-definition ststement,
I'm wondering if the "cannot create" is being
raised when auto_parms attempts to source
set_parms.util, and is never managing to source
the script - otherwise, I'd expect the PATH to
be updated. If PATH isn't updated, even though
/usr should be mounted (MountLocalFilesystems
function in auto_parms) the PATH won't allow
the progs to be found.

So, back to why is "cannot create" being
reported? Does the shell create any temp files
while setting itself up to source a script?

[Alan Turner]

Kent (16:33 GMT post)

auto-parms.log should contain the DHCP disabled text, e.g.
Aug 11 12:03:03: DHCP is disabled for: lan0
but contains
Aug 26 17:11:19: Information: /sbin/auto_parms execution was stopped.
Aug 26 17:11:19: /sbin/auto_parms may only execute from /sbin/rc during
Aug 26 17:11:19: the initial transition from run level 'S' ( see init(1M) ).

(from the failure of CheckRunLevel)

/etc/rc.log is just "normal" - no adverse
inidcations, but then this all goes on before
the rc#.d scripts run
/tmp/dhcpclientlog doesn't exist (DHCP disabled)
There's nothing new in rc2.d, and anyway the
machine boots without error messages if I clear
out /tmp on root FS. mount -a reports no
formatproblems, and /etc/fstab pre-dates the last error-free boot by several days.

I'm wondering about a shell temporary file
(see my prvious reply).

[Tim Adamson_1]

Alan,

Apologies for my error in other post. I was meant to say the /sbin/rc calls /sbin/rc.utils and /etc/rc.config, the latter sources the stuff in /etc/rc.config.d.

Anyway, the auto_parms.log file indicates that it might be getting caught on some network related thing. You mention that dhcp is disabled, but that message is NOT in the log file.

Maybe you could put a set -x in the /sbin/auto_parms script. You would need to also put it in all the functions too. At least you will get an idea of where the hell it is in the script that it is hanging or looping.

My guess is a network thing but I have no proof.

Tim

[Tim Adamson_1]

Alan,

Ok. I did some more testing. I placed some set -x statements in auto_parms and it shows the following functions are called up to and including CheckRunLevel;

function LogToStdoutOff
function _LocalDebug
function MountLocalFileSystems
function CheckRunLevel

The first 2 don't do anything special.

Interesting to see the MountLocalFileSystems just before the CheckRunLevel. This could confirm the suspicions about /usr not being mounted.

I did another test to confirm that the filesystems were indeed mounted:

function MountLocalFileSystems
retStatus = 1
bdf output
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 163840 57923 99352 37% /
/dev/vg00/lvol1 119637 47518 60155 44% /stand
/dev/vg00/lvol8 2048000 1790149 242386 88% /var
/dev/vg00/lvol7 1024000 581961 414416 58% /usr
/dev/vg00/lvol6 102400 60528 39531 60% /tmp
/dev/vg02/lvol1 4190208 2757689 1352983 67% /tim2
/dev/vg00/lvol5 1024000 886993 133622 87% /opt
/dev/vg00/lvol4 102400 4031 92247 4% /home
function CheckRunLevel

As you can see, my filesystems are mounted fine!

As you correctly pointed out, the CheckRunLevel function issues the who and grep commands:

ppid=$(ps -ef | grep 'bin/r[c]' | awk '{print $2}')

Based on what you are seeing, I would say that the MountLocalFileSystems function may be failing.

I would concentrate on that area. Put in a set -x or redirect the output from some commands to $AUTO_PARMS_LOGFILE.

bdf >> $AUTO_PARMS_LOGFILE 2>&1 (best to make sure you include stderr because if /usr is NOT mounted, then bdf will fail as it lives in /usr/bin).

Hope that helps.


Tim.

[Tim Adamson_1]

Ha. Another reply.

When looking at the return status for retStatus it is 1 which is TRUE. Just to confuse things more :)~

export SUCCESS=1
export FAILURE=0

function MountLocalFileSystems {
typeset retStatus

mountall -l >/dev/null 2>&1

if [[ $? -ne 0 || ! -d /usr/lbin ]]; then
retStatus=$FAILURE
else
retStatus=$SUCCESS
fi
return $retStatus

}


Tim.

[Alan Turner]

Tim (re several overnight posts)

Thanks for the midnight oil. I think we're
homing in on 2 different areas as the cause
of the "not found" messages:
a) MountLocalFilesystems failing, OR
b) PATH not getting set to include /usr/bin
These are secondary to the first problem, which
is the "cannot create" message, but may help us
to track down what's going on.
Your posts have given me the courage to have
a go at editing /sbin/auto_parms. I hope to
have a go at it this afternoon (BST) - like
you, I sometimes have to do other bits of my
day job.

regards

Alan

[Tim Adamson_1]

Alan,

Done some more testing. This time I can say that a whole heap of stuff is run before the filesystems are mounted. :-0

I didn't get any file with the exact syntax you describe (/tmp/sh##.###). The closest I got was /tmp/##.sh. And this is done in the ParseLanDefinitions function.

Assuming this is the correct function, it does the following:

function ParseLanDefinitions {
typeset fileName=$1

if [[ ! -f $1 ]]; then
echo "Panic: file does not exist, exit"
exit 1
fi

rm -rf /tmp/$$.sh # Avoid security issue.
cat $1 | grep "INTERFACE_NAME\[" >/tmp/$$.sh

. /tmp/$$.sh

rm -f /tmp/$$.sh

return 1
}

fileName=$1 just happened to be /etc/rc.config.d/netconf.

So /etc/rc.config.d/netconf is checked to make sure it exists and is a regular file.

Any old /tmp/$$.sh files are removed.

Finally "INTERFACE_NAME\[" is grabbed from /etc/rc.config.d/netconf and stored in /tmp/$$.sh and then it is sourced.

In other words it sets up some variable definitions for your lan devices. For example:

INTERFACE_NAME[0]="lan0"
INTERFACE_NAME[1]="lan2"
INTERFACE_NAME[2]=lan1

Phew. So, is your /etc/rc.config.d/netconf file valid? What does cat /etc/rc.config.d/netconf | grep "INTERFACE_NAME\[" return?

The only thing that concerns me about all this is the name of the /tmp file. Perhaps we are running a different /sbin/auto_parms script.

# cksum auto_parms
1288082985 44375 auto_parms

# what auto_parms
auto_parms:
RCS: $Header: /ignite/ii/src/UX-CORE/src/auto_parms/auto_parms 10.48 2000/08/16 22:10:47 paulc Exp $


Knowing my luck, they will be different.

I think I mentioned in an earlier post that the problem could be network related.

Will wait for your test results and/or confirmation we are dealing with the same file.


Cheers.


Tim.

[Alan Turner]

Tim

Thanlks for this - unfortunately, I'll have to suspend investigation for a few days as we've got to ship the first set of the boxes to site (I'll clear out /tmp on root FS to give a reasonable grace period).
One thing I keep coming back to is that, since the problem goes away if I clear /tmp, it's unlikely to be config file corruption because how would that magically fix itself when I deletdelete /tmp files?
Anyway, I've just completed a copyutil of the system disk, (isn't copyutil SLOW) so I've got the starting point for further investigation when site installation completes.

[Tim Adamson_1]

In the initial post you mentioned 11i. Copyutil is ancient. Install Ignite-UX and create a make_tape_recovery tape for recovery purposes.

Assuming you are running 11i and have the latest version of Ignite-UX:

# make_tape_recovery -a /dev/rmt/0mn -x inc_entire=vg00

Refer to the make_tape_recovery man page for further details.

If you are running a different version of the OS, what version is it? Can you post the cksum of auto_parms?


Cheers.


Tim

[Alan Turner]

cksum of auto_parms matches yours (also the RCS header).
I ought to investigate ignite, but never seem to have the time.