topic Re: ksh script error "no space" in Operating System - HP-UX

ksh script error "no space"

lorenzo bosisio — Wed, 02 Jul 2008 13:26:17 GMT

Hi everybody,

I have a cluster monitor script which runs every two minutes ( a while true loop with sleep 120 ).

A couple of times, I said erratically, it exits with error. This happened always near 2 o'clock in the morning.

In SG logfile I found the following lines:
/etc/cmcluster/syslog-db/db-monitor-script.sh[18]: no space
/etc/cmcluster/syslog-db/db-monitor-script.sh[18]: no space
/etc/cmcluster/syslog-db/db-monitor-script.sh[18]: no space
/etc/cmcluster/syslog-db/db-monitor-script.sh[20]: test: argument expected
/etc/cmcluster/syslog-db/db-monitor-script.sh[5]: no space
/etc/cmcluster/syslog-db/db-monitor-script.sh[8]: no space

We work on HP-UX 11.23 ia64.

What does this message mean?

The full script is:
#!/usr/bin/ksh

while true
do
sleep 120
> /tmp/monitor.out

. /home/oracle/env_oracle10
$ORACLE_HOME/bin/sqlplus /nolog < connect / as sysdba
set heading off;
spool /tmp/monitor.out
select status from V\$instance;
spool off
exit
EOF

CHECK=$(cat /tmp/monitor.out|grep OPEN |wc -l)

if [ $CHECK -ne 1 ]
then
exit 1
fi

CHECK2=$(ps -ef |grep LISTENER10 |grep -v grep |wc -l)

if [ $CHECK2 -ne 1 ]
then
exit 1
fi

done
exit 0

Many thanks,
Lorenzo

Re: ksh script error "no space"

James R. Ferguson — Wed, 02 Jul 2008 13:33:48 GMT

Hi Lorenzo:

It appears that your '/tmp' filesystem is full (out of space) during the periods you see the error.

In addition to your 'montor.out' file, HERE documents are files in '/tmp'.

You should be able to see the filesystem space errors in '/var/adm/syslog/syslog.log'.

Regards!

...JRF...

Re: ksh script error "no space"

Oviwan — Wed, 02 Jul 2008 13:35:50 GMT

Hey

-Run the script in debug mode
#!/usr/bin/ksh -x

-check whether /tmp is full

-check whether the instance is running by grep the pmon process

$(ps -ef | grep -v grep | grep -c ora_pmon_${ORACLE_SID})

instead of count the lines with wc -l you can use grep -c

Regards

Re: ksh script error "no space"

lorenzo bosisio — Wed, 02 Jul 2008 13:57:17 GMT

Hi,

It happened just a couple of times and always at 2 o'clock in the morning.

During daytime, /tmp is round 10% full.

I supposed it has something to do with space on file system, but I cannot figure out what can full the /tmp.
In the /var/adm/syslog/syslog.log there NO entry regarding file system full.

Any other suggestion?

Re: ksh script error "no space"

Dennis Handly — Wed, 02 Jul 2008 14:22:39 GMT

>Any other suggestion?

We told you, /tmp is full. Fix that. Also you can do "bdf /tmp" to see how full it is.

In general you should be using /var/tmp/ instead.

Re: ksh script error "no space"

Oviwan — Wed, 02 Jul 2008 14:32:33 GMT

so monitor /tmp during the night with a simple script:

while true ; do
if [ $(bdf | grep /tmp |awk '{sub(/%/,"",$5); print $5}') -gt 90 ] ; then
date >> output.log
bdf | grep /tmp >> output.log
ll /tmp >> output.log
fi
sleep 300
done

but output.log should not be in /tmp ;)
you can start the script with rman for example at 00:00

Re: ksh script error "no space"

James R. Ferguson — Wed, 02 Jul 2008 14:33:37 GMT

Hi Lorenzo:

Don't be fooled if you don't find "visable" files in the filesystem. It is a common paradigm to create a temporary file and immmediately unlink (remove) it. This means that you won't see the file in an 'ls' listing but the space will be taken until the last process using the file terminates and the file's link count drops to zero.

Regards!

...JRF...

Re: ksh script error "no space"

Peter Nikitka — Wed, 02 Jul 2008 16:56:58 GMT

Hi,

generally the shell MAY run out of space without any filesystem being full: Just try something like
variable=$(cat HUGEFILE) or simply
variable=$(
If I counted correctly, line 18 () of your script is that:
CHECK=$(cat /tmp/monitor.out|grep OPEN |wc -l)

I consider this as an overkill, an suggest to use
CHECK=$(grep -c OPEN /tmp/monitor.out)
instead. The assignment to CHECK2 can be made much simpler as well - read the XPG4 section of 'man ps'.

mfG Peter