Cron spawning multiple processes

[Adelheid]

Currently I have a few KSH scripts running on our server and have noticed that it is spawning multiple times. I was first alerted to it when one of the monitoring scripts sent out 3 emails instead of 1. The script also logs to a file, to which it had also written 3 lines. This behavior has only recently started and no scripts or executables have been changed.

I created a very simple ksh script which just writes the current unix time to a file to see if I could eliminate the script itself, however the same problem occurs.

From cron we are also calling a binary executable which reorganises a database. Again, multiple processes are spawned.

I have been seeing this behavior by performing a ps -ef|grep <scriptname> a few times around the time that the crontab starts. Here is an example of the output:

r0_adms 26363 2176 1 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1
r0_adms 26540 26367 1 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1
r0_adms 26542 26540 0 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1
r0_adms 26367 26363 1 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1

In this case set_dmsenv.sh simply adds the necessary paths and variables for our DMS environment.

We are running HPUX B.11.23 U ia64.

Has anyone experienced this? If so is this expected behavior and does anyone know how to prevent it?

Thanks in advance

[Patrick Wallek]

>>If so is this expected behavior

I wouldn't think so!

How many 'cron' processes do you have?

What does 'UNIX95= ps -C cron' show? (Note the space between the = and the 'ps')

If there is more than 1 then that may be your problem.

You could try stopping a restarting cron.

# /sbin/init.d cron stop

Make sure there are no 'cron' processes running.

# /sbin/init.d/cron start

[Adelheid]

Hi,
Thank you for your response. I tried this but only see one cron process:

UNIX95= ps -C cron
PID TTY TIME CMD
2716 ? 01:34 cron

Kind regards

[Patrick Wallek]

You still might try stopping and restarting the cron daemon. If the problem persists you should open a support call with HP.

[Dennis Handly]

>multiple processes are spawned.

Not by cron(1m). These are all done by the shell. You need to get a hierarchical dump of the process tree:
UNIX95=EXTENDED_PS ps -H -fu r0_adms
r0_adms 26363 2176 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)
r0_adms 26367 26363 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)
r0_adms 26540 26367 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)
r0_adms 26542 26540 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)

It looks like the shell is forking itself.

>In this case set_dmsenv.sh simply adds the necessary paths and variables

I assume this completes and you do get to tb_monitor.sh?

[Adelheid]

Hi,
I created a small file which simply writes the current unix time to a file and called that, here is the output from it. What I don´t understand here is the awk line. There is no awk scripting in the set_dmsenv.sh file at all, nor in my example script. As I say, this file does nothing apart from setup environment variables.

r0_pdms 19792 2716 0 09:10:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; ~/rh/test.sh)
r0_pdms 19794 19792 2 09:10:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; ~/rh/test.sh)
r0_pdms 19953 19794 0 09:10:00 ? 00:00 /usr/bin/awk {for (i=1; i<= NF; ++i) if( $i != VAR ) print $i":"} FS=: VAR=/opt/IBM/db2ia32/sqllib/java/db2jcc.jar
r0_pdms 19954 19953 0 09:10:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; ~/rh/test.sh)
r0_pdms 19955 19954 0 09:10:00 ? 00:00 /usr/bin/echo /opt/IBM/db2ia32/sqllib/java/db2java.zip

Here is my test script:
#!/bin/ksh
epoch=`date "+%s"`
echo $epoch>>test.out

Although this script appears to spawn 3 times using my previous method I do only get one line of output in the test.out file.

[Adelheid]

Furthermore, I also have an executable which is producing the following output:

r0_pdms 23173 2716 1 10:05:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; mdmsadminclient r) >>$HOME/cron_log/cron.log 2>&1
r0_pdms 23180 23173 2 10:05:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; mdmsadminclient r) >>$HOME/cron_log/cron.log 2>&1
r0_pdms 23364 23180 0 10:05:00 ? 00:00 /usr/bin/sed s/\: /\:/g
r0_pdms 23368 23364 0 10:05:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; mdmsadminclient r) >>$HOME/cron_log/cron.log 2>&1

[Dennis Handly]

>What I don't understand here is the awk line. There is no awk scripting in the set_dmsenv.sh file at all, nor in my example script. As I say, this file does nothing apart from setup environment variables.

Do you export ENV to set up a .kshrc file?
What's in set_dmsenv.sh?

You may want to add "set -x" to your script AND to set_dmsenv.sh.

[Adelheid]

I made a copy of the set_dmsenv.sh for testing...

I took out the scripts altogether and just call the set_dmsenvtest.sh script:

UID PID PPID C STIME TTY TIME CMD
r0_pdms 20840 2716 1 11:43:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 20857 20840 2 11:43:01 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 21041 20857 0 11:43:01 ? 00:00 /usr/bin/awk {for (i=1; i<= NF; ++i) if( $i != VAR ) print $i":"} FS=: VAR=.
r0_pdms 21043 21041 0 11:43:01 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 21045 21043 0 11:43:01 ? 00:00 /usr/bin/echo /opt/IBM/db2ia32/sqllib/java/db2java.zip:/opt/IBM/db2ia32/sqllib/java/db2jcc.jar:/opt/IBM/db2ia32/sqllib/java/db2


I have attached the set_dmsenvtest.sh file. In the .environ file we have the following:

set -o trackall
alias history='fc -l -20'
alias h='fc -l -20'
alias dir='ls -la |more -cs '
alias -t
alias um='set -- -1;. um'
alias psu='ps -fu $LOGNAME|grep -v "ps -fu"'
alias lb='cd ${MBASE}/locbin;x'
alias sm='cd ${MBASE}/seriem;x'
alias mb='cd ${MBASE};x'

Thanks for your help so far.

[Dennis Handly]

>I took out the scripts altogether and just call the set_dmsenvtest.sh script:

It seems it is bad.

>In the .environ file we have the following:

That seems safe.

What's in ${DB2HOME}/sqllib/db2profile?

>HostName=*********

This isn't going to work, change to:
HostName="*********"
(Unless the censor got to it. :-)

You have "set +x" at the top. You need "set -x" to trace.

[Adelheid]

Hi Dennis,
I commented out the hostnames :-)

I have changed the + to a - and let it run again, I am not sure what I am looking for in terms of a trace, or how to get more info. Anyhow, this is the output from the ps:

r0_pdms 22869 2716 0 11:45:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 22871 22869 1 11:45:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 22872 22871 1 11:45:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 22874 22872 0 11:45:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 22875 22874 0 11:45:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)

Thanks

[Adelheid]

ah, found it....uploaded :-)

[Dennis Handly]

>found it....uploaded :-)

Where? /var/spool/cron/tmp/? Or $HOME/cron_log/cron.log?

You seemed to get to ${DB2HOME}/sqllib/db2profile & /opt/IBM/db2ia32/sqllib/userprofile

It might help to have "set -x" in the latter.

>export DB2CODEPAGE=1051 shows you got back.

>HARES=/opt/VRTSvcs/bin/hares

It seems you got to the end of the file.

There seems to be something broken with the shell. You might try making sure it and cron hasn't be corrupted:
swverify OS-Core.UX2-CORE OS-Core.CMDS2-MIN

Or you might want to use tusc on cron to see why those shells get forked.
tusc -fp -ea -o tusc.out 2716 #cron-PID

You can kill tusc after you see another bunch of nested shells hung.

I've had cases with the scummy C shell where it hangs. You could try redirecting stdin to /dev/null:
(. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1 < /dev/null