Cron spawning multiple processes

[Adelheid]

Currently I have a few KSH scripts running on our server and have noticed that it is spawning multiple times. I was first alerted to it when one of the monitoring scripts sent out 3 emails instead of 1. The script also logs to a file, to which it had also written 3 lines. This behavior has only recently started and no scripts or executables have been changed.

I created a very simple ksh script which just writes the current unix time to a file to see if I could eliminate the script itself, however the same problem occurs.

From cron we are also calling a binary executable which reorganises a database. Again, multiple processes are spawned.

I have been seeing this behavior by performing a ps -ef|grep <scriptname> a few times around the time that the crontab starts. Here is an example of the output:

r0_adms 26363 2176 1 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1
r0_adms 26540 26367 1 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1
r0_adms 26542 26540 0 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1
r0_adms 26367 26363 1 14:17:00 ? 0:00 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh) >> $HOME/cron_log/cron.log 2>&1

In this case set_dmsenv.sh simply adds the necessary paths and variables for our DMS environment.

We are running HPUX B.11.23 U ia64.

Has anyone experienced this? If so is this expected behavior and does anyone know how to prevent it?

Thanks in advance

[Patrick Wallek]

>>If so is this expected behavior

I wouldn't think so!

How many 'cron' processes do you have?

What does 'UNIX95= ps -C cron' show? (Note the space between the = and the 'ps')

If there is more than 1 then that may be your problem.

You could try stopping a restarting cron.

# /sbin/init.d cron stop

Make sure there are no 'cron' processes running.

# /sbin/init.d/cron start

[Adelheid]

Hi,
Thank you for your response. I tried this but only see one cron process:

UNIX95= ps -C cron
PID TTY TIME CMD
2716 ? 01:34 cron

Kind regards

[Patrick Wallek]

You still might try stopping and restarting the cron daemon. If the problem persists you should open a support call with HP.

[Dennis Handly]

>multiple processes are spawned.

Not by cron(1m). These are all done by the shell. You need to get a hierarchical dump of the process tree:
UNIX95=EXTENDED_PS ps -H -fu r0_adms
r0_adms 26363 2176 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)
r0_adms 26367 26363 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)
r0_adms 26540 26367 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)
r0_adms 26542 26540 sh -c (. $HOME/set_dmsenv.sh; tb_monitor.sh)

It looks like the shell is forking itself.

>In this case set_dmsenv.sh simply adds the necessary paths and variables

I assume this completes and you do get to tb_monitor.sh?

[Adelheid]

Hi,
I created a small file which simply writes the current unix time to a file and called that, here is the output from it. What I don´t understand here is the awk line. There is no awk scripting in the set_dmsenv.sh file at all, nor in my example script. As I say, this file does nothing apart from setup environment variables.

r0_pdms 19792 2716 0 09:10:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; ~/rh/test.sh)
r0_pdms 19794 19792 2 09:10:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; ~/rh/test.sh)
r0_pdms 19953 19794 0 09:10:00 ? 00:00 /usr/bin/awk {for (i=1; i<= NF; ++i) if( $i != VAR ) print $i":"} FS=: VAR=/opt/IBM/db2ia32/sqllib/java/db2jcc.jar
r0_pdms 19954 19953 0 09:10:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; ~/rh/test.sh)
r0_pdms 19955 19954 0 09:10:00 ? 00:00 /usr/bin/echo /opt/IBM/db2ia32/sqllib/java/db2java.zip

Here is my test script:
#!/bin/ksh
epoch=`date "+%s"`
echo $epoch>>test.out

Although this script appears to spawn 3 times using my previous method I do only get one line of output in the test.out file.

[Adelheid]

Furthermore, I also have an executable which is producing the following output:

r0_pdms 23173 2716 1 10:05:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; mdmsadminclient r) >>$HOME/cron_log/cron.log 2>&1
r0_pdms 23180 23173 2 10:05:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; mdmsadminclient r) >>$HOME/cron_log/cron.log 2>&1
r0_pdms 23364 23180 0 10:05:00 ? 00:00 /usr/bin/sed s/\: /\:/g
r0_pdms 23368 23364 0 10:05:00 ? 00:00 sh -c (. $HOME/set_dmsenv.sh; mdmsadminclient r) >>$HOME/cron_log/cron.log 2>&1

[Dennis Handly]

>What I don't understand here is the awk line. There is no awk scripting in the set_dmsenv.sh file at all, nor in my example script. As I say, this file does nothing apart from setup environment variables.

Do you export ENV to set up a .kshrc file?
What's in set_dmsenv.sh?

You may want to add "set -x" to your script AND to set_dmsenv.sh.

[Adelheid]

I made a copy of the set_dmsenv.sh for testing...

I took out the scripts altogether and just call the set_dmsenvtest.sh script:

UID PID PPID C STIME TTY TIME CMD
r0_pdms 20840 2716 1 11:43:00 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 20857 20840 2 11:43:01 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 21041 20857 0 11:43:01 ? 00:00 /usr/bin/awk {for (i=1; i<= NF; ++i) if( $i != VAR ) print $i":"} FS=: VAR=.
r0_pdms 21043 21041 0 11:43:01 ? 00:00 sh -c (. $HOME/set_dmsenvtest.sh)
r0_pdms 21045 21043 0 11:43:01 ? 00:00 /usr/bin/echo /opt/IBM/db2ia32/sqllib/java/db2java.zip:/opt/IBM/db2ia32/sqllib/java/db2jcc.jar:/opt/IBM/db2ia32/sqllib/java/db2


I have attached the set_dmsenvtest.sh file. In the .environ file we have the following:

set -o trackall
alias history='fc -l -20'
alias h='fc -l -20'
alias dir='ls -la |more -cs '
alias -t
alias um='set -- -1;. um'
alias psu='ps -fu $LOGNAME|grep -v "ps -fu"'
alias lb='cd ${MBASE}/locbin;x'
alias sm='cd ${MBASE}/seriem;x'
alias mb='cd ${MBASE};x'

Thanks for your help so far.

[Dennis Handly]

>I took out the scripts altogether and just call the set_dmsenvtest.sh script:

It seems it is bad.

>In the .environ file we have the following:

That seems safe.

What's in ${DB2HOME}/sqllib/db2profile?

>HostName=*********

This isn't going to work, change to:
HostName="*********"
(Unless the censor got to it. :-)

You have "set +x" at the top. You need "set -x" to trace.