MC/SG pkg control.log filling filesystem up with "sh: sleep: not found." messages

[John Kittel]

I just rebooted my cluster, which of course restarted my MC/SG packages. Now my root filesystem is filling up; the package control.log file is rapidly filling up with "sh: sleep: not found." messages.

The sleep command stil exists in /usr/bin and still works fine from the command line. The /etc/PATH hasn't changed. I didn't change anything to do with the package prior to the reboot.

For now I'm able to prevent / from filling totally up by periodically truncating the package control.log file.

I don't see how this could be a cause, but I did change some aspects of the root account, such as I created a /root directory, moved all root's setup files from / to /root, set root's account to use that as the home dir, created /etc/default/security SU_ROOT_GROUP parameter, modified root's .profile in /root to use/create different shell history files depending on who su'd to root. I did that a week or two ago, and hadn't seen any problems. I also did the same things on the other member of this SG cluster and it's package isn't having any trouble with the sleep command.

any help certainly appreciated!!

- John Kittel

[Karthik S S]

Hi John,

Check the PATH variable in the package control file. If that does not contain /usr/bin then include the same. Or replace the sleep command with its full path /usr/bin/sleep in the package control file. Then restart the package.

Alternatively you can halt the package. Move the package log to a different file system and create a softlink.

-Karthik S S

[John Kittel]

The PATH variable in the control file hasn't changed in forever. It does have /usr/bin in it. I did think of putting the full path to sleep in the control file, but I am trying to avoid restarting the package just now; our whole plant rather depends on it. I may have to resort to that though just so I can go get some sleep. I have to believe putting the full path in woould fix it, but I'm very interested in learning the cause of this.

- John

[Shaikh Imran]

Hi,
Assuming that you did no change to your packages control script or any other changes.
You can check whether your sleep command works properly from the command prompt by a simple while true loop
while true
do
date
sleep 10
done
Press Control+C to come out of the loop
Also Please check for any BAckgroud or cronjobs which you may have scheduled earlier
and the parent process of that job might have started and still running.,

Regards,

[John Kittel]

The sleep command works fine (see original post).

I'm not sure how to check for background or cron jobs that may have been started earlier...??

- John

[John Kittel]

ok, just discovered a major clue.

I just did a cmcheckconf. It says

cmcheckconf : Error found in package file: apps01.control.
Error: Unknown keyword (PATH=/USR/BIN:/USR/SBIN:/ETC:/BIN) on line 1


But when I look at apps01.control with vi, that whole string is in lowercase, like this:

root cmcluster/pkg # grep PATH apps01.control
# Set PATH to reference the appropriate directories.
PATH=/usr/bin:/usr/sbin:/etc:/bin


what's up with that??

- John

[Karthik S S]

John,

Do a "grep PATH apps01.control" on the other node. I think the package files are not consistent across the nodes.

-KarthiK S S

[John Kittel]

They're the same on both nodes.

root cmcluster/pkg # grep PATH apps01.control
# Set PATH to reference the appropriate directories.
PATH=/usr/bin:/usr/sbin:/etc:/bin

Just to make extra sure, I just ftp'd a copy from the node with the problem to /tmp on the node that isn't running the package and then did a diff on them.

They do have different dates on them though. Not sure why. The one on the node that isn't running the package is newer. I normally (always) make changes on the node that normally runs /owns the package, then copy it to the other node.

- John

[Karthik S S]

John,

Date stamp should not create any problems as far as the contents are the same. Can you post apps01.control file here?? Atleast the first few lines as the cmcheckconf is reporting error on line 1.

KarthiK S S

[John Kittel]

apps01.control attached

- John

[Karthik S S]

John,

I am sorry .. Just now it striked my mind ... I think you are doing a cmcheckconf on the package control file. You should be doing it on the package ascii file instead.

This error was expected.

-KarthiK S S

[Karthik S S]

Also try doing a,

cmgetconf [-v] [-c cluster_name] [-p package_name] for the packge in question.

-KarthiK S S

[John Kittel]

oops.

my bad.

I did cmcheckconf on the control file, not the config file. chcheckconf on the config file says, "not errors".

Back to the drawing board...

- John

[Karthik S S]

Did you try cmgetconf -p packagename??

-Karthik S S

[John Kittel]

sorry for delay in responding, I have HP tech support on phone...

result of cmgetconf attached.

- John

[Karthik S S]

Oh that is great.. so you can have your problem fixed soon. By the way the output you have posted is that of the cluster and not the package. Try doing a,

cmgetconf -v -p pkg_name

If you dont get any errors you need not post it back here.

-KarthiK S S

[John Kittel]

I have been awake too many hours.

No errors on cmgetconf on pkg.

Tech support seems to be temporarily stumped...

- John

[Karthik S S]

John,

Any updates???

-KarthiK S S

[John Kittel]

... still no solution. Still working with tech support.

- John

[John Kittel]

update...

Several hours on the phone last night with HP support and 3 or so more hours today and still no solution. No good ideas what could be the cause. I'm scheduling an outage so that I can restart the packages, and/or reboot the systems, and just hope it was a one-time glitch that won't happen again.

- John

[John Kittel]

OK, it is fixed now.

Problem was a sleep command without the path specified in a program started by a service. The service monitor script starts the program with an 'su username -c "command" ', where command is a shell script that starts the program. And the program is wriiten in a langauge called DML ( data manipulation language), and it was using the DML CLI statement to call the Unix sleep command, without specifying the path. The DML program had been like that all along, but the change that caused this to surface now was that I had changed the monitor script to do "su username", rather than " su - username".

- John