topic Re: Hung Process in Operating System - HP-UX

Hung Process

Brian Campbell_3 — Tue, 02 Sep 2003 13:47:15 GMT

Encountering a problem due to a hung process, associated with a daily file cleanup job, which simply removes (rm) files from our file content directory. This job has run nightly, past 1+ year with no problems; however, last couple of days, the job kicks of a delete process which seems to hang on the rm command that ultimately causes sluggish performance for the remainder of the application that runs on the same box ??? which is fighting for resources locked by the hung process.

Our tech staff has been unsuccessful with tracing the root cause of the problem - only upon restarting the box and performing the magical kill -9 command with the root access, does the problem seem to go away - until the next evenings clean-up process kick-off.

I am unable to kill the process myself.
Unable to truss the problem.
Can???t seem to get the UNIX admin staff to do anything other than cold startup and kill command - only fixes the symptom and not the problem.

Any advice?
Any sort of fix disk command or known hardware issue that I can recommend to our UNIX gurus look into?

Re: Hung Process

Stanimir — Tue, 02 Sep 2003 14:02:42 GMT

Please show as the full row on your crontab.
You may need to caught the error-output of
this row. Try to do "rm" without cron, by
command row. Did you see any message?

Re: Hung Process

Kevin O'Donovan — Tue, 02 Sep 2003 14:05:10 GMT

Hi Brian,

sounds like something has changed on the box...

* Are all the filesystems that the rm script deletes stuff from local? Or are any NFS? Check whether there are any stale nfs file handles (bdf command). Maybe its just tripping up over one of those.

* Is it just a simple rm in the script or does it do anything fancier? In case its not a problem with the rm command in the script, maybe with one of the other commands?

* Can you run the script manually, or even better (depending on the script) run each line from the script manually? That should show you exactly where in the script its having problems and you might even get an error message if you're lucky.

* if you can't run the script manually, does it log its output into a file? Or can you get it to send its output to a logfile?

hope that helps,
Kevin.

Re: Hung Process

A. Clay Stephenson — Tue, 02 Sep 2003 14:08:26 GMT

My best at this point is a corrupt filesystem; have your UNIX guys run a full fsck (not simply a log replay) on the filesystem in question. You might have a circularly linked directory structure which could cause exactly this kind of problem.

I am making the assumption that this is a local filesystem rather than NFS/automount.

Re: Hung Process

Bernhard Mueller — Tue, 02 Sep 2003 14:19:39 GMT

Hi,

I would take a close look at your "file content directory". If you have not changed anything then there is probably something in it which causes problems for rm to continue, this could be something like a subdirectory consisting of non-printing characters only or a file name containing semicolon, circular soft links or whatever weird can happen.

Ideally you would create a new "file content directory" and copy over ONLY what you need, carefully examining each file and directory.

As a preparation I would cd into it and do
# find . > /tmp/check_names_list
# find -type l | xargs ll > /tmp/check_link_list

Go through that list to rule out things I mentioned above.

Regards,
Bernhard

Re: Hung Process

Scot Bean — Wed, 03 Sep 2003 15:52:08 GMT

Check /var/adm/syslog/syslog.log for any local hard drive errors.

Run online diagnostics on local hard drives.

Re: Hung Process

Krishna Prasad — Wed, 03 Sep 2003 19:10:49 GMT

You can also try to do a

sh -x "script name" from the command line.

This should show you where it is hanging.