Unknown processes running at night

[Randy Lowe]

I have a HP 9000 K570 running
HP-UX 11.11 32 bit OS. We are having problems with an unknown process taking up resources on certain nights of the week. These process or processes are causing our Omniback nightly backups on this server to run longer than normal and this is causing the backups scheduled
to follow these backups to fail because the devices are still occupied by the latter
backup.How can I monitor the server and find out what process or even better what user is causing all these problems on my server? I need to do this at night and send this to a file or a log so I can print this out and show the print out to my supervisor. Any brilliant ideas are welcome!

[Steven E. Protter]

You can run some performance collection scripts, which I am attaching.

These will allow you to pinpoint the time and load factors. Then you can run this to a file and see whats going on.

ps -ef >> /tmp/mystery.process

with cron.

I'd refine it a bit, but you should be able to identify it.

Also, just go through crontab -l and see what is scheduled to run when. Maybe somene added a little cron script.

It only takes 4-6 find commands to bring a system to its knees.

SEP

[Pete Randall]

Randy,

You could look at setting up a cron'd "top" job that would run every 5 or 10 minutes. Use the top -f option, appending a time stamp to the name of the output file, then review your results in the morning.


Pete

[Michael Steele_2]

Use 'lsof' to see what open files are being used:

http://hpux.cict.fr/hppd/hpux/Sysadmin/lsof-4.55/

lsof -p pid

[John Poff]

Hi,

If you have Measureware installed on the box you can use it to see what is happening. If not, you can write a little script that will do a 'ps -ef' every five or ten minutes and write that to a file. That should be a start at least. You can also use sar if you have it configured to run.

Is this a problem that just came up recently? If so, has anything changed on the box? Any new applications loaded? Take a look at all the crontab files [/var/spool/cron/crontabs] and see if anybody is running something in the middle of the night.

JP

[Ramkumar Devanathan]

Have a cron job to run every 1/2 hr from 20hrs to 12 midnight and thereafter from 00 hrs to 08 hrs next day -

top -f /tmp/top ; cat /tmp/top >> /tmp/log

The next morning, you would definitely be able to find any new processes that ran during the night consuming too much resources from the top utility output. the good thing is that the timestamps are written out too.

- ramd.

[Ramkumar Devanathan]

Hey Pete... we think alike...

- ramd.

[Tim Sanko]

First check that your cron jobs that are running are what you expect in /var/adm/cron/log

I would set up a cron Job also with
ps -ef >>/tmp/pslist ; echo "delimiter" >> /tmp/pslist.

I would schedule top to run at an atternating

I would have it run at 3,8,13,18,23 and some other 2,4,6,8 pattern.


I would schedule top to run at an atternating timeslot as in 2,7,12, etc.

To see if anything is submitting itself use at -l to list the at jobs, and examine your crontabs. I would look at at -l as that is the first spot to look.

If root is not performing maintenance routines at night.

If you want to eliminate the problem, I would fill in the
at.allow and examine all user crontab files.

Tim

[Tim Sanko]

Dang, I write slower than you guys, plus I am fighting budget wars...

[Sachin Patel]

Hi Tim,
If you win that pass it up some $$... heheh

Hi Randy,
What if your backup is large that is not finishing in 24 hour window.

I had a same problem with my rdist. Now I have put chechking in script to see if any rdisk is running or not if yes send email and do not run script.

Sachin