hpux 11.31 stops responding to requests periodically

[Craig Sterley]

Have a hpux 11.31 itanium blade, everything is patched to the latest and greatest, it is used as an application server connected to an identical system running the database (Oracle). Once or maybe twice a week this box will become completely unresponsive to user connections for a minute or two - 5 minutes is the longest we've saw. It wont respond to ssh, telnet, rexec or anything else we've tried. I've been watching the inetd process and the PID never changes. Have several hundred people that connect to this box on a daily basis and it's starting to get a bit annoying. Not sure where to go with this - hoping the patch sets put on last month would have taken care of the issue.

Any ideas out there of where to go next?

[Tim Nelson]

collect performance stats and review for those times when it seems to be unresponsive.

first check on mem usage

[Jayakrishnan G Naik]

Hi Sterley,

You have to run various commands to see what is the real bottleneck here, It can be a low cpu, or a too high disk io or lack of enough memory (too much usage of swap), --system may be running out of resource perticularly during the event. The high network traffic & denial of service attacks are chances but may be rare.

Please check whether your issue is having a perticular pattern. Say if any periodic backup is running when usage is high and if that is causing issue, then you can change the backup schedule to a time when the usage is very low.

You can check the logs & dmesg to see the previous logs during the events and see whether anything suspicious there?

And have a thourouch check of cpu utilization, memory utilization and disk utilization over a peiod of time and see whether any of these crossing the limit.

Thanks & Regards
Jayakrishnan G Naik

[Craig Sterley]

Attached is a measureware report. the system quit responding at about 15:23.

[Laurent Menase]

difficult to tell with only that but it is very probable to be swapping.
the system is short in memory, 96% before the problem and we see 100% of disk io & high pg out rate.

[Rita C Workman]

You've got measureware, and here you have the exact 5 minutes window it happened. Look down the other options to see what processes were running then so you know what direction you need to head to.
Is it an Oracle process (or whatever db software you run)?
>>>could be some rogue query or batch job run amuck.
Is it some system resource being done - like some massive disk sync process causing huge i/o that nobody told you about?

Just a couple thoughts,
Rita

[Jayakrishnan G Naik]

Hi

As per the log, it looks like system is running out of memory. See what processes are running and what is taking more memory. Is this the peak time where the usage is too high? The swap usage is too high which tells
there is no enough physical memory. You need to check your system for running processes taking more memory before reaching to a conclusion.

If all the processes making use of memory are needed to be running, then you may need to add more physical memory to the system.

I hope system goes to a state of thrashing which causes the system hang, May be thorough check to identify and stop unnecessary memory eating programs, adding more physical memory etc. can solve your issue.

Thanks & Regards
Jayakrishnan G Naik