System hang problem on HP9000 R390/2 running Clearcase

[David Grant]

I have a long standing problem with my R390/2 HP-UX server where the system periodically freezes for no apparent reason. This leaves me with no option but to do a hard reset of the system with the usual results (i.e. either no problem or catastrophe!). The system is used primarily as a ClearCase Repository and we're also trying to set up a DDTS system on the same platform (Both software products from Rational Software ltd.). Over the past few months we have experienced many such incidents but have as yet failed to trace the cause.

We have only been able to accumulate some very broad information that may or may not be directly related to the problem. The first indication we usually get is the users reporting general performance problems but the UNIX system performance tools show only a lightly loaded system. Shortly after this (can even be a no. of days....) the system will hang. This seems to indicate a cumulative effect as if something in the background is building up until it brings the system down.
We have also noticed, or think we have, that the situation arising most frequently when we have had a consultant on site for DDTS configuration work. The DDTS system is being configured to use a web interface, and that in turn is using the Netscape Fasttrack Server bundled with the system. So it might be one or the other of these packages is to blame. Again a look at the logs doesn't turn up any indication that this is so.

So.......over to the forum! Are any of you running similar systems? Have any of you experienced similar problems and if so how did you solve them?

Thanks in advance for any help, this one has us all stumped (and that includes tech support on the HP and Rational side)

David

[Paula J Frazer-Campbell]

Hi David

1. When the system hangs does it come to a full stop with no responce anywhere or is it just very very slow.

2. When it reboots does it do a core dump?

3. In its hung state can you log in at the console? ( If you can, at next hang fire up top -q and examine what is going on.

4. Have you looked at all of the log files?

5. In some cases a hung state can be cleared by disconnecting the network from the server.

Just a few ideas

Paula

[David Grant]

Hi Paula,

Thanks for the swift response! To answer your questions:-
1.System totally frozen, no response at all.
2. No core dump generated on reboots (even toc reset generates "unable to determine process" message and produces a 0x0000 type dump).
3. No console access to system (using web console tho')but do get a ping response.
4. I've looked at all the log files I know about.....
5. Can't imagine that diconnecting thesystem from the network will have any effect in this case.....but I can certainly try next time.

[John Palmer]

Hi David,

This sounds as though it might be some sort of memory leak. Have you monitored memory and swap usage?

The following one-liner script is useful for tracking down processes which grow in size, it displays a list of processes sorted in descending order of virtual size.

UNIX95= ps -e -o ruser,vsz,sz,pid,args | sort -rnk2 | more

(Attributed to Bill Hassell from a previous post to this forum).

Regards,
John

[David Grant]

Hi John,

I have been monitoring memory usage as part of my overall diagnostics and it rarely gets above 12% utilisation (total RAM is 1GB). I've also looked at the process table both in the short and long term but have not noticed anything untoward. Unfortunately the one line script doesn't work on my system but I can access the information in other ways.

Thanks anyway
David

[Paula J Frazer-Campbell]

Hi David

A system stopping dead in the water and not leaving any trace of why it did it requires time to identify why it does it. So it looks you will have to monitor over a period of time running via cron some monitoring scripts ie :-

1. UNIX95= ps -e -o ruser,vsz,sz,pid,args | sort -rnk2 >> #WatchPS

2. Swapinfo >> #Watchswap

3. uptime >> ~WatchUsers-load

These log files will grow depending on how often you cron them - so give them an area in which they can expand.
I would cron them for either 10 or 15 mins.

HTH

Paula