HP-UX 11.11 seems to pause and go

[Joe Johnsen]

This is an rp3410, running HP-UX 11.11, Oracle 10g two instances, and is my Openview Management Server. I have noticed frequently that the machine will appear to lock up for 20 - 30 seconds. During this "lock up" you can't type in any xwindows on the machine. If you attempt a new telnet it appears to take the connection, but doesn't offer a login prompt. Network performance on the subnet seems fine. I haven't noticed any problems with any other machines on the same subnet. After the lock up is over, the machines picks up where it left off, any typing done during the lockup now shows up on the screen. These lockups appear to be affecting the application using the Oracle Database, which also locks up.
syslog and dmesg report nothing other than routine su messages during the lockup.

Any suggestions on how to further define this problem would be greatly appreciated.

[Patrick Wallek]

How much RAM do you have? How heavily is it used? Is swap used at all? What is the top process running during this time? How big are your Oracle SGA areas?

With 2 10G instances, I would have at least 8GB of RAM, if not more. If you are paging out, then performance will be slow to absolutely terrible.

If possible, run 'swapinfo -tam' and 'vmstat 5 20'. With the vmstat command, check the value of the 'po' column. If it is double digits or higher, then lack of RAM is probably your problem.

[James R. Ferguson]

Hi Joe:

> If you attempt a new telnet it appears to take the connection, but doesn't offer a login prompt.

If the behavior only manifests itself when you connect by name and not by IP addresss, then this is probably a reverse DNS resolution issue. Verify that you have reverse name pointers available in DNS.

Regards!

...JRF...

[Joe Johnsen]

Patrick, I currently have 2GB of RAM and 4GB of swap. swapinfo is reporting around 30% used. The behaviour has exhibited itself whether the lightly used small oracle databases are running or not, so I don't think this is a memory issue. When it occurs and I have been on the machine, it has behaved normally up to the point of the lock up, load average is 1 or less, with ram available. I have had top frequently running in another window when this has been occuring, it quits responding but the last information is still on the screen which appears normal to me. Typically there is no run up of processor load or memory load just prior to it locking up. It's like someone (if this were a sun) halted it and then resumed it a a little while later.

The problem is not strictly related to gaining access to the machine, so I don't suspect a DNS problem. I tried the telnet to determine initially if it was a problem with the window in question or the machine in general.

[Steven E. Protter]

Shalom,

http://www.hpux.ws/system.perf.sh

Collect data over time.

This problem seems typical of either an I/O problem or a system coming on the network with the same IP address, causing a network freeze.

SEP

[James R. Ferguson]

Hi (again) Joe:

For an Oracle database, 2GB is an awfully small memory footprint.

I'd also check your dynamic buffer cache size. Make sure that your 'dbc_max_pct' isn't still set to the default 50%.

If your buffer cache is high, you may be seeing the effects of the 'syncer' daemon flushing modified buffers every 30-seconds.

Regards!

...JRF...

[kenj_2]

Hi Joe -

Although it doesn't appear to to be network caused - all of the symptoms you report have a network component. Is there non-network impacted activity on the system that is also showing these stalls.

The key is finding out what the processes are blocking on. A simple technique would be to start up some "canary" scripts and see if they are affected. Measureware data and/or sar scripts might also add some clues.

Ken

[Pete Randall]

I would suspect that it might be network related, too. Run "lanadmin" > "lan" > "display" and look for high numbers of errors and collisions. It could be as simple as having an incorrect NIC setting.


Pete

[Joe Johnsen]

I have been running sar statistics with 10 minute intervals for a couple of years. These haven't really revealed anything of consequence. I will bump the time from 10 minutes to every 2 minutes, to see if something stands out. In addition when the lock up occurs I will utilize the console port (unfortunately it is network attached, but it is on the same subnet as my workstation and holds only console access) to see if it is network related. In addition I will pursue getting more memory, though in my years of experience this doesn't seem to be a memory shortage issue, the stoppages are too abrupt and recovery as soon as it is over.

I appreciate all the suggestions I have received so far. Just my luck, now that I have some time to pursue this, it hasn't happened.

To summarize, I will increase sar monitoring and add the monitor script or a modified version suggested earlier in this thread, in addition I will try to eliminate the network as a culprit by utilizing the console. Do any of these systems suggest bad hardware, e.g. a failing network card or memory...

Thanks for you help everyone.

[TwoProc]

2G ram and two databases.

This is not enough ram to run Oracle 8i properly, much less 10g. I've got more ram than this in PC, and exactly that much ram in my laptop!

You're at 30% swap, and during a swap in-out process, you're going to have the system exhibit this horrible shudder and lack of response while it rethinks its resources, and puts some of this off to disk, and shuffles in other stuff back into memory.

I believe that you need ram, badly.

I'm with Patrick, 8G minimum, 12G or 16G would be better.

[Joe Johnsen]

I appreciate the memory comments, however I think you missed something in the description of the problem. The machine will lock up even if the oracle process are not running. So unless you are saying that I need 8GB of RAM to have oracle installed on a filesystem without a single process running, it doesn't appear to be on point for the problem.

I am more inclined to lean towards a network issue at this time. It has come to my attention that we have a similar problem on a sun machine which for reasons I won't get into here has been hide NAT'd at our firewalls to the same address as the HP Machine in question. This NAT may be confusing the network.

[TwoProc]

Sorry Joe, I missed that point about Oracle running or not - I took it to mean that Oracle processes weren't doing anything much, not that Oracle wasn't even up.

Please excuse the mistake.

[Dennis Handly]

I've had hangs where mail checking and home directories on NFS would hang for a time.

[Joe Johnsen]

I think this problem is being caused by multiple factors. I think the primary cause is i/o performance. To address this I am planning on shifting some file systems around to balance the load. The NAT issue which is suspect was addressed yesterday. I have also noticed that at two particular times in the day, (ignite backups, and OpenView History dumps), the memory peaks causing swapping, however these two times do not correlate with the lockups, but as memory is pretty cheap we plan on bumping that as well.

Thank you for all the suggestions, they were very helpful. At this time I am closing this issue out, if necessary I will open a new one if the problem persists.

[Joe Johnsen]

issue closed.