L2000 hangs

[Jacob Levin]

Hi all, I have a older L2000 that has started hanging lately. The system responds to pings but you cannot log onto it and all applications on the machine stop responding. The only way to bring it back is to do a reset of the system. No errors appear in any logs that I have been able to find. We did notice right before it hangs that Disk IO goes thru the roof, but other then that, we can't seem to find anything that points at this problem. Help!

Some info about the server:
B.11.00, 64-bit 9000/800/L2000-5X 2048 MB 2 x 540 MHz

[Patrick Wallek]

If disk I/O is going "thru the roof" then it is possible that you are doing some massive paging. A lot of paging can cause the system to seem like it is hung.

When / if this happens again, here are a couple of things to check:

# swapinfo -tam

# vmstat 5 12

In particular, look at the 'po' column of the vmstat ouput. If that is a large number (greater than 30) consisetntly, then that may be your problem.

From here, you should start checking applications and see if anything has changed. The first thing is to see if memory usage by an application has suddenly increased drastically, or if someone is running some sort of process, possibly incorrectly, that is using huge amounts of memory.

[Steven E. Protter]

Shalom,

Please define through the roof.

http://www.hpux.ws/system.perf.sh

It is possible that the system is crashing due to lack of an I/O hang patch.

vi /etc/rc.config.d/savecrash

Set the first variable to 1.

Restart the system.

Next time it hangs, press the TOC button ont he back. That forces a crash dump. Get the dump analyzed by HP after performing q4 analysis and they'll probably find a missing patch.

This has happened to me even on very well patched systems.

SEP

[Steven E. Protter]

Doh,

You'll find the crash dump in /var/adm/crash. If savecrash is already configured you may already have crash dumps there.

SEP

[Jacob Levin]

Glance reports disk IO above 90%, the server then stops responding so I am unable to tell how high the IO gets. Savecrash was not configured but I have taken care of it now. As I was monitoring the system earlier, we had an oracle RMAN archive job kick in. IO and memory spiked to 85% and 99% respectively. I ran vmstat 5 12 and saw page outs jump to 969 before dropping back to 0 when the rman job completed. During this time swap never got about 45% Shouldn't swap get used before paging occurs?

[A. Clay Stephenson]

There is no correlation between swap usage (45%) and the pageout rate. Think about fueling your car with the ignition switch on so that you can see the fuel quantity. It now indicates slightly below 1/2 full (45%); does this tell you anything about the rate at which you are pumping fuel into your tank?

You might have tons of swap space --- meaning that your system has a huge virtual address space so that it is possible to run (well, crawl might be better) many processes but if you are paging out significantly then performance is going to be very, very poor.

You needs lots more physical memory and/or a much smaller SGA and/or a reduced load.

Ideally the po rate should be zero, ~ 15 or so is ok but in all cases 900+ is bad, bad, bad.

[Patrick Wallek]

I agree with Clay 100%.

Haveing PO of 969 just about guarantees that your system performance will be VERY VERY BAD.

2GB of RAM on a Oracle DB machine is not very much at all.