System unreachable

[Keith Bryson]

Looking at your swapinfo/dmesg output, this is a 32Gb server with 30Gb (ish) swap? IF you run up glance or top, how much RAM is being reported as used? If you are hitting 90%+, I can't see that you have enough SWAP configured as HP-UX wants to reserve the same amount of RAM in swap at all times.

Let us know.
Keith

[nchat504]

System: sapux083 Fri Feb 5 17:59:07 2010
Load averages: 0.02, 0.01, 0.02
187 processes: 171 sleeping, 15 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.06 4.2% 0.0% 0.4% 95.4% 0.0% 0.0% 0.0% 0.0%
2 0.00 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
3 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
4 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
5 0.03 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
6 0.01 3.4% 0.0% 0.8% 95.8% 0.0% 0.0% 0.0% 0.0%
7 0.01 0.0% 0.0% 0.6% 99.4% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.02 1.0% 0.0% 0.2% 98.8% 0.0% 0.0% 0.0% 0.0%

Memory: 6177968K (1233628K) real, 18980620K (3751632K) virtual, 22543108K free
Page# 1/38

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 6148 prdadm 155 20 14891M 81092K sleep 0:31 12.51 12.49 dw.sapPRD_D
6 ? 15530 prdadm 154 20 14903M 93220K sleep 2:32 1.39 1.39 dw.sapPRD_D
7 ? 2435 root 152 20 587M 96616K run 0:21 0.79 0.79 java
=============================================
using the top command

[nchat504]

sapux083 /var/adm/syslog swlist | grep -i online
B3929DA 3.5-ga15-04 HP OnLineJFS 3.5
OnlineDiag B.11.11.18.05 HPUX 11.11 Support Tools Bundle, Dec 2006

[Keith Bryson]

11iv1 is going to make this a little more difficult - when was this server last patched? Have you applied the latest QPK/HWE/Feature bundles? Also, any idea what dbc_max_pct dbc_min_pct figures you have in the kernel (use sysdef or SAM to find out).

I can remember several issues with v1 and vhand (which were fixed with patches). I don't tend to rely on memory stats from "top", it's a shame you don't have glance installed. It's difficult for me to see the RSS figures for your SAP processes (it looks like 30Gb - but that may be shared memory).

I'd definitely consider making SWAP 1.5x RAM (at least).

Now, I wonder if they still have the evaluation copy of Glance somewhere.........

( 8 )
Keith

[Michael Steele_2]

Dec 2006

You have messages being sent an nothing listening, that is, if your firmware is up to date. IF not, then you have nothing being sent and nothing listening.

[Dennis Handly]

>My last resort was to issue an RS

If you want to know why a system was hung, you need to use TC to get a memory dump. (You first need to make sure crash dumps are enabled.)

>swapinfo -t

(It would helpful next time to always use -tam.)

>Patrick: A ping indicates some modicum of network connectivity, but that's no guarantee that other things will work.

I've had that too, unfortunately. :-(

>Keith: as HP-UX wants to reserve the same amount of RAM in swap at all times.

As long as you have pseudo-swap enabled, that's a myth about device swap.

[nchat504]

dbc_max_pct=8
dbc_min_pct=5

Systems were patched in 2008

We are currently working on a patch plan to update the systems. I will look into installing glance as well.

GOLDAPPS11i B.11.11.0712.475 Applications Patches for HP-UX 11i v1, December 2007
GOLDBASE11i B.11.11.0712.475 Base Patches for HP-UX 11i v1, December 2007
HWEnable11i B.11.11.0612.458 Hardware Enablement Patches for HP-UX 11i v1, December 2006

[Michael Steele_2]

Hi

Question for you: Has your application been upgraded recently with any new rollouts?

I ask because a memory leak could freeze the system.

Let me know.

Anyway, you're not going to know where the problem is until you upgrade your diags and firmware. You're probably many, many version out of date.

[nchat504]

Not sure about the rollouts.. i know they did a database refresh a few months back.. but this is their app server, so not sure what changes were made on it.. but i will ask. In the meantime, I will take your suggestion and see what I can do to expedite the patch/firmware and diag update..

Thanks for your feedback.