Integrity VM not responding

[Tim Medford]

We have an rx3600 11.23 running 3 VMs, all VMs are also running 11.23.

This morning one of the VMs was "locked up" for lack of a better term. It was responding to a ping, but I could not login, even using hpvmconsole. The other VMs were responding normally. This particular VM is usually very busy, it runs about 14 oracle instances and a bunch of other java processes.

I eventually stopped all the VMs and restarted the host. All is fine now.

I'm trying to track down the problem, but not having much luck. The syslog.log file looks fine, it logged no errors at all. The vmguest log doesn't show any errors either. The only record I have is the listener.log for oracle which finally started throwing these errors:

28-APR-2008 06:41:03 * 12560
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
HPUX Error: 233: No buffer space available

If we were really out of buffer space, I'm surprised I didn't see any vmunix errors in syslog?

We did manage to get a quick snapshot in Glance from someone who was already logged in:

GlancePlus C.04.50.00 07:09:16 saifdev ia64 Current Avg High
--------------------------------------------------------------------------------
Cpu Util S SR RU U | 99% 98% 99%
Disk Util FF | 3% 7% 11%
Mem Util S SU UBB | 70% 70% 70%
Swap Util U UR R | 74% 74% 74%
--------------------------------------------------------------------------------
PROCESS LIST Users= 3
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
HPUX_System 8426 8235 152 root 83.1/77.5 18.3 0.0/ 0.0 4.5mb 1
DiskIO@saif 8425 8235 152 root 73.5/70.0 16.6 0.0/ 0.0 5.0mb 1
midaemon 14868 1 -16 root 17.6/16.5 3.9 0.0/ 0.5 15.5mb 2
saifdev-Ora 17900 1 168 root 5.4/ 4.1 1.0 0.0/ 0.0 11.9mb 1
java 18480 1 137 apptst1 1.3/ 2.6 0.6 0.0/ 0.0 105.2mb 107
java 17843 1 168 appdev2 1.2/ 0.7 0.2 0.0/ 0.0 110.3mb 83
oraclewcist 14976 14934 154 oracle 1.1/ 1.1 0.2 0.0/ 0.0 1.2mb 1
oraclewcist 14819 1 148 tnswcis 0.4/ 0.3 0.1 2.3/ 1.5 11.5mb 1
vxfsd 51 0 134 root 0.4/ 0.3 0.1 5.0/ 4.8 21.9mb 43
glance 14856 289 158 rambal 0.4/ 0.4


Thanks in advance for any ideas you might have.
Tim

[skt_skt]

the buffer space errors mostly due to the 100% MEM/SWAP usage. By the time you get in to the server the conditions mighe have improved.

there may be a separate log file created/updated for each of the VM client when ever stopped or started.

what is the current MEM/swap in the system. I would recomend a 2X phy ram as swap considering you are using 14 databases and especially Oracle one MEM intensive

[Ramesh S]

Hi

Probably, you would have TOC'd that particular VM to get a dump for root cause analysis.

Still you could check the following logs for any traces:

*/var/opt/hpvm/guests//log
*/var/opt/hpvm/common/command.log
* Try hpvmcollect for that VM

Best Regards,
Ramesh