VM guest with vhand processes

[Deepak Seth_1]

hpux guest 11.31 , VM 4.0 , vm guest is showing vhand as top process and system is really slow . the system is running 3-4 oracle DB . the system have a memory of 10 gb and 3 cpu . called HP . installed phkl_38949. system works fine for 2-3 hours and then again not accessible or very very slow . this is how the top looks like

System: herhxd02 Sun Jan 11 22:05:35 2009
Load averages: 0.65, 0.68, 0.49
282 processes: 99 sleeping, 45 running, 135 stopped, 3 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.58 2.2% 0.0% 13.4% 84.4% 0.0% 0.0% 0.0% 0.0%
1 0.59 0.9% 0.0% 1.4% 97.8% 0.0% 0.0% 0.0% 0.0%
2 0.79 0.3% 0.0% 4.5% 95.2% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.65 1.2% 0.0% 6.4% 92.4% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 58048K (30244K) real, 8470716K (1378520K) virtual, 18488K free Page# 1/11

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 2 root 128 20 72K 64K run 20:33 17.85 17.82 vhand
0 ? 12985 oracle 128 20 59068K 3664K run 0:02 1.02 1.00 gdb
0 ? 13607 root 130 20 3944K 136K sleep 0:00 6.06 0.84 grep
1

swapinfo output is - oh no system is again unaccessible. but last time i checked was ok.

i also increased the shmmax to 5 gb.

any help appreciated.

[Deepak Seth_1]

more information , rx6600 with 3 vm . total 8 cpus and 24 gb of memory. also seeing some gdb processes running on the top list of processes owned by oracle.
oracle 12388 23907 00:00:00 20:22:14 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12390 12388 00:00:00 20:22:14 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12392 12390 00:00:01 20:22:14 178 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12357
oracle 12828 23907 00:00:00 21:56:10 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12837 12828 00:00:00 21:56:12 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12842 12837 00:00:00 21:56:12 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12561
oracle 12935 2578 00:00:00 21:57:12 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12938 12935 00:00:00 21:57:14 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12940 12938 00:00:00 21:57:14 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12366
oracle 12934 2769 00:00:00 21:57:12 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12942 12934 00:00:00 21:57:16 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12944 12942 00:00:00 21:57:16 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12514
oracle 12412 2874 00:00:00 20:22:22 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12414 12412 00:00:00 20:22:23 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12417 12414 00:00:01 20:22:23 179 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12370
oracle 12416 3141 00:00:00 20:22:23 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12423 12416 00:00:00 20:22:24 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12425 12423 00:00:01 20:22:25 179 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12369

[Venkatesh BL]

vhand is the hp-ux pager daemon responsible for handling memory crunch situation by paging memory pages. what is the value of filecache_max and filecache_min kernel parameters? If you have 'kmeminfo' utility, you could post that output as well.

[Deepak Seth_1]

(herhxd02 root):/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 716931072 / 5104238592
maxdsiz 2031616 / 1073741824
maxdsiz_64bit 9437184 / 4294967296
maxfiles_lim 53 / 4096
maxssiz 163840 / 209715200
maxssiz_64bit 786432 / 268435456
maxtsiz 3829760 / 1073741824
maxtsiz_64bit 201326592 / 1073741824
maxuprc 70 / 4096
max_thread_proc 101 / 1200
msgmni 2 / 3200
msgtql 0 / 3200
nflocks 141 / 4096
ninode 2003 / 8192
nkthread 442 / 8416
nproc 227 / 4200
npty 0 / 60
nstrpty 6 / 256
nstrtel 0 / 256
nswapdev 1 / 32
nswapfs 0 / 32
semmni 27 / 4096
semmns 1641 / 4096
shmmax 2181046272 / 5368709120
shmmni 11 / 1024
shmseg 3 / 1024

system is not responding but this is what i got couple of hours back.

[Venkatesh BL · Accepted solution]

> (herhxd02 root):/> kcusage
> Tunable Usage / Setting
> =============================================
> filecache_max 716931072 / 5104238592

When the output was taken, the filecache usage was less.

Looks like filecache_max was set to 50% of total memory. This seems little high. As this can dynamically changed, you could try setting a smaller value and see if the system response is better during high load.

[Deepak Seth_1]

i changed the value to 20% . the output is like that
(herhxd02 root):/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 129933312 / 2041695436
maxdsiz 3915776 / 1073741824
maxdsiz_64bit 8650752 / 4294967296
maxfiles_lim 53 / 4096
maxssiz 98304 / 209715200
maxssiz_64bit 786432 / 268435456
maxtsiz 3829760 / 1073741824
maxtsiz_64bit 201326592 / 1073741824
maxuprc 32 / 4096
max_thread_proc 101 / 1200
msgmni 2 / 3200
msgtql 0 / 3200
nflocks 54 / 4096
ninode 1839 / 8192
nkthread 387 / 8416
nproc 173 / 4200
npty 0 / 60
nstrpty 1 / 256
nstrtel 0 / 256
nswapdev 1 / 32
nswapfs 0 / 32
semmni 22 / 4096
semmns 830 / 4096
shmmax 2181046272 / 5368709120
shmmni 7 / 1024
shmseg 3 / 1024
(herhxd02
i will monitor it . any other suggestions ,

[likid0]

If you are using the server only for oracle database you can lower the cache to 10% without problems, but i wouldn't think that's all the problems you have. can you also post hpvmstatus -v -p of the vm, and also info on the VM host status.

[likid0]

Looks like a memory problem, have a look at:

#swapinfo -atm
#kmeminfo (if you have it)

Have a look in the syslog for swap reservation failures or fork errors.

[Eric SAUBIGNAC]

Bonjour Deepak,

I would tend to say that low usage of file cache means that you have memory pressure ...

Could you post result of "vmstat -S 5 13" and "swapinfo -atm"

Eric

[Dennis Handly]

>vm guest is showing vhand as top process and system is really slow

Then you don't have enough memory.

>also seeing some gdb processes running on the top list of processes owned by oracle.
>0 ? 12985 oracle 128 20 59068K 3664K run 0:02 1.02 1.00 gdb

Oracle has aborted if it is running gdb.