topic Re: VM guest with vhand processes in Operating System - HP-UX

VM guest with vhand processes

Deepak Seth_1 — Mon, 12 Jan 2009 06:11:58 GMT

hpux guest 11.31 , VM 4.0 , vm guest is showing vhand as top process and system is really slow . the system is running 3-4 oracle DB . the system have a memory of 10 gb and 3 cpu . called HP . installed phkl_38949. system works fine for 2-3 hours and then again not accessible or very very slow . this is how the top looks like

System: herhxd02 Sun Jan 11 22:05:35 2009
Load averages: 0.65, 0.68, 0.49
282 processes: 99 sleeping, 45 running, 135 stopped, 3 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.58 2.2% 0.0% 13.4% 84.4% 0.0% 0.0% 0.0% 0.0%
1 0.59 0.9% 0.0% 1.4% 97.8% 0.0% 0.0% 0.0% 0.0%
2 0.79 0.3% 0.0% 4.5% 95.2% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.65 1.2% 0.0% 6.4% 92.4% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 58048K (30244K) real, 8470716K (1378520K) virtual, 18488K free Page# 1/11

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 2 root 128 20 72K 64K run 20:33 17.85 17.82 vhand
0 ? 12985 oracle 128 20 59068K 3664K run 0:02 1.02 1.00 gdb
0 ? 13607 root 130 20 3944K 136K sleep 0:00 6.06 0.84 grep
1

swapinfo output is - oh no system is again unaccessible. but last time i checked was ok.

i also increased the shmmax to 5 gb.

any help appreciated.

Re: VM guest with vhand processes

Deepak Seth_1 — Mon, 12 Jan 2009 06:26:04 GMT

more information , rx6600 with 3 vm . total 8 cpus and 24 gb of memory. also seeing some gdb processes running on the top list of processes owned by oracle.
oracle 12388 23907 00:00:00 20:22:14 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12390 12388 00:00:00 20:22:14 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12392 12390 00:00:01 20:22:14 178 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12357
oracle 12828 23907 00:00:00 21:56:10 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12837 12828 00:00:00 21:56:12 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12842 12837 00:00:00 21:56:12 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12561
oracle 12935 2578 00:00:00 21:57:12 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12938 12935 00:00:00 21:57:14 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12940 12938 00:00:00 21:57:14 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12366
oracle 12934 2769 00:00:00 21:57:12 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12942 12934 00:00:00 21:57:16 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12944 12942 00:00:00 21:57:16 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12514
oracle 12412 2874 00:00:00 20:22:22 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12414 12412 00:00:00 20:22:23 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12417 12414 00:00:01 20:22:23 179 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12370
oracle 12416 3141 00:00:00 20:22:23 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12423 12416 00:00:00 20:22:24 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12425 12423 00:00:01 20:22:25 179 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12369

Re: VM guest with vhand processes

Venkatesh BL — Mon, 12 Jan 2009 06:34:47 GMT

vhand is the hp-ux pager daemon responsible for handling memory crunch situation by paging memory pages. what is the value of filecache_max and filecache_min kernel parameters? If you have 'kmeminfo' utility, you could post that output as well.

Re: VM guest with vhand processes

Deepak Seth_1 — Mon, 12 Jan 2009 06:37:13 GMT

(herhxd02 root):/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 716931072 / 5104238592
maxdsiz 2031616 / 1073741824
maxdsiz_64bit 9437184 / 4294967296
maxfiles_lim 53 / 4096
maxssiz 163840 / 209715200
maxssiz_64bit 786432 / 268435456
maxtsiz 3829760 / 1073741824
maxtsiz_64bit 201326592 / 1073741824
maxuprc 70 / 4096
max_thread_proc 101 / 1200
msgmni 2 / 3200
msgtql 0 / 3200
nflocks 141 / 4096
ninode 2003 / 8192
nkthread 442 / 8416
nproc 227 / 4200
npty 0 / 60
nstrpty 6 / 256
nstrtel 0 / 256
nswapdev 1 / 32
nswapfs 0 / 32
semmni 27 / 4096
semmns 1641 / 4096
shmmax 2181046272 / 5368709120
shmmni 11 / 1024
shmseg 3 / 1024

system is not responding but this is what i got couple of hours back.

Re: VM guest with vhand processes

Venkatesh BL — Mon, 12 Jan 2009 06:53:41 GMT

> (herhxd02 root):/> kcusage
> Tunable Usage / Setting
> =============================================
> filecache_max 716931072 / 5104238592

When the output was taken, the filecache usage was less.

Looks like filecache_max was set to 50% of total memory. This seems little high. As this can dynamically changed, you could try setting a smaller value and see if the system response is better during high load.

Re: VM guest with vhand processes

Deepak Seth_1 — Mon, 12 Jan 2009 07:18:54 GMT

i changed the value to 20% . the output is like that
(herhxd02 root):/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 129933312 / 2041695436
maxdsiz 3915776 / 1073741824
maxdsiz_64bit 8650752 / 4294967296
maxfiles_lim 53 / 4096
maxssiz 98304 / 209715200
maxssiz_64bit 786432 / 268435456
maxtsiz 3829760 / 1073741824
maxtsiz_64bit 201326592 / 1073741824
maxuprc 32 / 4096
max_thread_proc 101 / 1200
msgmni 2 / 3200
msgtql 0 / 3200
nflocks 54 / 4096
ninode 1839 / 8192
nkthread 387 / 8416
nproc 173 / 4200
npty 0 / 60
nstrpty 1 / 256
nstrtel 0 / 256
nswapdev 1 / 32
nswapfs 0 / 32
semmni 22 / 4096
semmns 830 / 4096
shmmax 2181046272 / 5368709120
shmmni 7 / 1024
shmseg 3 / 1024
(herhxd02
i will monitor it . any other suggestions ,

Re: VM guest with vhand processes

likid0 — Mon, 12 Jan 2009 08:37:54 GMT

If you are using the server only for oracle database you can lower the cache to 10% without problems, but i wouldn't think that's all the problems you have. can you also post hpvmstatus -v -p of the vm, and also info on the VM host status.

Re: VM guest with vhand processes

likid0 — Mon, 12 Jan 2009 08:40:59 GMT

Looks like a memory problem, have a look at:

#swapinfo -atm
#kmeminfo (if you have it)

Have a look in the syslog for swap reservation failures or fork errors.

Re: VM guest with vhand processes

Eric SAUBIGNAC — Mon, 12 Jan 2009 08:45:12 GMT

Bonjour Deepak,

I would tend to say that low usage of file cache means that you have memory pressure ...

Could you post result of "vmstat -S 5 13" and "swapinfo -atm"

Eric

Re: VM guest with vhand processes

Dennis Handly — Mon, 12 Jan 2009 10:18:42 GMT

>vm guest is showing vhand as top process and system is really slow

Then you don't have enough memory.

>also seeing some gdb processes running on the top list of processes owned by oracle.
>0 ? 12985 oracle 128 20 59068K 3664K run 0:02 1.02 1.00 gdb

Oracle has aborted if it is running gdb.

Re: VM guest with vhand processes

Deepak Seth_1 — Mon, 12 Jan 2009 19:52:45 GMT

thanks for quick replaies. The only thing i did on the server in the last 12 hours was to reduce the filesystem_max value from 50 to 20% . Here are few things you folks wanted to see.
Attached is a file . The current system is system is still running slow . The simplae commands at times take some time to execute .
The vmstat ouput is showing page outs as well .

Any more suggestions.

Re: VM guest with vhand processes

Deepak Seth_1 — Tue, 13 Jan 2009 00:32:58 GMT

to add one more thing , any interactive commands is taking few extra time to show up. its gives a feeling that you are remotely accessing the server . you type and the char comes little late.

time vgdisplay -v /dev/vg00
real 0.9
user 0.0
sys 0.0
took almost 0.9 sec . it should be done with 0 sec .

Re: VM guest with vhand processes

Emil Velez — Tue, 13 Jan 2009 03:30:23 GMT

Your filecache max is set to 1/2 of memory. What happens is your filecache is low and as the system runs the filecache grows (previous called DBC_MAX) to 50% of memory. IF Physical memory is being used by buffercache and more processes start running and making memory requests you get low on memory and vhand has to shrink the filecache and page out some process pages.

Solution: Make filecache_max lower 1/4 or 1/5 of physical memory. Oracle does not need buffer cache anyways since it handles its own cacheing. IN fact the oracle tablespace mount points should use mincache=direct mount option so it does not use buffer cache (filecache on 11.31)

good luck

All of these rules should be the same for virtual machines as well as physical machines

Re: VM guest with vhand processes

Dennis Handly — Tue, 13 Jan 2009 04:28:54 GMT

>was to reduce the filesystem_max value from 50 to 20.

That's still too big. Your page PO counts are very large: 2089

Your swapinfo info shows using 19% of about 10 Gb. (This seems strange if collected at the same time as your vmstat.

Re: VM guest with vhand processes

Deepak Seth_1 — Tue, 13 Jan 2009 04:32:09 GMT

i have already reduced the file cache max value to 20% and its has not improved . in fact server is again not accessible . there is something else going on with this server .

Re: VM guest with vhand processes

Eric SAUBIGNAC — Tue, 13 Jan 2009 09:37:07 GMT

Bonjour,

As underlined by Dennis, PO is too high in vmstat, but swapinfo would tend to say that 10 Go is enough. Limit, but enough.

Do you have configured dynamic memory in your vm ? Post a verbose status of VM configuration with "hpvmstatus -V -P hervmd02"

Eric

Re: VM guest with vhand processes

Deepak Seth_1 — Thu, 15 Jan 2009 18:50:16 GMT

figure out few things . The memory was set to 10 GB asMAX value but target value was was less . Increased the memory to 9 GB now (see attached file) . Also changed the filecache_max value to 10% . The server is alteast working for the last 2 days . but the performance is not great inspite of very little load.It remains slow responding to interactive commands. I just ran this command "timex vgdisplay -v vg00" . the output is
real 0.25
user 0.00
sys 0.02
The top command currently showing 63% cpu used by SYSTEM processes. Don't know why its used that much . Attaching
kmeminfo output
vmstat samples
hpvmstatus -v -P
This vm guest is runing with 3 oracle db - 10g version with abt 6 gb of shared memory .

Re: VM guest with vhand processes

Deepak Seth_1 — Fri, 16 Jan 2009 01:11:46 GMT

more screen shot of top

CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.88 1.8% 41.7% 51.9% 4.7% 0.0% 0.0% 0.0% 0.0%
1 0.84 21.7% 2.4% 58.7% 17.3% 0.0% 0.0% 0.0% 0.0%
2 0.90 0.0% 47.2% 35.8% 16.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.88 7.9% 30.3% 48.8% 13.0% 0.0% 0.0% 0.0% 0.0%

why nice % os 30.3 and system is 48.8% .

Re: VM guest with vhand processes

Eric SAUBIGNAC — Mon, 19 Jan 2009 09:12:13 GMT

Bonjour Deepak,

In my opinion, you should first try to resolve memory issue before analyzing abnormal usage of CPU in sys mode. vmstat clearly shows heavy paging out activity and free memory is very low. Memory pressure can explain this kind of usage of cpu. How is sga configured ? Could you post "ipcs -mb"

From VM point of view, I had in the past some performance troubles with HP VM 3.0 and oracle 10.2.0.2 One of the symptoms was cpu usage in mode sys ... We have solved the problem with upgrading Oracle to 10.2.0.3 and applying oracle patch 6058758, and doing some Oracle tuning. May be you could check, but I am afraid that you are not concerned since your system is newer.

Still from VM, you should extend usage of AVIO to all your storage : only 2 virtual disks on 7 are avio_stor. It will not give you a spectacular change, but every thing is good to use when you tune a system.

Eric

PS : if you want a verbose status of a VM, don't use "-v" but "-V"

Re: VM guest with vhand processes

Deepak Seth_1 — Mon, 19 Jan 2009 17:23:05 GMT

thanks Eric . I did change more one thing . Change the entitlement of cpu from 30% to 95% . also planning to use avio on all the LUNS as you also mentioned. Here is output of ipcs
IPC status from /dev/kmem as of Mon Jan 19 09:12:35 2009
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 0 0x411c016b --rw-rw-rw- root root 348
m 1 0x4e0c0002 --rw-rw-rw- root root 61760
m 2 0x41202399 --rw-rw-rw- root root 8192
m 3 0x411c14d6 --rw------- root root 4096
m 4 0x4914a8b8 --rw-r--r-- root root 22908
m 1966085 0x77d23114 --rw-r----- oracle dba 1140854784
m 6 0xbb4853fc --rw-r----- oracle dba 2181046272
m 8 0x1c0b79b8 --rw-r----- oracle dba 1140854784
m 9 0x01a1a079 --rw-rw-rw- root root 50331648
m 10 0x01d626e8 --rw-rw-rw- root root 50331648
m 11 0x0102d9f2 --rw-rw-rw- root root 50331648
m 12 0x01c488d2 --rw-rw-rw- root root 50331648

We are on oracle 10.2.0.4 with latest patch set . I think system looks little better after changing the CPU entitlement value . I have asked the users to test it again and provide me the feedback.
but thanks for your follow up. Do u have the syntax to change the normal disk to avio . i think i need to export the VG , remove the disk and then re-assign them using avio option . correct ?

attaching the latest output of hpvmstatus.