VM guest with vhand processes

[Deepak Seth_1]

hpux guest 11.31 , VM 4.0 , vm guest is showing vhand as top process and system is really slow . the system is running 3-4 oracle DB . the system have a memory of 10 gb and 3 cpu . called HP . installed phkl_38949. system works fine for 2-3 hours and then again not accessible or very very slow . this is how the top looks like

System: herhxd02 Sun Jan 11 22:05:35 2009
Load averages: 0.65, 0.68, 0.49
282 processes: 99 sleeping, 45 running, 135 stopped, 3 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.58 2.2% 0.0% 13.4% 84.4% 0.0% 0.0% 0.0% 0.0%
1 0.59 0.9% 0.0% 1.4% 97.8% 0.0% 0.0% 0.0% 0.0%
2 0.79 0.3% 0.0% 4.5% 95.2% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.65 1.2% 0.0% 6.4% 92.4% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 58048K (30244K) real, 8470716K (1378520K) virtual, 18488K free Page# 1/11

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 2 root 128 20 72K 64K run 20:33 17.85 17.82 vhand
0 ? 12985 oracle 128 20 59068K 3664K run 0:02 1.02 1.00 gdb
0 ? 13607 root 130 20 3944K 136K sleep 0:00 6.06 0.84 grep
1

swapinfo output is - oh no system is again unaccessible. but last time i checked was ok.

i also increased the shmmax to 5 gb.

any help appreciated.

[Deepak Seth_1]

more information , rx6600 with 3 vm . total 8 cpus and 24 gb of memory. also seeing some gdb processes running on the top list of processes owned by oracle.
oracle 12388 23907 00:00:00 20:22:14 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12390 12388 00:00:00 20:22:14 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12392 12390 00:00:01 20:22:14 178 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12357
oracle 12828 23907 00:00:00 21:56:10 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12837 12828 00:00:00 21:56:12 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12842 12837 00:00:00 21:56:12 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12561
oracle 12935 2578 00:00:00 21:57:12 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12938 12935 00:00:00 21:57:14 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12940 12938 00:00:00 21:57:14 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12366
oracle 12934 2769 00:00:00 21:57:12 158 20 S HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12942 12934 00:00:00 21:57:16 158 20 S HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12944 12942 00:00:00 21:57:16 128 20 S HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12514
oracle 12412 2874 00:00:00 20:22:22 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12414 12412 00:00:00 20:22:23 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12417 12414 00:00:01 20:22:23 179 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12370
oracle 12416 3141 00:00:00 20:22:23 158 20 T HPUX sh -c /bin/sh -c '/usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/or
oracle 12423 12416 00:00:00 20:22:24 158 20 T HPUX /bin/sh -c /usr/bin/echo "set pagination off ^J bt" | /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12
oracle 12425 12423 00:00:01 20:22:25 179 20 R HPUX /opt/langtools/bin/gdb -quiet /db/app/oracle/product/10.2.0/bin/oracle 12369

[Venkatesh BL]

vhand is the hp-ux pager daemon responsible for handling memory crunch situation by paging memory pages. what is the value of filecache_max and filecache_min kernel parameters? If you have 'kmeminfo' utility, you could post that output as well.

[Deepak Seth_1]

(herhxd02 root):/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 716931072 / 5104238592
maxdsiz 2031616 / 1073741824
maxdsiz_64bit 9437184 / 4294967296
maxfiles_lim 53 / 4096
maxssiz 163840 / 209715200
maxssiz_64bit 786432 / 268435456
maxtsiz 3829760 / 1073741824
maxtsiz_64bit 201326592 / 1073741824
maxuprc 70 / 4096
max_thread_proc 101 / 1200
msgmni 2 / 3200
msgtql 0 / 3200
nflocks 141 / 4096
ninode 2003 / 8192
nkthread 442 / 8416
nproc 227 / 4200
npty 0 / 60
nstrpty 6 / 256
nstrtel 0 / 256
nswapdev 1 / 32
nswapfs 0 / 32
semmni 27 / 4096
semmns 1641 / 4096
shmmax 2181046272 / 5368709120
shmmni 11 / 1024
shmseg 3 / 1024

system is not responding but this is what i got couple of hours back.

[Venkatesh BL · Accepted solution]

> (herhxd02 root):/> kcusage
> Tunable Usage / Setting
> =============================================
> filecache_max 716931072 / 5104238592

When the output was taken, the filecache usage was less.

Looks like filecache_max was set to 50% of total memory. This seems little high. As this can dynamically changed, you could try setting a smaller value and see if the system response is better during high load.

[Deepak Seth_1]

i changed the value to 20% . the output is like that
(herhxd02 root):/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 129933312 / 2041695436
maxdsiz 3915776 / 1073741824
maxdsiz_64bit 8650752 / 4294967296
maxfiles_lim 53 / 4096
maxssiz 98304 / 209715200
maxssiz_64bit 786432 / 268435456
maxtsiz 3829760 / 1073741824
maxtsiz_64bit 201326592 / 1073741824
maxuprc 32 / 4096
max_thread_proc 101 / 1200
msgmni 2 / 3200
msgtql 0 / 3200
nflocks 54 / 4096
ninode 1839 / 8192
nkthread 387 / 8416
nproc 173 / 4200
npty 0 / 60
nstrpty 1 / 256
nstrtel 0 / 256
nswapdev 1 / 32
nswapfs 0 / 32
semmni 22 / 4096
semmns 830 / 4096
shmmax 2181046272 / 5368709120
shmmni 7 / 1024
shmseg 3 / 1024
(herhxd02
i will monitor it . any other suggestions ,

[likid0]

If you are using the server only for oracle database you can lower the cache to 10% without problems, but i wouldn't think that's all the problems you have. can you also post hpvmstatus -v -p of the vm, and also info on the VM host status.

[likid0]

Looks like a memory problem, have a look at:

#swapinfo -atm
#kmeminfo (if you have it)

Have a look in the syslog for swap reservation failures or fork errors.

[Eric SAUBIGNAC]

Bonjour Deepak,

I would tend to say that low usage of file cache means that you have memory pressure ...

Could you post result of "vmstat -S 5 13" and "swapinfo -atm"

Eric

[Dennis Handly]

>vm guest is showing vhand as top process and system is really slow

Then you don't have enough memory.

>also seeing some gdb processes running on the top list of processes owned by oracle.
>0 ? 12985 oracle 128 20 59068K 3664K run 0:02 1.02 1.00 gdb

Oracle has aborted if it is running gdb.

[Deepak Seth_1]

thanks for quick replaies. The only thing i did on the server in the last 12 hours was to reduce the filesystem_max value from 50 to 20% . Here are few things you folks wanted to see.
Attached is a file . The current system is system is still running slow . The simplae commands at times take some time to execute .
The vmstat ouput is showing page outs as well .

Any more suggestions.

[Deepak Seth_1]

to add one more thing , any interactive commands is taking few extra time to show up. its gives a feeling that you are remotely accessing the server . you type and the char comes little late.

time vgdisplay -v /dev/vg00
real 0.9
user 0.0
sys 0.0
took almost 0.9 sec . it should be done with 0 sec .

[Emil Velez]

Your filecache max is set to 1/2 of memory. What happens is your filecache is low and as the system runs the filecache grows (previous called DBC_MAX) to 50% of memory. IF Physical memory is being used by buffercache and more processes start running and making memory requests you get low on memory and vhand has to shrink the filecache and page out some process pages.

Solution: Make filecache_max lower 1/4 or 1/5 of physical memory. Oracle does not need buffer cache anyways since it handles its own cacheing. IN fact the oracle tablespace mount points should use mincache=direct mount option so it does not use buffer cache (filecache on 11.31)

good luck

All of these rules should be the same for virtual machines as well as physical machines

[Dennis Handly]

>was to reduce the filesystem_max value from 50 to 20.

That's still too big. Your page PO counts are very large: 2089

Your swapinfo info shows using 19% of about 10 Gb. (This seems strange if collected at the same time as your vmstat.

[Deepak Seth_1]

i have already reduced the file cache max value to 20% and its has not improved . in fact server is again not accessible . there is something else going on with this server .

[Eric SAUBIGNAC]

Bonjour,

As underlined by Dennis, PO is too high in vmstat, but swapinfo would tend to say that 10 Go is enough. Limit, but enough.

Do you have configured dynamic memory in your vm ? Post a verbose status of VM configuration with "hpvmstatus -V -P hervmd02"

Eric

[Deepak Seth_1]

figure out few things . The memory was set to 10 GB asMAX value but target value was was less . Increased the memory to 9 GB now (see attached file) . Also changed the filecache_max value to 10% . The server is alteast working for the last 2 days . but the performance is not great inspite of very little load.It remains slow responding to interactive commands. I just ran this command "timex vgdisplay -v vg00" . the output is
real 0.25
user 0.00
sys 0.02
The top command currently showing 63% cpu used by SYSTEM processes. Don't know why its used that much . Attaching
kmeminfo output
vmstat samples
hpvmstatus -v -P
This vm guest is runing with 3 oracle db - 10g version with abt 6 gb of shared memory .

[Deepak Seth_1]

more screen shot of top

CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.88 1.8% 41.7% 51.9% 4.7% 0.0% 0.0% 0.0% 0.0%
1 0.84 21.7% 2.4% 58.7% 17.3% 0.0% 0.0% 0.0% 0.0%
2 0.90 0.0% 47.2% 35.8% 16.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.88 7.9% 30.3% 48.8% 13.0% 0.0% 0.0% 0.0% 0.0%


why nice % os 30.3 and system is 48.8% .

[Eric SAUBIGNAC]

Bonjour Deepak,

In my opinion, you should first try to resolve memory issue before analyzing abnormal usage of CPU in sys mode. vmstat clearly shows heavy paging out activity and free memory is very low. Memory pressure can explain this kind of usage of cpu. How is sga configured ? Could you post "ipcs -mb"

From VM point of view, I had in the past some performance troubles with HP VM 3.0 and oracle 10.2.0.2 One of the symptoms was cpu usage in mode sys ... We have solved the problem with upgrading Oracle to 10.2.0.3 and applying oracle patch 6058758, and doing some Oracle tuning. May be you could check, but I am afraid that you are not concerned since your system is newer.

Still from VM, you should extend usage of AVIO to all your storage : only 2 virtual disks on 7 are avio_stor. It will not give you a spectacular change, but every thing is good to use when you tune a system.

Eric

PS : if you want a verbose status of a VM, don't use "-v" but "-V"

[Deepak Seth_1]

thanks Eric . I did change more one thing . Change the entitlement of cpu from 30% to 95% . also planning to use avio on all the LUNS as you also mentioned. Here is output of ipcs
IPC status from /dev/kmem as of Mon Jan 19 09:12:35 2009
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 0 0x411c016b --rw-rw-rw- root root 348
m 1 0x4e0c0002 --rw-rw-rw- root root 61760
m 2 0x41202399 --rw-rw-rw- root root 8192
m 3 0x411c14d6 --rw------- root root 4096
m 4 0x4914a8b8 --rw-r--r-- root root 22908
m 1966085 0x77d23114 --rw-r----- oracle dba 1140854784
m 6 0xbb4853fc --rw-r----- oracle dba 2181046272
m 8 0x1c0b79b8 --rw-r----- oracle dba 1140854784
m 9 0x01a1a079 --rw-rw-rw- root root 50331648
m 10 0x01d626e8 --rw-rw-rw- root root 50331648
m 11 0x0102d9f2 --rw-rw-rw- root root 50331648
m 12 0x01c488d2 --rw-rw-rw- root root 50331648

We are on oracle 10.2.0.4 with latest patch set . I think system looks little better after changing the CPU entitlement value . I have asked the users to test it again and provide me the feedback.
but thanks for your follow up. Do u have the syntax to change the normal disk to avio . i think i need to export the VG , remove the disk and then re-assign them using avio option . correct ?

attaching the latest output of hpvmstatus.

[Eric SAUBIGNAC]

Bonsoir,

Really I don't think that growing entitlement to 95% has solved any problem. Entitlement should only play when you have cpu pressure from several VM Guests. In this case, the VM host will distribute the CPU to the VM in a ratio accorded to entitlement of each VM.

More : if you leave 95% as an entitlement to this VM, and if you stop other(s) VM(s), there is a strong chance that you will not be able to start again the stopped VM :-(

From memory point of view, ipcs shows that you take about 4.2 GB for SGAs. It is about 50% of the VM memory (9 GB ?) and it might be a lot depending on how others processes are using memory. It could confirm that you have memory bottleneck.

Since we are speaking of memory, you have posted "hpvmstatus -V" from the wrong VM hervmd01. I thought that you have problems with hervmd02. Right ?

Eric

[Eric SAUBIGNAC]

Oups ... forgot last question of your post.

"i think i need ... remove the disk and then re-assign them using avio option "

With "hpvmmodify -m" you can only modify the physical part of a ressource aasigned to a VM. You can't modify "scsi" to "avio_stor". So Yes, you are right, you will have to remove "hpvmmodify -d" then re-assign with "hpvmmodify -a"

"i think i need to export the VG ..."

If guest OS is 11iv2 you can avoid exporting the VG if you stop the VM before modifying backing store and carefully add back the virtual disks with the same bus / device / target than before removing. Then restart the VM.

If guest is 11iv3 and you use persistent DSF you will have to export-import the VG. Disk ID changes when you switch from scsi to avio_stor and a new DSF is created.

One more consideration : on a given virtual bus / device, you can't mix scsi and avio_stor. When one device is of type avio, all the bus / device is of type avio. Thus all devices on the same bus / device must be of type avio


Eric

[Deepak Seth_1]

yes , i send the wrong VM information .
But the entitlement of 95% is only related to number of CPU assign to this VM . Correct ?
(herhxv01 root):/> hpvmstatus -v
Version B.04.00.00
[Virtual Machines]
Virtual Machine Name VM # OS Type State #VCPUs #Devs #Nets Memory Runsysid
==================== ===== ======= ========= ====== ===== ===== ======= ========
hervmd01 2 HPUX Off 1 1 1 4 GB 0
hervmd02 3 HPUX On (OS) 3 7 1 10 GB 0
hervmd03 4 HPUX On (OS) 2 1 1 3 GB 0
(
Is the 95% of CPU entitlement means that a VM can go upto a 95% of CPU utilization in case other VM have free CPU or its just restricted to only the maximum defined in the VM .
Regarding the memory , this is a test server and very lightly used and still users complaint about its slow performance . Even i feel a slowness just typing commands.
i updated the correct hpcmstatus output .

[Eric SAUBIGNAC]

Bonjour Deepak,

Your latest hpvmstatus seems to confirm that memory is THE bottleneck :

- Memory pressure is 80. Page 141 of HP Integrity Virtual Machines Version 4.0 Installation, Configuration, and Administration "A value between 0 and 100 used as an indicator of memory deficit and paging. The higher the number the longer the system has been in a memory deficit. A memory pressure value approaching 100 usually means the system is hung."

- Comfortable minimum is 10234 MB. Same guide, same page "A value for ram_target which can be used to reduce the guest memory but allow it sufficient memory resources to continue running a minimal workload." Don't know exactly how "Comfortable memory" is computed, but I guess you should understand that 10240 MB is a minimum to allocate to your VM

- Current memory is 8250 MB. You should size it at leats to the maximum.

==> Don't known how many physical RAM you have on this host, but you should try to allocate many more than 10GB. Even, you should stop others VM and allocate the maximum you can to hervmd02 then determine what would be the correct sizing for this VM.

==> When others VM are not under CPU pressure, a VM under cpu pressure can goes up to 100% (not exactly, see below) of the same number of physical cpu than virtual cpu it has. Entitlement will only play when several VM are under CPU pressure.

From Integrity VM 4.0, there is a new entitlement properties which is max_percent that will cap VM's usage of CPU. So when I previously said that a VM can reach 100%, it is false. It is able to reach only max_percent of CPU only. Your VM is configured at 100%

Eric