VM guest with vhand processes

[Deepak Seth_1]

thanks for quick replaies. The only thing i did on the server in the last 12 hours was to reduce the filesystem_max value from 50 to 20% . Here are few things you folks wanted to see.
Attached is a file . The current system is system is still running slow . The simplae commands at times take some time to execute .
The vmstat ouput is showing page outs as well .

Any more suggestions.

[Deepak Seth_1]

to add one more thing , any interactive commands is taking few extra time to show up. its gives a feeling that you are remotely accessing the server . you type and the char comes little late.

time vgdisplay -v /dev/vg00
real 0.9
user 0.0
sys 0.0
took almost 0.9 sec . it should be done with 0 sec .

[Emil Velez]

Your filecache max is set to 1/2 of memory. What happens is your filecache is low and as the system runs the filecache grows (previous called DBC_MAX) to 50% of memory. IF Physical memory is being used by buffercache and more processes start running and making memory requests you get low on memory and vhand has to shrink the filecache and page out some process pages.

Solution: Make filecache_max lower 1/4 or 1/5 of physical memory. Oracle does not need buffer cache anyways since it handles its own cacheing. IN fact the oracle tablespace mount points should use mincache=direct mount option so it does not use buffer cache (filecache on 11.31)

good luck

All of these rules should be the same for virtual machines as well as physical machines

[Dennis Handly]

>was to reduce the filesystem_max value from 50 to 20.

That's still too big. Your page PO counts are very large: 2089

Your swapinfo info shows using 19% of about 10 Gb. (This seems strange if collected at the same time as your vmstat.

[Deepak Seth_1]

i have already reduced the file cache max value to 20% and its has not improved . in fact server is again not accessible . there is something else going on with this server .

[Eric SAUBIGNAC]

Bonjour,

As underlined by Dennis, PO is too high in vmstat, but swapinfo would tend to say that 10 Go is enough. Limit, but enough.

Do you have configured dynamic memory in your vm ? Post a verbose status of VM configuration with "hpvmstatus -V -P hervmd02"

Eric

[Deepak Seth_1]

figure out few things . The memory was set to 10 GB asMAX value but target value was was less . Increased the memory to 9 GB now (see attached file) . Also changed the filecache_max value to 10% . The server is alteast working for the last 2 days . but the performance is not great inspite of very little load.It remains slow responding to interactive commands. I just ran this command "timex vgdisplay -v vg00" . the output is
real 0.25
user 0.00
sys 0.02
The top command currently showing 63% cpu used by SYSTEM processes. Don't know why its used that much . Attaching
kmeminfo output
vmstat samples
hpvmstatus -v -P
This vm guest is runing with 3 oracle db - 10g version with abt 6 gb of shared memory .

[Deepak Seth_1]

more screen shot of top

CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.88 1.8% 41.7% 51.9% 4.7% 0.0% 0.0% 0.0% 0.0%
1 0.84 21.7% 2.4% 58.7% 17.3% 0.0% 0.0% 0.0% 0.0%
2 0.90 0.0% 47.2% 35.8% 16.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.88 7.9% 30.3% 48.8% 13.0% 0.0% 0.0% 0.0% 0.0%


why nice % os 30.3 and system is 48.8% .

[Eric SAUBIGNAC]

Bonjour Deepak,

In my opinion, you should first try to resolve memory issue before analyzing abnormal usage of CPU in sys mode. vmstat clearly shows heavy paging out activity and free memory is very low. Memory pressure can explain this kind of usage of cpu. How is sga configured ? Could you post "ipcs -mb"

From VM point of view, I had in the past some performance troubles with HP VM 3.0 and oracle 10.2.0.2 One of the symptoms was cpu usage in mode sys ... We have solved the problem with upgrading Oracle to 10.2.0.3 and applying oracle patch 6058758, and doing some Oracle tuning. May be you could check, but I am afraid that you are not concerned since your system is newer.

Still from VM, you should extend usage of AVIO to all your storage : only 2 virtual disks on 7 are avio_stor. It will not give you a spectacular change, but every thing is good to use when you tune a system.

Eric

PS : if you want a verbose status of a VM, don't use "-v" but "-V"

[Deepak Seth_1]

thanks Eric . I did change more one thing . Change the entitlement of cpu from 30% to 95% . also planning to use avio on all the LUNS as you also mentioned. Here is output of ipcs
IPC status from /dev/kmem as of Mon Jan 19 09:12:35 2009
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 0 0x411c016b --rw-rw-rw- root root 348
m 1 0x4e0c0002 --rw-rw-rw- root root 61760
m 2 0x41202399 --rw-rw-rw- root root 8192
m 3 0x411c14d6 --rw------- root root 4096
m 4 0x4914a8b8 --rw-r--r-- root root 22908
m 1966085 0x77d23114 --rw-r----- oracle dba 1140854784
m 6 0xbb4853fc --rw-r----- oracle dba 2181046272
m 8 0x1c0b79b8 --rw-r----- oracle dba 1140854784
m 9 0x01a1a079 --rw-rw-rw- root root 50331648
m 10 0x01d626e8 --rw-rw-rw- root root 50331648
m 11 0x0102d9f2 --rw-rw-rw- root root 50331648
m 12 0x01c488d2 --rw-rw-rw- root root 50331648

We are on oracle 10.2.0.4 with latest patch set . I think system looks little better after changing the CPU entitlement value . I have asked the users to test it again and provide me the feedback.
but thanks for your follow up. Do u have the syntax to change the normal disk to avio . i think i need to export the VG , remove the disk and then re-assign them using avio option . correct ?

attaching the latest output of hpvmstatus.