Oracle Performance Issues in HPVM 3.5

[Rikki hinn Ogurlegi]

I have two rx3600 with 4 CPU's (2 dual core chips I think). Each machine has 16 gigs of RAM.

The machines both have 4 lan cards. Two of them are configured as a etherchannel (LinkAgg) to one switch. The third card is conected to another switch and acts as a standby for the channel. The fourth card is connected to the other machine via a crossover cable (Cluster Heartbeat)

The machines also have two SAN cards connected in a HA fashion via two switches to an EVA6000 array. Autopath is installed and configured:

==================================================================
HPswsp Version : A.3.0F.03F.00F
==================================================================
Auto Discover : ON
==================================================================
Array Type : EVA-AA
Array WWN : 5000-1FE1-500F-C250
Read Optimize : ON
Path Verification Period : 00:10
==================================================================
Lun WWN : 6005-08B4-0006-C7E5-0000-A000-01FF-0000
Virtual Device File : /hpap/dsk/hpap0
Load Balancing Policy : Round Robin
Lun Timeout : Infinite Retry (-1)
==================================================================
Device Path Status
==================================================================
/dev/dsk/c1t0d1 Active
/dev/dsk/c3t0d1 Active
/dev/dsk/c5t0d1 Active
/dev/dsk/c7t0d1 Active

[snip]

Both machines have HPVM 3.5 installed.

I created 1 guest pr. machine (more will be created later) which will run Oracle and then a buisness app and be clustered with ServiceGuard.

husacl2# hpvmstatus -P basalt
[Virtual Machine Details]
Virtual Machine Name VM # OS Type State
==================== ===== ======= ========
basalt 1 HPUX On (OS)

[Authorized Administrators]
Oper Groups:
Admin Groups:
Oper Users:
Admin Users:

[Virtual CPU Details]
#vCPUs Entitlement Maximum
====== =========== =======
4 80.0% 100.0%

[Memory Details]
Total Reserved
Memory Memory
======= ========
8 GB 64 MB

[Storage Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
disk avio_stor 0 4 0 0 0 disk /hpap/rdsk/hpap0
disk avio_stor 0 4 0 1 0 disk /hpap/rdsk/hpap9
disk avio_stor 0 4 0 2 0 disk /hpap/rdsk/hpap10
disk avio_stor 0 4 0 3 0 disk /hpap/rdsk/hpap11
disk avio_stor 0 4 0 4 0 disk /hpap/rdsk/hpap12
disk avio_stor 0 4 0 5 0 disk /hpap/rdsk/hpap13
disk avio_stor 0 4 0 6 0 disk /hpap/rdsk/hpap8
dvd scsi 0 5 0 0 0 null /dvd

[Network Interface Details]
Interface Adaptor Name/Num PortNum Bus Dev Ftn Mac Address
========= ========== ========== ======= === === === =================
vswitch lan Switch1 1 0 0 0 9a-e1-3e-11-87-39
vswitch lan Switch2 1 0 1 0 8a-2a-3c-ea-6f-5c
vswitch lan Switch3 1 0 2 0 12-e7-03-b9-d5-4e

[Misc Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
serial com1 tty console


If I understand this correctly I created this guest to have 4 vCPU's and issued 80% of each CPU to the guest.
There is no configuration (no LVM groups or anything else) on the LUN's on the host machines and the lun's are presented to the guests via avio. To my understanding this should give me the very best performance possible.

The lun's seen from the guest:

/dev/vgora/lvol1 26148864 17939861 7695969 70% /opt/oracle
/dev/vgora/lvol2 104792064 50584 98195144 0% /archive
/dev/vgora/lvol3 628948992 497676606 123067916 80% /oradata

granit# cd /oradata/
granit# prealloc testfile 100000000000

Meanwhile I run iostat in another window:

granit# iostat 3 | grep c0t5d0
c0t5d0 112734 3942.2 1.0
c0t5d0 156103 5463.9 1.0
c0t5d0 119417 4178.3 1.0
c0t5d0 112445 3934.4 1.0
c0t5d0 114498 4007.3 1.0

Very acceptable performance here for writes.

Reads give slightly higher numbers.

So from where I'm sitting the IO power of the guest looks OK. Still, the Oracle DBA's are reporting devestatingly slow performance of the Database. An Indexing operation that took 4 hours on the old machines (also Integrity running HP-UX 11iV2) now takes more than 24 hours on the new guests.

Did I configure something incorrectly (kernel parameters on the guests are as Oracle likes them) in this system or is Oracle prehaps not a candidate for HPVM ?

There is one issue I'm not sure how HPVM handles. Oracle really likes the async IO kernel subsystem to be active and working for maximum performance. Both hosts and guests have them active but do they work in the VM environment ?

Can anyone offer any suggestions on what to do next or how to improve performance?

Thanks in advance.

[Eric SAUBIGNAC · Accepted solution]

Bonjour Rikki,

In a first view I don't see any obvious missconfiguration. Simply be aware that 80% of entitlement means nothing if you have just one VM. And if you plan to have several VM, distribute entitlement in a relative way rather than in absolute (for more detailed explanations see : http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1198243)

One of my client had very very bad performances under HPVM 3.0 and Oracle 10.2.0.2. Some symptoms were very long connection times (depending how applications are written it can be disastrous) and high usage of cpu in %sys mode in the guest.

One solution was to upgrade to 10.2.0.3 and to apply Oracle Patch number 6058758. It seems this patch has been specially designed for HPVM ...

Hope this will help

Regards

Eric

[Eric SAUBIGNAC]

Sorry, hhtp link badly formatted : ')' at end.

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1198243

[Rikki hinn Ogurlegi]

Thanks. I will ask the DBA's to look into Oracle versions and that patch.
About the CPU entitlements. I suddenly had a thought. I created the guest with 4 vCPU's and 80% entitlement.
That means 80% of each vCPU ? Or does it mean 80% of one vCPU ?

[Eric SAUBIGNAC]

It means 80% for each vCPU of the guest. For example, if you have a hardware with 4 core, and you build a VM with 4 vCPU and give 80% of entitlement to the VM : at any time the VM Host must be able to give the VM guest 80% of the 4 cores.

But remember, if you have 2 VM Guests, and want to give 80% entitlement to the first one and 20% to the other, it is better to give 20% for the first one and 5% for the second.

Eric

[Tim Nelson]

Post some stats while the DBAs are running the indexing operation.

How is your memory allocated on the guest. 8GB - OS - buffer cache does not leave loads of room for SGA. Post some info for this as well. If you are going to create more guests you do not have much more memory to play with.
vHost needs about 4GB
vGuest with OS needs at least 1GB
So two guest will only leave you 10GB to spread between for Oracle. I do not know your Oracle environment so you will have to judge.


I would not entitle the 80%. The idea of virtualizing is to allow HW resources to be shared. If you are not going to share then dump the virtulization.

[Rikki hinn Ogurlegi]

Ahh. I was under the impression that even tho I set the entitlement to 80% the entitlement would only come into play when there was contention for the CPU.

Meaning that when guest 1 is not using it's 80% the remaining power is free for other guests.

Is that a misunderstanding on my part ?

[Tim Nelson]

The manual states as you mentioned. If there is no contention than one guest or the other can have more than its share. When there is contention then the entitlements come into play.

I do know that you cannot create entitlements that add up to more than 100%.

Let's say vGuest1 = 60% and vGuest2 = 60%. the system will not allow you to start vGuest2 as the total entitlements are now more than 100%.

The entitlements will limit the number of vGuests that you can create.

[Rikki hinn Ogurlegi]

The Oracle machine will run two other guests. One guest that is a test machine running a test instance of Oracle and then there is a devel guest. I planned a 10% entitlement for each because they really dont matter that much but the main guest is critical.

SGA size is 2,5G, PGA is 1G which should fit within the guests memoryspace.

I've asked the DBA's to run that indexing thing again and I will be watching the machines when that happens.

[Eric SAUBIGNAC]

"Meaning that when guest 1 is not using it's 80% the remaining power is free for other guests. Is that a misunderstanding on my part ? "

Sorry Rikki if I let you missunderstood that entitlement is someting like a reservation : entitlement begins when there is some cpu bottleneck at host level. So yes, remaining power is free for other guests. If you want fixed cpu, or capping or more enhanced features, consider gWlm.

"One guest that is a test machine running a test instance of Oracle and then there is a devel guest. I planned a 10% entitlement for each because they really dont matter that much but the main guest is critical."

Maybe I am doing some drivel, but rather than 80% + 10% + 10% entitlement, it would be better to plan 40% + 5% + 5% ;-)

For memory, you will find detailed informations at http://www.docs.hp.com/en/T2767-90105/ch02s01.html#hw_require But this document assumes that VM Host needs only 750 Mo. If you want to avoid some inconveniences in the future, it is better to consider 4 Go for the host as said by Tim. See a discussion on HPVM memory at http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1198245

Regards

Eric