Unusual disk I/O in HPVM environment

[George N]

Hello,

We are new to HPVM, and are having a problem with unusual disk I/O patterns. We are using an rx6600 server with 4 cores and 48 GB. We have a single VM configured (hpvmstatus output is below).

Within our VM, we are running Oracle 10g, and some custom java applications using WebLogic 9.2.MP3.

The problem we are experiencing, is that any time we start processing transactions on our VM, we see heavy I/O on the disks belonging to the VM Host! The activity lights on the drives light up solid ON for the VM Host drives, while the activity lights for our VM only show the occasional light blinking. Also, disk I/O performance within the VM is very poor.

Why does activity within our VM cause large amounts of I/O on the VM Host??? And why would disk I/O on the VM Host cause a performance bottleneck within a VM??

Thank you in advance!

----
# hpvmstatus -P VMitl1
[Virtual Machine Details]
Virtual Machine Name VM # OS Type State
==================== ===== ======= ========
VMitl1 5 HPUX On (OS)

[Authorized Administrators]
Oper Groups:
Admin Groups:
Oper Users:
Admin Users:

[Virtual CPU Details]
#vCPUs Entitlement Maximum
====== =========== =======
4 10.0% 50.0%

[Memory Details]
Total Reserved
Memory Memory
======= ========
10 GB 64 MB

[Storage Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
disk avio_stor 0 0 0 0 0 lv /dev/VGitl1/ritl1
disk avio_stor 0 0 0 1 0 disk /dev/rdisk/disk31
disk avio_stor 0 0 0 2 0 disk /dev/rdisk/disk32
disk avio_stor 0 0 0 3 0 disk /dev/rdisk/disk35
dvd scsi 0 1 0 0 0 null /var/opt/hpvm/ISO-images/hpux/

[Network Interface Details]
Interface Adaptor Name/Num PortNum Bus Dev Ftn Mac Address
========= ========== ========== ======= === === === =================
vswitch avio_lan lan6sw 1 0 2 0 1e-15-aa-87-6b-60

[Misc Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
serial com1 tty console

[Olivier Masse]

Mmmm you're using AVIO and raw disks, and that's the best configuration I know to optimize I/O performance. I've been using this for almost a year, and performance is good, my raw devices are located on an EVA.

From what I see here, I'm assuming /dev/VGitl1/ritl1 is mapped to the root disk of your VM, while disk31,32,25 are other disks which hold your data? Check if your VM is paging. If it does, it will use the swap device which is probably behind /dev/VGitl1. 10Gb is not that much if you have a big Oracle database, and the SGA might have to be tuned for that amount of memory.

Besides that, I don't know. You might want to use glance to confirm which disks are being written to on the VM Host. Don't forget that any read on your .iso file will also turn the lights on, you should disconnect it from the VM if you don't need it. Also note that when using AVIO, it's actually the swapper process that will handle all the disk I/O, and not hpvmapp. That can be misleading.

Good luck

[Anonymous]

Just a note. It's not swapper process handling IO. That was a bug fixed in A.03.50, I believe. Prior that fix HPVM did not initialize certain fields in buffer structure and all IOs thus looked as if originating from pid=0 (swapper).

[Tonny Sejr Kromann]

Hi

We use lvm and present logical volumes to the vm's. It has shown that using only one LUN from the SAN to one lvol on the vm gives serious disk bottleneck. If we make 4 LUN's on the SAN and combine them with LVM and extent based striping and thereby get 4 io channels for the vm, it looks much better.
We also had to increase scsi queue depth on both the vm-host and guests.

Hope this could also help you.

--
Best regards
Tonny

[George N]

I appreciate the disk tuning suggestions, however, performance tuning is not our issue. Our issue is that activity on the VM guest's causes intensive disk I/O on the VM host. Once activity on the VM guest stops, the disk I/O on the host also stops. Clearly there is some correlation.

Ignore the running processes for a moment. Just looking at the activity lights on the drives - why do the VM Host drives light up like a Christmas tree any time we have activity on the VM Guest's? They are not sharing drives in any way - totally separate spindles entirely - so disk I/O on the VM Host should be nearly 0. But it's not.

[Tonny Sejr Kromann]

So you get activity on vg00 on the HPVM host?
Try to identify what lvols it is with sar?
Perhaps that can lead to a solution ...

--
Best regards
Tonny