Extreme performance difference between file backed and disk backed VMs

[Ben Kinder]

I think this looks like a bug to me, but wanted to see if anyone had some thoughts on this.  We are running Integrity VM 6.1 on BL860c blades, some old ones and some newer i2.  The HP-UX installations are 11iv3 Nov 2012, with appropriate patches applied.

2CPU 8GB RAM File backed disk:

dd command writing 4GB file, all zeroes, using 8kb block size = 18 second runtime

Same dd command, but overwriting the file, consistently runs between 1-2 minutes long.

If I remove the ddfile, the dd command runs again in 18 seconds.

1CPU 4GB RAM File backed disk:

dd command writing 4GB file, all zeroes, using 8kb block size = consistently over 1 minute runtimes, when overwriting the file obviously the time goes way up.

Any HP-UX Physical server, Linux Physical Server, or Linux VM (VMWare) complete the same task within the range of 12-20 seconds, regardless of the number of CPUs (tested with a single CPU Linux VM), and regardless of whether the operation is overwriting an existing file.  In the case of an overwrite, tyically it would add about 2 seconds anywhere other than Integrity VM.

Now the documentation alludes to some performance benefits from either logical volume, or raw disk backed VMs.  They also talk about flexibility vs performance, which is far from damning the file backed solution altogether.  They do not talk about astronomical differences, nor do they advise that file backed is unusable.  A number of HP folks have recently advised to use raw disks for our workloads which are coming under pressure.  They are relatively light app servers.  No high I/O databases or anything like that.

I wanted to get thoughts, or experiences from other Integrity VM customers on the disk backing options.

EDIT: passing a raw disk to the VM, I re-ran the tests

2CPU, 8GB RAM dd test completes in 16 seconds, subsequent ovrerwrite test completes in 20 seconds (file backed overwrite bug disappears).

1CPU, 4GB RAM dd test completes in 17 seconds, subsequent overwrite test completes in 20 seconds (file backed CPU contention bug disappears)

[Stan_M]

The astronomical differences are differences between writing the whole file to UFC (file cache) or not.

Sequention write to a file using dd is not particularly good test to measure disk performance especially

if the file size and the amount of memroy that you use in the tests are comparable. The difference

between the first 2 test you describe is the size of the file cache which fits the whole file in the first case.

Here you measure 2 file caches (guest and the host, which is on purpose very small) and not the disk itself.

dd against the raw dsf (/dev/rdisk/ ) would give you better idea of performance characteristics.

As far as I can say the file backing stores are rarely used, it's been known since begining that they

performance was not good enough (which was becaus eof implementation). The implementation has

changed a lot since, but most people simply stick to more 'raw' (whole disk or lvol) backing stores.

Stan

[Ben Kinder]

That is an interesting idea regarding filecache.  I added more memory to the VM, now it has 16GB.  I also changed the kernel tuning to utilize 50% of it for both min and max filecache.  This is now locked in at double the size of the test (8GB).

Re-running the test with a single CPU: first write 1min50sec (we still have the CPU contention problem)

Re-running the test with two CPUs: first write 15sec, overwrite now completes in 19sec (overwrite problem appears to be improved via filecache)

I still contend that this is bugged.  HP should document the fact that Integrity VM cannot be used with file backed stores.  The filecache does not explain why VMWare is able to achieve near physical device performance in every concievable configuration through VMDKs.  I think HP should be able to get closer to VMWare performance.

[Emil Velez_2]

These numbers do not seem too unexpected. A VM accesses a real disk or a raw logical volume as a disk much faster than if you use a file as a disk for a VM. WHen you use a file for a disk on a VM when the VM writes to its disk the VM host must go through the file system overhead to write or read the data.

I would think other hypervisors would have the same type of issue. Most people assign real disks to VMs that are san based to allow for dyanmic migration between hosts too.

I hope this makes sense.