Win2k3 VM performance degraded after migration to ESXi 4 from VMWare Server 1.0.7.0

[Al_56]

We've been using VMWare Server for years, and have been transitioning over to ESXi 4.0 (free version).

Things have gone great with the following exception.

I migrated a Win2k3 64 VM from VMWare Server 1.0.7 to ESXi thin provisioned (VM was stored on an NFS server).

I copied the VM to a stand alone ESXi 4.0 server (HP DL360 G5 6x146GB 10k drives, 400i RAID card with 512 MB cache, 2 x quad core CPU's, 16GB RAM using the servers local storage.

The VM's performance has been abysmal, about 10 times slower than the same VM on a busier VMWare 1.0.7 server!

There are only 3 VM's running in total on the new server, each with only 3GB RAM, and single CPU, so I doubt it's a hardware performance issue.

The VM is thin provisioned on the local Storage which is VMFS. Could this be the issue?

Should the VM be thick provisioned? If so, is it easy to convert to thick provisioning?

I migrated the VM by creating a monolithic drive (on VMWare server 1.0.7), copied that to the NFS server, and changed the VMDK file to show VMFS instead of FLAT).

I then converted that disk to thin provisioned, and copied it to the ESXi 4.0's local Storage.

I then built a new VM with the same specs and config on the VMWare ESXi 4.0 using the created thin provisioned disk.

Is it the thin provisioning on Local Storage (VMFS)?, what could I have missed?

Any advice would be appreciated.

Thanks in advance,

Alan

[Al_56]

Come to think of it, I did upgrade the firmware on the server (last week) prior to seeing the performance degradation.

I don't have any HP management tools on the server, so I can't even check the hardware status (do I really have the BBWC on the P400i RAID card on that server).

How can i check the servers status, running ESXi 4.0 without the management tools?

Thanks,

Alan

[Uwe Zessin]

It is really a good idea to use a BBWC. And if you're thin-provisioning it will cause additional I/O during the first time allocation.

[Al_56]

I think it is installed, but now I notice sluggishness across the other VM's as well.

They are not even live yet, and with only 3 VM's the system should fly, but even the VM UI is shaky (disconnects), so I'm wondering whether I have a hardware issue, and need to check the hardware configuration (I think I have the BBWC but am not sure).

Without the HP management tools installed how do I check the hardware remotely? Any ILO commands to check the hardware configuration or status?

Are there any HP management utils I can install on the free version of ESXi 4.0?

Thanks in advance,

Alan

[Uwe Zessin · Accepted solution]

It could also be a network problem if by 'VM UI' you mean the vSpere client which loses access.
The iLO has _some_ access to the server's hardware and some log information, but for deep insight you do need access via the operating system.

For hardware monitoring on ESXi you need the vendor's CIM providers:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3884319&prodNameId=3884320&swEnvOID=4040&swLang=13&mode=2&taskId=135&swItem=MTX-8758956156b94fc1a59e078961

In the past they were also bundled with ESXi on the VMware pages and there is one on HP's software kiosk:
https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPVM06

[Rob Buxton]

And probably as just a first cut check simple things like the duplex setting on the NIC ports.
That should be visible from the VI Client.

[Al_56]

Thanks,

I did install the "HP ESXi Offline Bundle", and i now see info about the drives and the BBWC (fully charged).

I have not configures SNMP on the VM yet, but will. Thanks for the Veeam recommendation. I have it installed, but did not use it on that server for monitoring yet.

I did use the smart start CD on the server to check the RAID cache settings and found that fast write was not enabled, so I enabled that (server has redundant power supplies and separate power circuits, as well as the BBWC), and RAID was also set to 75% write. What setting should I set of that parameter for ESXi and VM's?

I also recreated the vmdk files and kept them at hardware version 4, but did not notice any performance improvements from that.

How is using the P2V converter better than the conversion method I used? Should I see performance or stability improvements out of using it for the windows VM's?

SO enabling fate write on the RAID has given a performance improvement, but still not to where (I think) this server is capable of delivering.

I'll do the P2V conversion if you guru's think it is better than the conversion method I used, and will provide better performance.

Thanks in advance,

Al

[wobbe]

I'm not sure if your method of migtation will upgrade you to the latest virtual hardware. If it does then there is no real issue but if it doesn't it would be shame to keep running with the older virtual hardware.

[Al_56]

I did create the new VM using the latest (version 7) when I used the migrated disk image.

The only issue that showed up was that when I assigned the same IP's on the new virtual NIC's i received an error message stating that the same IP was assigned to another NIC that is currently hidden, so the virtual hardware upgrade process does not delete the old virtual NIC's. I was still able to use the IP addresses, but that makes you wonder what else is lurking under the skin.....

The resolution for the IP issue is to show hidden devices in the device manager and delete them. That does not work so well. The other way is to go into the registry and delete the old hardware NIC devices.

The Linux VM's seem to be performing fine, but they were created brand new on ESXi. The Windows VM's would be a real pain, and very time consuming to configure on a new ESXi VM, so I'm trying to avoid that as much as possible.

I may not have mentioned that the physical RAID is configured using RAID 6, so performance is not as fast as RAID 5 would provided, but should still be very adequate for hosting only 3 - 4 VM's which are not very CPU, memory or disk intensive...

Al

[Al_56]

Last night i ran the P2V conversion, and had the same behavior as the method I'd used to create the VM as a version 7 hardware VM using the converted disk image.

Lot's of UN-recognized hardware until the new tools were installed. Had to re-authenticate Windows with MS due to the hardware changes.

Same IP address overlay issues due to the old NIC's still on board but hidden in the device manager.

I'm currently copying the VM to the data center, to try its performance, but it did not seem much faster (if at all) than the VM's created via the other conversions.

Thanks for all advice,

Al

[Uwe Zessin]

> Lot's of UN-recognized hardware until the new tools were installed.
> so the virtual hardware upgrade process does not delete the old virtual NIC's

Right, it is just inactive in the Windows hot-plug database.

> The resolution for the IP issue is to show hidden devices
> in the device manager and delete them.
> That does not work so well.

I use:
SET DEVMGR_SHOW_DETAILS=1
SET DEVMGR_SHOW_NONPRESENT_DEVICES=1
devmgmt.msc

view->hidden devices
(or something like that - I'm working with a German Windows XP right now ;-)

...
Those PCI-to-PCI bridges look great, don't they ;-)
ESX did provide a 6-slot PCI motherboard for its virtual machines. One slot was taken for the video card and the rest could be assigned to parallel SCSI adapters or NICs.

In ESX 4 you can configure up to 10(!) NICs.

> Had to re-authenticate Windows with MS due to the hardware changes.

Yep, too many changes at once and no volume license, right?

...
Some other things:

- For some reason the Converter _loves_ to stick a Buslogic SCSI adapter into the VM, but today's VMs should use the LSI Logic adapter.

- The Converter adds USB, serial and parallel devices and(!) connects them to the host's hardware. I strongly recommend to shut down and remove them from the virtual hardware.

- Make sure that virtual floppy and virtual CD/DVD are disconnected unless you really need them.