topic Re: VMM 2.1 causing VM crash in Server Management - Systems Insight Manager

VMM 2.1 causing VM crash

dave rowland_2 — Mon, 12 Feb 2007 16:42:54 GMT

Has anyone experienced a problem when running VMM 2.1. We are suffering occasional failures when removing snapshots as part of Vmware Consolidated backup. We have also suffered from occasional vmotion failures. THey all point to a process locking the vm machine files and if you do an lsof command in the vmware host you can see the VMM agent has the files open.

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Tue, 13 Feb 2007 05:55:47 GMT

We are experiencing the same errors regarding VCB and have an open SR with VMware, but I never gave it a thought that VMM could be to blame.

I just stopped hpvmmcntsvc on all cluster members, to see if that makes a difference.

If not I'll try uninstalling the VMM agent.

Ole Thomsen

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Tue, 13 Feb 2007 12:26:05 GMT

Stopping the VMM agent did not solve any problem here.

Right now I am trying to remove a snapshot that did not get cleaned out during VCB last night, and everytime get the infamous "Operation failed due to concurrent modification by another operation"

lsof does not show any process using the files, oddly it does not report any used files in vmfs filesystems.

How did you use lsof, I tried "lsof +d /vmfs/volumes/path-to-vm" and "lsof | grep vmfs" ?

Ole Thomsen

Re: VMM 2.1 causing VM crash

dave rowland_2 — Wed, 14 Feb 2007 16:46:52 GMT

I just used the command lsof

withut any parameters. You can the grep the output to filter the results based on the filename of the .vmx file.

Re: VMM 2.1 causing VM crash

dave rowland_2 — Wed, 14 Feb 2007 16:48:47 GMT

We have had further snapshot issues even with VMM stopped. DId you hear anything from vmware, we have vmware support via HP but they have not come back with anything yet....

Re: VMM 2.1 causing VM crash

Rob Buxton — Wed, 14 Feb 2007 19:54:17 GMT

Have you also gone through the process of linking HPSIM with VirtualCentre?
You might want to reconfigure that so it no longer can connect to VC to further isolate the problem.

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Thu, 15 Feb 2007 01:28:54 GMT

Dave: I did run lsof without parameters and inspect the output, and further lsof with a grep of /vmfs which is a part of the path to every vm. Nothing.

A discussion of the problem, which also implies vms shutting down when deleting snapshots, can be found here

http://www.vmware.com/community/thread.jspa?messageID=556970羪

http://www.vmware.com/community/thread.jspa?messageID=553365熕

We just had the first dataloss (that we know of) caused by this. After numerous failed attempts to delete a leftover snapshot, the deletion reported a timeout but removed the snapshot.
Unfortunately this crashed the Oracle database running in the vm, reporting a lot of disk errors and broken files. Spent a day restoring export (as our VCB did not succeed for days).

Rob: You might have a point there, I will remove the VC settings immediately. Thanks.

Ole Thomsen

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Mon, 19 Feb 2007 03:09:00 GMT

Update:

Apart from the problem described earlier (manually deleting snapshot), it seems that stopping VMM helped a lot!

During the weekend our full VCB backup succeeded without errors for the first time ever, and no snapshot files left :-)

We only changed one other thing in the ESX cluster, shutting down a host that had some faulty memory modules.

Ole Thomsen

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Wed, 07 Mar 2007 08:16:02 GMT

Vmware has confirmed to us that there is a problem with VMM, VCB (perhaps snapshots generally) and EVA SAN.

They are working on a solution with HP.

So far we have decided to stop VMM agents in our ESX cluster. Quite a pain.

Ole Thomsen

Re: VMM 2.1 causing VM crash

Rob Buxton — Wed, 07 Mar 2007 14:42:45 GMT

Ole,
Thanks for the update, any idea if this might also include the issue of the "disk is corrupt" when creating a VM on EVA SAN's?

I know you've contributed to the thread on the vmware forum.

cheers,

Rob.

Re: VMM 2.1 causing VM crash

dave rowland_2 — Wed, 07 Mar 2007 15:41:43 GMT

ole

are you able to share the VMware case number and/or details so I can ask HP/vmware about this bug as they have not come back with anything even thogh I have had a case open for a long time with them.

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Thu, 08 Mar 2007 00:15:17 GMT

My collegue, who got the explanation from a supporter over the phone, said that they were not very clear on the reason for the errors.

But it had something to do with a driver for MSA also being used for EVA, which reacts a little different.

To me it sounds like other various strange disk problems we had over time (and we had quite a bit too many) could have a root here.

A solution will include updates for VMM as well as ESX.

All according to VMware :-)

Dave, mail me - ot at networks.dk

Ole Thomsen

Re: VMM 2.1 causing VM crash

dave rowland_2 — Mon, 12 Mar 2007 14:50:15 GMT

HP have confirmed that there are issues with Insight Agents and HP+VMWare are working on a solution.

According to HP "HP VMM and Insight Manager installs agents like VMM Agent, FC (fiber channel agent) and SIM Agent, now what happens is when these agents are polled this clashes with the VCB hence causing VCB to shut down VM's"

Sounds like HP agents are locking resources, this is what we told them 2 months ago that what we thought the problem was!!!!

Workaround is to disable VMM Agent, FC Agent and SIM Agent.

.......makes you wonder wht we purchased VMM Agent, it was 6 months late, causes MAJOR issues and now we cant use it.... Will HP give us a refund???????

Re: VMM 2.1 causing VM crash

Rob Buxton — Mon, 12 Mar 2007 19:07:59 GMT

In this months list of VMWare ESX patches there's one listed as being for MSA's ;
ESX-9865995
I'd flagged this as not for us as we do not have MSA's. Any idea if this is related or is this a completely separate issue.

It does sound as though it's different.

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Tue, 13 Mar 2007 00:14:22 GMT

My experience is that removing (or just stopping) the VMM agent cures the problems.

We do not disable any other agent, but we did disable the VirtualCenter connection from HPSIM.

Ole Thomsen

Re: VMM 2.1 causing VM crash

Ed_178 — Tue, 13 Mar 2007 19:19:59 GMT

I seem to be having a problem related to the issues mentioned in this thread. We do not use VCB however upon installing ESX patches up to :
ESX-1271657 Patch (and all prior patches)
VMM no longer works on ANY of my esx3 hosts. They aren't recognized in SIM as being VMWare hosts any longer. Is HP aware of this? If so what is the ETA for a fix/workaround? Why hasn't there been and HP email alerts about this issue? Any answers/suggestions are welcome.

Re: VMM 2.1 causing VM crash

dave rowland_2 — Mon, 18 Jun 2007 15:28:16 GMT

Well, its been a year since VMWARE released ESX3 and still HP dont have a working VMM.THey admit there is a bug but have not fixed it, neither can they tell us when. Judging on past experience vmware will be on version 4 before they fix it !

Worst piece of software we wasted our money on..........

Re: VMM 2.1 causing VM crash

Ole Thomsen_1 — Mon, 18 Jun 2007 15:35:00 GMT

I'm a bit disappointed too.

At least HP should tell us when a working version will be available.

Ole Thomsen

Re: VMM 2.1 causing VM crash

Rob Buxton — Sun, 24 Jun 2007 16:00:56 GMT

Yes, and I've just realised that some of the issues I'm seeing recently is probably this as well.
So I'm about to disable HP VMM.
I've also got a call logged with HP as they support VMware for us. I've pointed back to this thread.
Has anyone got any calls logged with either HP or VMware regarding this?
Could you post or e-mail me directly at rob dot buxton at wcc dot govt dot nz.
We may get a bit more traction if we tie the logged calls together.

Re: VMM 2.1 causing VM crash

Michael Aldo — Thu, 19 Jul 2007 12:10:12 GMT

We are experiencing the same problem of the VM shutting down after removing the snap. I am installing VMM 2.2 today. Has anybody seen this issue with VMM 2.2 installed?