VMM 2.1 causing VM crash

[dave rowland_2]

Has anyone experienced a problem when running VMM 2.1. We are suffering occasional failures when removing snapshots as part of Vmware Consolidated backup. We have also suffered from occasional vmotion failures. THey all point to a process locking the vm machine files and if you do an lsof command in the vmware host you can see the VMM agent has the files open.

[Ole Thomsen_1]

We are experiencing the same errors regarding VCB and have an open SR with VMware, but I never gave it a thought that VMM could be to blame.

I just stopped hpvmmcntsvc on all cluster members, to see if that makes a difference.

If not I'll try uninstalling the VMM agent.

Ole Thomsen

[Ole Thomsen_1]

Stopping the VMM agent did not solve any problem here.

Right now I am trying to remove a snapshot that did not get cleaned out during VCB last night, and everytime get the infamous "Operation failed due to concurrent modification by another operation"

lsof does not show any process using the files, oddly it does not report any used files in vmfs filesystems.

How did you use lsof, I tried "lsof +d /vmfs/volumes/path-to-vm" and "lsof | grep vmfs" ?

Ole Thomsen

[dave rowland_2]

I just used the command lsof

withut any parameters. You can the grep the output to filter the results based on the filename of the .vmx file.

[dave rowland_2]

We have had further snapshot issues even with VMM stopped. DId you hear anything from vmware, we have vmware support via HP but they have not come back with anything yet....

[Rob Buxton]

Have you also gone through the process of linking HPSIM with VirtualCentre?
You might want to reconfigure that so it no longer can connect to VC to further isolate the problem.

[Ole Thomsen_1]

Dave: I did run lsof without parameters and inspect the output, and further lsof with a grep of /vmfs which is a part of the path to every vm. Nothing.

A discussion of the problem, which also implies vms shutting down when deleting snapshots, can be found here

http://www.vmware.com/community/thread.jspa?messageID=556970羪

http://www.vmware.com/community/thread.jspa?messageID=553365熕

We just had the first dataloss (that we know of) caused by this. After numerous failed attempts to delete a leftover snapshot, the deletion reported a timeout but removed the snapshot.
Unfortunately this crashed the Oracle database running in the vm, reporting a lot of disk errors and broken files. Spent a day restoring export (as our VCB did not succeed for days).

Rob: You might have a point there, I will remove the VC settings immediately. Thanks.

Ole Thomsen

[Ole Thomsen_1]

Update:

Apart from the problem described earlier (manually deleting snapshot), it seems that stopping VMM helped a lot!

During the weekend our full VCB backup succeeded without errors for the first time ever, and no snapshot files left :-)

We only changed one other thing in the ESX cluster, shutting down a host that had some faulty memory modules.

Ole Thomsen

[Ole Thomsen_1]

Vmware has confirmed to us that there is a problem with VMM, VCB (perhaps snapshots generally) and EVA SAN.

They are working on a solution with HP.

So far we have decided to stop VMM agents in our ESX cluster. Quite a pain.

Ole Thomsen

[Rob Buxton]

Ole,
Thanks for the update, any idea if this might also include the issue of the "disk is corrupt" when creating a VM on EVA SAN's?

I know you've contributed to the thread on the vmware forum.

cheers,

Rob.

[dave rowland_2]

ole

are you able to share the VMware case number and/or details so I can ask HP/vmware about this bug as they have not come back with anything even thogh I have had a case open for a long time with them.

[Ole Thomsen_1]

My collegue, who got the explanation from a supporter over the phone, said that they were not very clear on the reason for the errors.

But it had something to do with a driver for MSA also being used for EVA, which reacts a little different.

To me it sounds like other various strange disk problems we had over time (and we had quite a bit too many) could have a root here.

A solution will include updates for VMM as well as ESX.

All according to VMware :-)

Dave, mail me - ot at networks.dk

Ole Thomsen

[dave rowland_2]

HP have confirmed that there are issues with Insight Agents and HP+VMWare are working on a solution.

According to HP "HP VMM and Insight Manager installs agents like VMM Agent, FC (fiber channel agent) and SIM Agent, now what happens is when these agents are polled this clashes with the VCB hence causing VCB to shut down VM's"

Sounds like HP agents are locking resources, this is what we told them 2 months ago that what we thought the problem was!!!!

Workaround is to disable VMM Agent, FC Agent and SIM Agent.

.......makes you wonder wht we purchased VMM Agent, it was 6 months late, causes MAJOR issues and now we cant use it.... Will HP give us a refund???????

[Rob Buxton]

In this months list of VMWare ESX patches there's one listed as being for MSA's ;
ESX-9865995
I'd flagged this as not for us as we do not have MSA's. Any idea if this is related or is this a completely separate issue.

It does sound as though it's different.

[Ole Thomsen_1]

My experience is that removing (or just stopping) the VMM agent cures the problems.

We do not disable any other agent, but we did disable the VirtualCenter connection from HPSIM.

Ole Thomsen

[Ed_178]

I seem to be having a problem related to the issues mentioned in this thread. We do not use VCB however upon installing ESX patches up to :
ESX-1271657 Patch (and all prior patches)
VMM no longer works on ANY of my esx3 hosts. They aren't recognized in SIM as being VMWare hosts any longer. Is HP aware of this? If so what is the ETA for a fix/workaround? Why hasn't there been and HP email alerts about this issue? Any answers/suggestions are welcome.

[dave rowland_2]

Well, its been a year since VMWARE released ESX3 and still HP dont have a working VMM.THey admit there is a bug but have not fixed it, neither can they tell us when. Judging on past experience vmware will be on version 4 before they fix it !

Worst piece of software we wasted our money on..........

[Ole Thomsen_1]

I'm a bit disappointed too.

At least HP should tell us when a working version will be available.

Ole Thomsen

[Rob Buxton]

Yes, and I've just realised that some of the issues I'm seeing recently is probably this as well.
So I'm about to disable HP VMM.
I've also got a call logged with HP as they support VMware for us. I've pointed back to this thread.
Has anyone got any calls logged with either HP or VMware regarding this?
Could you post or e-mail me directly at rob dot buxton at wcc dot govt dot nz.
We may get a bit more traction if we tie the logged calls together.

[Michael Aldo]

We are experiencing the same problem of the VM shutting down after removing the snap. I am installing VMM 2.2 today. Has anybody seen this issue with VMM 2.2 installed?

[Ole Thomsen_1]

Wow, I hadn't seen that a 2.2 was released, thanks.

There might be a chance that, along with the new management agents for ESX 7.8.0, this mess is resolved.

Ole Thomsen

[Rob Buxton]

Ole,
Likewise, I've a call open with HP / VMware and no mention of VMM 2.2 even though we've related the VMM issue to the VMotion issue we're seeing.
I'm not feeling that optimistic so I'll proceed with caution!

[Ole Thomsen_1]

I installed the latest ESX patches, new HPASM/firmware and upgraded VMM to v2.2 - no progress :-(

In a small VCB job 2 servers out of 3 failed:

[2007-07-23 02:01:13.489 'BlockList' 1312 error] Generic block list error code.

[2007-07-23 02:01:13.504 'vcbMounter' 1312 error] Error: Failed to open the disk: Unspecified error

[2007-07-23 02:01:13.504 'vcbMounter' 1312 error] An error occured, cleaning up...

[2007-07-23 02:01:28.299 'vcbMounter' 1312 warning] Snapshot deletion failed. Attempting to clean up snapshot database...

Oh well...must disable the VMM agents again. What a mess.

Ole Thomsen

[Ole Thomsen_1]

Does VMM v3 solve this problem?

Ole Thomsen

[Rob Buxton]

I don't have VCB, but we do seem to have an issue where VMM interferes with VMotion.

We're not on the latest ESX 3.0.2 release as yet, but we did try VMM 3.0. It did seem better, but we still saw a VMotion failure during a test with VMM enabled.

Also we did a cold migration from an ESX 2.5 to our 3.0.1 infrastructure and that also failed while VMM was enabled. It succeeded on the second attempt with VMM disabled.

We've gone back to having this disabled and I'm seriously re-evaluating whether to continue with it. With SMP now split and not requiring HPSIM I'm not convinced there are sufficient benefits.