vCenter Snapshots freeze MS-SQL server

[#1]

I appreciate your reply.  Your recommendations are similar to those of support, however, we have just P2V'd this machine and migrated it to VMDKs for the data/log volumes so we aren't interested in doing the reverse of this.  We were not aware that the snapshots would act differently otherwise we wouldn't have chosen this route, but it is where we are currently.

I'd really hoped to find a way to get the quiesced snaps to happen more efficiently so that we could keep our RPO low with Nimble replication like we had previously when it was a physical server directly connected to the array.

[dbauder92]

Unfortunately I have had zero luck with support when reporting these kinds

of issues in the past. I see nothing to be gained by going that route

You can have a stand alone volume that is protected and don't have the

ability to take a volume collection snapshot. In this case I would hope

the take a snapshot button would use the settings from the stand alone

protection and integrate with vCenter. It does not appear to work that way.

Thanks for the info.

Dan Bauder, VCP 3, 4, 5, DCV

UNC Charlotte, Information & Technology Services

9201 University City Blvd., Charlotte, NC 28223

Phone: 704-687-0274

dbauder@uncc.edu <username@uncc.edu>* |* http://www.uncc.edu

[chrismcqueen62]

Hi Adam/Dan,

     This sounds very similar to a problem which we experienced and went through the process or Nimble Support, VMWare Support and finally on to Microsoft...

In Windows Event log we had errors about disks being surprised removed, we actually found that all VMs were having these however it was only the latency sensitive ones which we noticed it on.

We use Veeam along with VMWare, what we found was that Veeam did not display any errors as it uses its own VSS, VMWare calls on Microsoft VSS, Nimble calls on VMWare which in turn calls on Microsoft VSS

This link relates to the original issue which we experienced, and the proposed workaround

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006849

We tested the theory by building new VMs to test with, both as MBR and GPT, we found that the MBR VMs continued to have the issue, the GPT ones were resolved. Quite a lot of work as its a full infrastructure rebuild but it did resolve the problems

Further to this we had the same issues with EMC AppSync with RecoverPoints, same solution for us

Thanks

[dbauder92]

Chris - Thanks for the input.  We've got about a dozen VMs running SQL server that we placed on volumes with vCenter Synchronization implemented.   This is where we are seeing the issues, yes they do have MBR disks.  If I understand your post is sounds like rebuilding these with GPT disks might help us out.  Or do we have to do both the GPT disks and implement the VMware KB work around as well?

Thanks

[milovanov88]

Chris,

Thank you very much! This is very good bit of information.

[Nick_Dyer]

Hello Adam,

Sadly this is the downside to using VMDKs and relying on vCenter snapshots, and it's why for ultimate best practice we do recommend using in-guest iSCSI. Having said that, I'm told there are significant improvements to vCenter snapshot procedures in vSphere 6. I personally haven't tested this myself - here's a blog detailing the benefits from a friend at Veeam: http://www.virtualtothecore.com/en/vsphere-6-snapshot-consolidation-issues-thing-past/.

All of the above disappears with Virtual Volumes and NimbleOS 3 you'll be pleased to hear.

[chrismcqueen62]

Hi Dan,

Unfortunately the answer for us was to rebuild the VMs as GPT, we have since updated all templates in VMWare to use GPT as a standard and do not use MBR anymore

On the VMWare link, the first Microsoft KB 2853247, Resolution 2 is the GPT fix. Resolution 1 outlines having a single partition on an MBR disk which is not possible on newer OS's so ruled that straight out.

It sounds exactly along the lines of what we experienced and we went through all the proper support channels to get this outcome and it was VMWare which pointed us to this article, I'm sure you would but spin up a VM and give it a test to ensure it is the same issue which you have

Thanks

[dbauder92 · Accepted solution]

Thanks for your insights. Yes I could / would have gone down the support

role, I will take your example and see if that solves our issue too. If

not I guess I will have to do the support route.

Thanks again and have a wonderful weekend - Cheers

Dan Bauder, VCP 3, 4, 5, DCV

UNC Charlotte, Information & Technology Services

9201 University City Blvd., Charlotte, NC 28223

Phone: 704-687-0274

dbauder@uncc.edu <username@uncc.edu>* |* http://www.uncc.edu

[dukeboles26]

Adam,

I dealt with this since we purchased our Nimbles. I have solved the problem. It is as @Nick Dyer says. If your application is at all latency sensitive you must use direct attached iscsi disks. There are lots of gotchas that I would be glad to share if you decide to go that route. There are many but one is that you don't want to go with in-guest iscsi if you use Vmware Site Recovery Manager and array based recovery. In that case you need physical mode Raw Device Mapping. These are identical to in guest on the Nimble side but instead of the iscsi initiator in the guest attaching the hosts attach instead and present to the VM.

There are many more gotchas I am glad to share if you want.

Duke

edit: P.S. I love Nimble support and they do a great job but if you take this to them they will lead you down many wrong paths. I got the final key pieces of info from an internal engineer. I was only offered that after complaining to my account rep. Once I had access to this person I had the whole thing solved in two weeks after eight months of gut wrenching struggle bus action.

[#10]

Hi Nick,

Sorry I let this one go dark.  I did upgrade to ESXi6 and the snapshots seem to be better, however I am having an unrelated problem where the OS doesn't seem to be recognizing the quiesced snapshots as there are no entries in the application log indicating the freeze, thaw, and I/O resumed like we would expect.  We're working with Microsoft now to see who wins the finger pointing match - Microsoft or VMware. 

I have a question about the VVols comment you had above.  What will the migration to that look like?  Will there be a way to convert the volumes over to this, or will it be a manual data migration in the OS to the new volumes?  (I fear it is the latter)