vCenter Snapshots freeze MS-SQL server

[#1]

We're battling an issue with a virtual SQL server and vCenter sync'd snapshots.  We're not having snapshot failures, but the snap is causing just a bit of disruption so that some of our apps are timing out with VSS snap creation.  We don't have the problem when using Veeam's application consistent snaps so I think there must be something else wrong on my guest OS.  I know I can disable unnecessary VSS writers, but I'm a bit unsure which ones it would be safe to disable and still get the consistent snaps in Nimble.

Here is what we have on our list of enabled VSS writers.  There is a way to disable them using this article: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031200

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'

   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

   State: [1] Stable

   Last error: No error

Writer name: 'VSS Metadata Store Writer'

   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

   State: [1] Stable

   Last error: No error

Writer name: 'Performance Counters Writer'

   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

   State: [1] Stable

   Last error: No error

Writer name: 'System Writer'

   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

   Writer Instance Id: {27c00a72-3215-48b6-a699-7bd01b674fa8}

   State: [1] Stable

   Last error: No error

Writer name: 'SqlServerWriter'

   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

   Writer Instance Id: {c7c46f7d-8a70-4530-9445-30a5a80d7fd2}

   State: [1] Stable

   Last error: No error

Writer name: 'ASR Writer'

   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

   Writer Instance Id: {c29e355b-c672-4ef3-8d34-7e75407e96ae}

   State: [1] Stable

   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'

   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

   Writer Instance Id: {0048c718-52fd-416e-bdcc-fc6efac7c369}

   State: [1] Stable

   Last error: No error

Writer name: 'Registry Writer'

   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

   Writer Instance Id: {2c79e02d-8724-469f-8bb8-280e73a21a89}

   State: [1] Stable

   Last error: No error

Writer name: 'BITS Writer'

   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}

   Writer Instance Id: {abc3b9c3-61f6-4de9-a4de-873133b6c8d9}

   State: [1] Stable

   Last error: No error

Writer name: 'COM+ REGDB Writer'

   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

   Writer Instance Id: {f0908cd6-c87c-4dc3-9bb1-2d4b7862e538}

   State: [1] Stable

   Last error: No error

Writer name: 'WMI Writer'

   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

   Writer Instance Id: {b8fc017d-7c54-439d-a612-e70e80bb98f7}

   State: [1] Stable

   Last error: No error

Which of these are necessary in order to get a quiesced snap in VMware?

[mkieran59]

Hi Adam,

I see it's been more than 24 hours without a response to this question. Please feel free to contact support@nimblestorage.com and they'll be happy to help you resolve the time-out problem.

Thanks for being part of the NimbleConnect community, and sorry we couldn't get this one answered promptly.

Michael

[dbauder92]

I would like to add that they are not the only ones seeing / experiencing this issue.  We too have had to deal with this and would appreciate a response a fix to this problem.

Thanks

[#4]

Thanks for the responses Michael and Dan.  We did have a ticket open with support, but it didn't really lead us anywhere. 

[dbauder92]

Further testing as led me to believe this is a Nimble Schedule replication issue:

Take a VMware snapshot including memory and quiescing the file system - no errors

From Nimble volume management click on the Take Snapshot button - no errors

Wait for the schedule Nimble protection scheduled snapshot / replication to occur - errors occur

To me this pretty much screams there is a bug in the protection schedule routines, or they are doing something much differently from the take snapshot button.  Either case is bad news.  Hello Nimble, time to fix this.

[#6]

This is not the issue we have.  We get the same freeze/pause with schedule, manual sync'd snaps from the array, or by checking the quiesce box in VMware.  We don't ever use the "snap memory" option and my understanding is that the Nimble snaps don't use that option either.

[dbauder92]

Understood, are you seeing the Windows event log errors about disk being

surprised removed and the SQL Server service failing? If not is sounds

like there may be more than one problem here.

Dan Bauder, VCP 3, 4, 5, DCV

UNC Charlotte, Information & Technology Services

9201 University City Blvd., Charlotte, NC 28223

Phone: 704-687-0274

dbauder@uncc.edu <username@uncc.edu>* |* http://www.uncc.edu

[#8]

Nope, the VM just pauses a bit longer than I'd like and some applications timeout when connecting to the DB.

[milovanov88]

Hello Adam,

I sincerely apologize the information shared with you during the course of the case was not satisfactory answer to your question. I would like to clarify the behavior from the Nimble Storage Array perspective.

When Nimble Storage makes an API call to vCenter to create a snapshot, the array has to wait until vCenter reported to array that freeze was successful before array can take a snapshot and only then let vCenter know that it has done so. The vCenter quiesce operation depends on the volume collection volumes membership on Array, VM's contained on each volume and the VSS writers present on each VM, which at times is not optimal.

I completely understand the need for the consistent application state for the SQL database and the best course of action is the following:

a) Create separate Nimble Array volume for database and one more for log

b) Connect the VM with direct connections to Nimble Storage Array

c) Move the DB to database and log to log volumes on Nimble Storage array

d) Install the Windows Toolkit on the VM which is directly attached

e) Disable vCenter sync on the Operating System , which is on VMware datastore

f) Enable Nimble Array SQL VSS integration on the volume collection with DB and log volumes for this VM

With direct connection to Nimble Storage array, the quiesce time is minimized because the hypervisor layer is by-passed. Since array is only engaging single VSS writer on a single VM, the time to quiesce is also minimized.

Please let Nimble Storage Support know if you need any assistance in completing above steps and ensuring the snapshots for your SQL server are application-consistent.

[milovanov88]

Hello Dan,

I am sorry you are having an issue with your snapshots. Have you opened a case with Support to troubleshoot the issue you are having?

From the little information I have from your post, I can state the following:

Replication engine does not involve any host/guest operation and the transfer is done after the snapshot already exists on the source array.

When you are making a snapshot from the volume level, the snapshot is Nimble Array crash-consistent. The vCenter is not engaged because this is managed by the Volume Collection.

When you are making a snapshot through vCenter you are making a snapshot of a single VM.

When you are making snapshots via a schedule on Nimble Storage array with volume collection which is configured for vCenter sync, all the volumes under the volume collection are calling to vCenter to quiesce all the VM's within those volumes.

If the vCenter is unable to finish the quiesce operation within reasonable time (due to performance or errors), Nimble Storage array will still take crash-consistent snapshot.

You may duplicate the behavior of the scheduled snapshots by taking a manually-triggered snapshot collection under the Volume Collection in question.