Inconsistent VSS backup behaviour

[FabrizioDV]

Hello all,

i have a two node Simplivity 325 G10 V2 Plus cluster in production.

The backup policy is one backup per hour from 7:00 AM to 7:00 PM

For the Windows VM that nedd VSS i have an inconsistent behaviour: i have a very high rate of

failed VSS backup

Example : 

Alarm SimpliVity VSS Backup Snapshot Failure. Crash-consistent Backup Taken Instead. on Virtual Machine : SVT-DC-001 is Warning

This email is to notify you that an alarm has been triggered in your vCenter:
[Warning] Alarm SimpliVity VSS Backup Snapshot Failure. Crash-consistent Backup Taken Instead. on VM XXX
because VM XXX VSS snapshot failed to execute. Crash-consistent backup taken instead..

Alarm name

SimpliVity VSS Backup Snapshot Failure. Crash-consistent Backup Taken Instead.

Description

VM VSS backup failed. Crash-consistent backup taken instead.

Target

VM XXX

Status

Warning (previous status: Normal)

Triggered time

11/22/2022 01:00:25 PM

This can happen multiple time in a row randomly or not happening for 1/2 days

In this policy i have 3 VM

VM1 is Windows Server 2022 DC and is a Domain Controller

VM2 is Windows Server 2022 DC with SQL Server 2019

VM3 is Windows Server 2016 with SQL Server 2014 Express

I have created a domain user and saved the password for VSS backup

The user is member of this domain group : Domain Admins, Domain Users, Enterprise Admins, Schema Admins, Backup Operators

The user on the machine are member of these groups on the local machine : Administrators, Backup Operators

All thr 3 machine are up to date and patched, VmWare tools are version 12320

The Simplivity ESXi host are version 7.0.3, build 19193900

The Vcenter is version 7.0.3.00800

The cluster was installed on september 2022, HPE Omnistack version 4.1.2.185 , HPE Simplivity Plugin version  4.1.2.162

Where i can gather detailed log?  In the Event Viewer of the affected virtual machine there is no error....  and over the Internet i can't find info...

What shloud i try ?

Thanks

[Deepak_K]

Hi FabrizioDV,

Thank you for writing to us.

We request you to check the guidelines mentioned in the below document link. Refer Page Nos. 69-70 & 80-82

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00123771en_us

Please find the below KB articles which would help you to fix the issue:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sf000043880en_us

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sv11641en_us

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sf000044124en_us

Note: You may notice an older version of OVC mentioned in the document. Please ignore it and consider only the steps.

Refer Page No. 156 in the below document:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00123769en_us

If none of the above shared documents does not resolve the issue. Please share the serial number of the server in a private post for further investigation.

Regards
Deepak
HPE

[FabrizioDV]

Hi Deepak

i have also responded on the reddit thread

in the page 70 of this document https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00123771en_us

i read : "To use VSS with OmniStack, the virtual machine must use a Windows Server 2008R2, 2012R2, or 2016 operating system and have VMware Tools installed. In addition, the supported VSS-aware applications are SQL Server 2008 R2, 2012, 2014, and 2016."

This implies that my VM (one WS 2022 with SSL Server 2019 and another one WS 2022 that is a DomainController) are both out of the ofiicially supported matrix ?

The official support for WS 2022 and SSL Server 2019 is planned ?

howewer i have tried on one of the affected machine (WS 2022 with SSL Server 2019) the steps on this link : https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sf000043880en_us

I followed the STEPS

STEP 1) Eliminate Simplivity from VMware Snapshot creation process:1.1) Create SnapShot:

Right-click on the VM in the vCenter.

Choose snapshot/Take snapshot.

Provide name ie Test.

Uncheck 'Snapshot VM's memory'.

Check 'Quiesce guest file system'.

Click OK.

1.2) Remove Snapshot:

Right-click on the VM in the vCenter.

Choose snapshot/Snapshot Manager.

Delete all.

The STEP 1 ha worked without issues

STEP 2) Create a VSS backup from Simplivity Backup >> VSS Application Consistent - Microsoft Volume Shadow Copy Service (VSS) Method.

I have tried the STEP 2 multiple time and sometimes work, sometimes it produces the error "[Warning] Alarm SimpliVity VSS Backup Snapshot Failure. Crash-consistent Backup Taken Instead. on VM XXX"

STEP 3) Confirm the user is a Local Admin on the server. >>> YES

STEP 4) Modify Registry User Account Control: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System.The key that we are interested in is ConsentPromptBehaviorAdmin: we want to be set to 0 (Elevate without prompting).

>>> YES, already in place

STEP 5) Modify Local Security Policy User Account Control:Local Security Policy application,Click Local Policies, Security Options.Right-click on the User Account Control: Run all administrators in Admin Approval Mode setting,Select Properties.Select the Disable radio button, and click OK.Reboot the guest VM for the changes to take effect.

>>> YES, already done

STEP 6) Manually Create A Volume Shadow Copy of all volumes

***************** START LOG *****************
DISKSHADOW> reset
DISKSHADOW> set context persistent
DISKSHADOW> set verbose on
DISKSHADOW> begin backup
DISKSHADOW> add volume c:\
DISKSHADOW> add volume e:\
DISKSHADOW> add volume g:\
DISKSHADOW> add volume h:\
DISKSHADOW> add volume i:\
DISKSHADOW> add volume f:\
DISKSHADOW> ADD Volume \\?\Volume{f18617de-b581-4655-9304-ae7d6efb2c6e}\
DISKSHADOW>
DISKSHADOW>
DISKSHADOW> createExcluding writer "Shadow Copy Optimization Writer", because all of its components have been excluded.
Component "\BCD\BCD" from writer "ASR Writer" is excluded from backup,
because it requires volume which is not in the shadow copy set.

* Including writer "Task Scheduler Writer":
+ Adding component: \TasksStore

* Including writer "VSS Metadata Store Writer":
+ Adding component: \WriterMetadataStore

* Including writer "Performance Counters Writer":
+ Adding component: \PerformanceCounters

* Including writer "System Writer":
+ Adding component: \System Files
+ Adding component: \Win32 Services Files

* Including writer "SqlServerWriter":
+ Adding component: \XXX\master
+ Adding component: \XXX\model
+ Adding component: \XXX\msdb
+ Adding component: \XXX\DB1
+ Adding component: \XXX\DB2
+ Adding component: \XXX\DB3

* Including writer "ASR Writer":
+ Adding component: \ASR\ASR
+ Adding component: \Volumes\Volume{b842e0b1-bae8-4300-9bb9-b2e02c06b9fa}
+ Adding component: \Volumes\Volume{2f3ac0a7-b99e-4627-aa7a-502bfb9b69e7}
+ Adding component: \Volumes\Volume{10373d4e-61a8-4b27-95a0-7493d60dd561}
+ Adding component: \Volumes\Volume{cb7879d6-e2f5-4e41-b3df-4984e3275381}
+ Adding component: \Volumes\Volume{083f9d7b-f5a5-4f4a-ab1b-9b1e003d43c7}
+ Adding component: \Volumes\Volume{9110a541-79d0-4811-8146-c9223419cf62}
+ Adding component: \Volumes\Volume{dc1dd100-14f1-4613-b93e-3314689009f7}
+ Adding component: \Volumes\Volume{f18617de-b581-4655-9304-ae7d6efb2c6e}
+ Adding component: \Disks\harddisk0
+ Adding component: \Disks\harddisk1
+ Adding component: \Disks\harddisk3
+ Adding component: \Disks\harddisk5
+ Adding component: \Disks\harddisk2
+ Adding component: \Disks\harddisk4

* Including writer "Registry Writer":
+ Adding component: \Registry

* Including writer "COM+ REGDB Writer":
+ Adding component: \COM+ REGDB

* Including writer "WMI Writer":
+ Adding component: \WMI

Alias VSS_SHADOW_1 for shadow ID {9bbaacc0-0e3c-4f02-8a6d-f85628e1b4df} set as environment variable.
Alias VSS_SHADOW_2 for shadow ID {e90db7ce-f92f-4d35-be7c-8b6b6bebd2f3} set as environment variable.
Alias VSS_SHADOW_3 for shadow ID {e4c52e0e-a7b4-40d0-b6fe-17c458479fe1} set as environment variable.
Alias VSS_SHADOW_4 for shadow ID {e6b65e1b-443c-4104-9d3e-f7996d8df7bb} set as environment variable.
Alias VSS_SHADOW_5 for shadow ID {49e132fc-0893-491a-af48-ba51568619da} set as environment variable.
Alias VSS_SHADOW_6 for shadow ID {69daeb61-c5e6-44e1-b960-4dd50de97340} set as environment variable.
Alias VSS_SHADOW_7 for shadow ID {2e812c60-964a-4c97-b2a9-7e9475fd0a26} set as environment variable.
Alias VSS_SHADOW_SET for shadow set ID {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} set as environment variable.
Inserted file Manifest.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file BCDocument.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM0.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM1.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM2.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM3.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM4.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM5.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM6.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM7.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM8.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file WM9.xml into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab
Inserted file diskshadow-28-11-2022.txt into .cab file 2022-28-11_-35-07_SVT-GEST-SERVER.cab

Querying all shadow copies with the shadow copy set ID {87e21e4f-2960-4f4c-86a9-644d2ec1adc3}

* Shadow copy ID = {9bbaacc0-0e3c-4f02-8a6d-f85628e1b4df} %VSS_SHADOW_1%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{b842e0b1-bae8-4300-9bb9-b2e02c06b9fa}\ [C:\]
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy612
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

* Shadow copy ID = {2e812c60-964a-4c97-b2a9-7e9475fd0a26} %VSS_SHADOW_7%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{f18617de-b581-4655-9304-ae7d6efb2c6e}\ []
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy618
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

* Shadow copy ID = {e90db7ce-f92f-4d35-be7c-8b6b6bebd2f3} %VSS_SHADOW_2%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{dc1dd100-14f1-4613-b93e-3314689009f7}\ [E:\]
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy613
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

* Shadow copy ID = {e4c52e0e-a7b4-40d0-b6fe-17c458479fe1} %VSS_SHADOW_3%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{9110a541-79d0-4811-8146-c9223419cf62}\ [G:\]
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy614
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

* Shadow copy ID = {e6b65e1b-443c-4104-9d3e-f7996d8df7bb} %VSS_SHADOW_4%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{083f9d7b-f5a5-4f4a-ab1b-9b1e003d43c7}\ [H:\]
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy615
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

* Shadow copy ID = {49e132fc-0893-491a-af48-ba51568619da} %VSS_SHADOW_5%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{cb7879d6-e2f5-4e41-b3df-4984e3275381}\ [I:\]
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy616
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

* Shadow copy ID = {69daeb61-c5e6-44e1-b960-4dd50de97340} %VSS_SHADOW_6%
- Shadow copy set: {87e21e4f-2960-4f4c-86a9-644d2ec1adc3} %VSS_SHADOW_SET%
- Original count of shadow copies = 7
- Original volume name: \\?\Volume{10373d4e-61a8-4b27-95a0-7493d60dd561}\ [F:\]
- Creation time: 28/11/2022 15:35:05
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy617
- Originating machine: XXX.local
- Service machine: XXX.domain.local
- Not exposed
- Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

Number of shadow copies listed: 7

DISKSHADOW>
DISKSHADOW> end backup

DISKSHADOW> delete shadows all
Deleting shadow copy {9bbaacc0-0e3c-4f02-8a6d-f85628e1b4df} on volume \\?\Volume{b842e0b1-bae8-4300-9bb9-b2e02c06b9fa}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...
Deleting shadow copy {2e812c60-964a-4c97-b2a9-7e9475fd0a26} on volume \\?\Volume{f18617de-b581-4655-9304-ae7d6efb2c6e}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...
Deleting shadow copy {e90db7ce-f92f-4d35-be7c-8b6b6bebd2f3} on volume \\?\Volume{dc1dd100-14f1-4613-b93e-3314689009f7}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...
Deleting shadow copy {e4c52e0e-a7b4-40d0-b6fe-17c458479fe1} on volume \\?\Volume{9110a541-79d0-4811-8146-c9223419cf62}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...
Deleting shadow copy {e6b65e1b-443c-4104-9d3e-f7996d8df7bb} on volume \\?\Volume{083f9d7b-f5a5-4f4a-ab1b-9b1e003d43c7}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...
Deleting shadow copy {49e132fc-0893-491a-af48-ba51568619da} on volume \\?\Volume{cb7879d6-e2f5-4e41-b3df-4984e3275381}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...
Deleting shadow copy {69daeb61-c5e6-44e1-b960-4dd50de97340} on volume \\?\Volume{10373d4e-61a8-4b27-95a0-7493d60dd561}\ from provider {b5946137-7b9f-4925-af80-51abd60b20d5} [Attributes: 0x00420009]...

Number of shadow copies deleted: 7

DISKSHADOW> list shadows all
Querying all shadow copies on the computer ...
No shadow copies found in system.

DISKSHADOW> exit
***************** END LOG *****************

STEP 6 >>> SUCCESSFUL

I have only problem in STEP 2

Thanks