Remote backup stuck rebuilding - deltas are protected, what happens at failure

[steez]

Hello everyone,

I need a bit of an in-depth advice on Simplivity remote backup behavior. This one will be hard to grasp so brace yourselves

We have 2 SVT clusters. We are migrating VMs from the old to the new as we plan to decomission the old cluster. For all VMs we used VMware Live migration with some hiccups but 99% of VMs are in the new cluster.

We have an issue with the very last VM in our environment that is large in terms of storage and memory usage. As its data is changing not stop we suspect that the network wont be able to keep up and live-migration will fail (had this with some other VMs but found some windows where they were less active).

In oder to get around this we plan to use Simplvity Remote backp (do initial backup, then turn off VM, do delta backup, restore at the new cluster). With this we have an issue - after initial remote backup, the backup state changes from saving to rebuilding and is stuck there (for 3 weeks now, sent size 3TB). As per HPE documentation, rebulding means that backup itself is not HA protected at this moment and backup data is being duplicated with another host in cluster to make it HA. Interestingly the subsequent backups sometimes go to degraded or protected state.

After working with HPE support (event escalating from L3 to LAB), they said that this is a software defect with a suggestion to still restore VM from subsequent protected backup, but I have my doubts with this so I have a question about how remote backups work and behave in case of a failure.

As subsequent backups are much smaller in size (about 10x in our case) I assume that they create a chain with initial backup (which I remind is in degraded state). What happens I restore VM from protected subsequent backup and the host that has non-HA initial backup fails? Will the VM also fail?

Asked HPE support about this for which they answered - should have identical data as the original VM (assuming no data change has been made on the original VM since the backup has taken place which the question was not about.

This is a hard concept to grasp, but really hope for Simplivity Guru/design architect to stumble accross this post and shine some light to this. In any case we will try to move VM to the new cluster using Veeam Replication (still far from ideal as VMware snapshots will be used with memory-storage active VM)

[Azr_geek · Accepted solution]

Hello @steez,

A remote backup is stored on a node in a different cluster. SimpliVity backups are deduplication-aware and pointer-based: subsequent backups reference unchanged data blocks from previous ones (like linked snapshots or deltas). A "full" backup is actually logical, meaning it's metadata + changed blocks, not a full physical copy.

Protected: Backup is fully written and duplicated across two nodes — safe from single node failure.

• Degraded: Backup exists but is not duplicated — if the hosting node fails, backup is lost.

• Rebuilding: SimpliVity is attempting to create the redundant copy — ideally transient.

• Saving: Backup is still being written.

A backup that stays in "rebuilding" state for weeks is abnormal and aligns with what HPE support said — likely a defect or failure in internal deduplication or data replication logic.

Subsequent (incremental) backups do not carry full block-level data. They reference unchanged blocks from earlier backups. This means:

• Backup 2 (protected) likely points to blocks in Backup 1 (degraded).

• If Backup 1’s node goes down, and it's not HA protected, the base data is lost.

• Even though Backup 2 is marked “protected,” its integrity depends on the availability of blocks from
  
  Regards,
  Azr_geek

[steez]

Thank you very much @Azr_geek for your input, this aligns to what we thought.

For some reason couldn't get this out of HPE. For now they recommend to still restore VM using healthy delta backup, then delete initial backup which is fine until the host with the initial backup fails making the VM innacessible.

As mentioned we will move VM using other tools so that Simplivity create a new data tree for that VM and it would be 100% HA protected.

Thanks again

[Azr_geek]

No problem!! 

I am a retired active member! You can post any question here and I will be glad to assist you with the best of my knowledge. 

I believe in helping and creating a good community.