Simplivity backups and StoreOnce

[Sean-PA]

I'm having some issues with our Simplivity catalyst backups that are sending to our StoreOnce catalyst stores not completing in a timely manner. I've tried to google any kind of Best Practices documentation for configuring the stores and any limitations or considerations to be aware of when setting up backup polices and was wondering if anyone here could point me in the direction of some quality documentation to make these two products work efficiently in harmony.

[Randolfini]

HI Sean,

How many VMs are you protecting with the StoreOnce and what are the backup schedules?

Thanks

[Sean-PA]

Randolfini

We have a 6 node extra small cluster with 109 vms being backed up and a 6 node medium cluster with 135 vms being backed up. 

Both clusters use the same policy which is configured to send to two different StoreOnce appliances, one local and one remote. 

The remote backups kick off first at 6pm and then the local backups kick off two hours later at 8pm.

So total we have 244 backups going remote at 6pm and 244 backups going local at 8pm.

Both the remote and local StoreOnce are the 3640 models with 10g connectivity, The WAN link to the remote StoreOnce is a 500Mbps pipe.

[Randolfini]

Ok thanks for the context.

Where are you seeing that the backups aren't completing in a timely manner, your description below:

Simplivity catalyst backups that are sending to our StoreOnce catalyst stores not completing in a timely manner

I assume you right click on the SimpliVity cluster> All SimpliVity Actions > Search backups and are looking at the Sent Duration time and Sent Completion time?

-Thanks

[Sean-PA]

Not quite - I have a scripted report that runs every morning at 7am that queries the status of all backups from the night before and reports on which ones say Protected and which ones don't. When I get in in the morning I check the reports and if there are backups that arent protected I have a bash script that I run that monitors the status of the backups and it is typically between 1030 and 1130 that they all finally finish, but it has gone into the afternoon a couple times before. This really only started once we deployed the new 2nd cluster with the medium nodes and moved vms over from our old blade center. 

We originally had the local and remote backps swapped meaning the local ran first and the remote ran two hours later and when we were doing that, we saw the remote ones were the last to finish. After swapping them so the remote runs first and the local rus two hours later, now the local ones are the last to finish.

When I look at the StoreOnce reports page at the number of sessions, it seems to never go above 90 to 100. If I sit and actually watch it while the backups are running, the cpu never seems to go above 75% and RAM never seems to go above 30%. 

Also, it seems that the backups from cluster 1 all seem to complete by the time the morning report gets ran, its just backups on the newer cluster 2 that seem to lag.

[Randolfini]

Hi Sean-PA,

Sorry for the delay. Honestly my first thought is that you are saturating the StoreOnce either the NIC, concurrent connections, or whatever landing capability it has.

Have you opened a support ticket to see what they say?

I'm not sure what is available from StoreOnce CLI to see if it has any additional reporting like TOP or something to see what might be happening.

Separately I'm curious if you have the script you use on Github or might be willing to share via PM?

Thanks

[Sean-PA]

I have not yet opened a support ticket but that will be my next step once the other three tickets i have opened with them for other issues get resolved.

The script I run for the daily reports is a modified version of atkinsroy's script found here: 

# based on DailyWeeklyMonthlyReport script by atkinsroy
# https://github.com/atkinsroy/HPESimpliVity/blob/master/UtilityScripts/DailyWeeklyMonthlyReport.ps1

and required the simplivity powershell modual hosted on the hpe dev blog.

[Steven Clementi]

Sean:

I don't remember where I seen it, but there was a default limit on the concurrent sessions to the StoreOnce and each VM opens a new session..  That said, sending that many backups at once won't help make it go quicker even if you have bandwidth and resources to spare.

Have you tried breaking up your backups into groups and from the different clusters to allow some time for each set to finish?  This assumes you have multiple datastores configured with different backup policies, if not... maybe it is worth considering

What version of Omnistack are you on?  There are lots of fixes for things in 4.0.1U1 and 4.1.0 if you are on an earlier version with regards to StoreOnce backups.

Networking plays a big part here as well since the SimpliVity nodes need to send actual data across the wire to another device.  Without more details we can't rule out any networking issues, but it sounds like the clusters are likely set up with the same or similar network configuration?

[Sean-PA]

Steven

We have a 6 node extra small cluster with 109 vms being backed up and a 6 node medium cluster with 135 vms being backed up.  Both clusters are running 4.1.0 and both have identical network configs.

Both clusters use the same policy which is configured to send to two different StoreOnce appliances, one local and one remote. Both have a single datastore that the policy is applied to.

The remote backups kick off first at 6pm and then the local backups kick off two hours later at 8pm.

So total we have 244 backups going remote at 6pm and 244 backups going local at 8pm.

Both the remote and local StoreOnce are the 3640 models on 4.2.3 code with 10g connectivity, The WAN link to the remote StoreOnce is a 500Mbps pipe.

[Randolfini]

Sean-PA, like the previous poster said, I'd experiment with the timings being different.

Regardless what you have, 244 local backups starting at 8PM is a lot of backups. It does really feel like there is some saturation happening somewhere.

Do you notice any latency increases from the SimpliVity storage performance during the local backup run?