HPE Storeonce 5200 Replication Very Slow

[andy91717]

We have a number of HPE Storeonce (D2D) devices onsite locations, and at our offsite location. These include 4700, 5100, and 5200's. We use both NAS (NFS/CIFS) shares, and also Catalyst Stores, with some D2D doing both, some doing just Catalyst.

We have recently (this year) installed a set of 5200's, two onsite, and one offsite, to ultimately replace the 4700's. We use Micro Focus Data Protector to backup onsite, and replicate offsite, to a remote offsite DP server. So two local D2D replicate to one remote, which is twice the size in terms of TiB available. This same two in to one we use for all the 4700, 5100, and 5200's. 

So the problem with the 5200's is how incredibly slow the transfer is, backups are fine, it's just replication between sites which is slow. Data Protector will start a replication session, make all the connections, create "Incomplete" catalyst items on the remote D2D, matching the number of objects in the DP job, i.e. 5 items to replicate, 5 items on the remote D2D in an Incomplete state, using up only MiB of data, so tiny (I assume this is some sort of MetaData connection). This is where DP will output "STARTING media Agent", and "COMPLETED Media Agent" every min or so, rising to every 5 mins, or 10 mins, polling the D2D for progress of the replication, and it does this for hours on end. Eventually, after many hours, it will return and replicate data ok. To us this is showing all the communication, firewalls, paths are in place to replicate data just fine, but for some reason the D2D is not doing it straight away, and is hanging waiting for something, but we don't know what that something is. 

Any example for a large replication we do weekly:

source A to remote C = 28TiB = 2 days!

source B to remote C = 28TiB = 4.5 days!

The two source D2D's are code level 4.2.3, the target is 4.3.0, we assume this would not be an issue. The D2D's are in use pretty much 24x7, having backups written locally, and replications running, but we can stop DP processes if we need to to reboot the D2D's, or upgrade code. 

By the way the 4700's, and 5100's, they don't have these replication performance issues, they are running on the same sites, using the same Data Protector servers, and networks, it is only the 5200's which are dead slow in replication. 

Thanks in advance for any help or advice,

Andy

[andy91717]

Did some more testing yesterday afternoon, left some small replications running in DP, two jobs, both from one source 5200, but going to two different 5200's:

• First one 3 hrs 48 mins to replicate ~50GiB
• Second one 2 hrs 18 mins to replicate ~40GiB

There was an overlap between the two replications for a couple of hours. Looking at the report graphs on the target 5200 GUI it show no replication activity for a good couple of hours before it kicks in. 

Regular scheduled job which ran from the same source and destination of as the first one above, 808GiB, ran for 6 hrs 43 mins. 

Compare these to a 5100-->5100 Catalyst replication, 50GiB, 36 mins!

[sbhat09]

Hello @andy91717,

What I understand is your Gen3 systems (4700, 5100) were fine. But Gen4 systems are performing slower while replicating while the replication software and networking are same.

Do you have any previous and current replication link performance data to compare?

Are there any changes in deduplication before and now?

Maybe the queued-up housekeeping jobs are crippling the replication. In this regard, the best practice is to allow housekeeping time to run by avoiding continuous scheduled backup and copy jobs.  

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[andy91717]

@sbhat09 

Hi Srinivas,

Thanks for replying.

The reality of our environment on these new 5200's is we only have selective backups on them for now as we don't want to overload the already slow replication performance. The vast majority of the day there is no backup activity at source, with backup at target only once a month, so general I/O is very low. Only at weekends we perform a large full backup of a database is a large amount of data written at source (50TB's), and then when complete, scheduled replication is run from Data Protector the day after the backup completes, which runs for days. 

I will be raising a ticket with our local networks team also to ask them to check the physical 10Gb port ethernet connections to make sure they are all setup properly and not being throttled in any way stopping the 5200's from replicating at full capacity compared to the 5100/4700's which run just fine.

Can you or someone else confirm that on the 5200's housekeeping cannot be stopped now like older generations of software, and we simply have graphs to work out when houskeeping is running?

Thanks,

Andy

[andy91717]

To add some more detail for this replication issue., which I should have said is a catalyst to catayst replication, we have no other type of backups on the 5200 D2D's, only catalyst.  We replicate a two-to-one scenario, so two source D2D's replicate to one target D2D which is the same model, twice the size of the source D2D's. Unfortunately I cannot copy/paste the performance graphs from the GUI, but to try and explain:

• Replication starts at the same time at both sources
• Throughput gets quickly up to 1 GiB/s, up to 1.4 GiB/s after a couple of hours
• Then it reduces throughput slowly over the next 20-24 hours down to MiB/.s speed, and then randomly from 0-100 MiB/s over days until it finishes up to 5 days later for the slower source D2D.
• The amount of replication sessions on the graph also goes ip to 5/6 quickly, but down to 1-2 after 20-24 hours, and then 0-1 over days. 
• The replication can have gaps of 5-6 hours where it does not even register on the D2D graphs of processing data, but DP still is showing the replication job running loading media from slots, starting replication, unloading media. 

We are really struggling to find the solution, the same Data Protector servers are used, the D2D's have the same backup/replication usage profile with only one large replication per week, no overlap with housekeeping which finishes before replication starts, same target D2D is used, but one source is slower than the other. It's a strange one this, we may have to reach out to our HPE account contacts asking for support. 

[sbhat09]

Hello @andy91717

Yes, the primary suspect is replication link performance. Please try to get the comparative network performance data before the issue and during the issue. Or if any network issues causing slowness in replication. In the next stage you can look for any issues in the source or target storeonce systems that are causing the slowness.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.