Very slow synchronization between two omnistacks

[jneau]

Hello everyone,

I recently deployed a new node in a federation, the deployment was successful.
Storage synchronization has started, but I'm seeing a throughput of no more than 120Mb. We checked the entire network chain, and all the cards are at 10 Gb (with HPE Aruba Gbic) with MTU at 9000, as well as all the switches, vswitch and VMware protgroup and Omnistack and Esxi host.
I tested an iperf between the two omnistacks with a result of 9.8 Gbits/s.

I I've also run a zeus net federation test, and everything is green.

During a full synchronization, does the omnistack throttle the throughput so as not to slow down production?

Thanks for your feedback,

Julian,

[Joshua610]

Thank you for reaching out - please allow me to review your query. I will be able to provide you some insight by end of day Thursday. 

[NeetSat]

Hi Jneau,
Good Day. 

I see that you have checked the details between the OVCs and also the Zeus Net federation test shows green. 
If you do not mind, I would like to know what data you have on the existing nodes which is getting synced to the new Node? 
Also how many nodes do we have inside the Cluster along with the new node?
For some reasons, i feel it could be the deduped & compressed data, but i could be wrong too, so if you can help with the above information, that would really be great. 

Regards

[jneau]

Hi,

I have 2 node. I've still got the other node to replace, and we're going to migrate it directly to attach instead of using the Aruba switches. Let's see what happens...

Below the result of cli : dsv-backup-size-estimate --node OmniStackVC-xx-xx-xx-xx, we have deleted the backups since the end of the 1st migration.

For node OmniStackVC-xx-xx-xx-xx in cluster 'xx':
- Total Node Data: 5.88TB = Data: 1.68TB + Local backup: 4.20TB + Remote backup: 0B
- Compressed: 2.34TB, Uncompressed: 3.10TB
- Comp ratio: 0.75470
- Dedup ratio: 0.52774
- Total number of backups: 36
- Number of backups shown: 36 = Local 9 + Offnode 27, unknown 27
- Total backup bytes: 20.75TB = local 4.20TB + offnode 16.54TB
- Gross heuristic estimate: 1.58TB

Regards,

[Joshua610]

Thank you for reaching out with your query. I am providing some documentation to support those that may be unfamiliar in order to assist and inform. 

In regards to "During a full synchronization, does the omnistack throttle the throughput so as not to slow down production?" 

We do offer prioritization within our network as to provide performance optimization as well as to maximize the performance of the VMs living within the Federation. Items such as IWO/DRS(if/when enabled) also assist with this: 

https://support.hpe.com/hpesc/public/docDisplay?docId=sd00005743en_us&page=GUID-14E91A88-5052-4598-A08F-7B0A7735FF65.html&docLocale=en_US

I have listed some more links below to help. 

For those interested throughput can be located by following this document/link: 

View the backup throughput between clusters <for those that are interested>

https://support.hpe.com/hpesc/public/docDisplay?docId=sd00005743en_us&page=GUID-87876224-5E13-4C51-A06C-2022664522BB.html&docLocale=en_US

For those not familiar with the SimpliVity Environment there is a prioritization on where and how traffic should flow with in the federation/clusters. We have three main networks Management, Federation, and Storage each responsible for specific tasks as to optimize the network traffic and data flow. I am providing this document/link for those unfamiliar or interested. 

https://support.hpe.com/hpesc/public/docDisplay?docId=sd00005743en_us&page=GUID-85A8A62F-864A-46BE-B0B1-FA5BFF93BD9B.html&docLocale=en_US

For those interested in the further troubleshooting their network that may not have access to Zeus or familiar with it feel free to view this document/link.

SimpliVity - Network Troubleshooting Guide

https://support.hpe.com/hpesc/public/docDisplay?docId=sf000063383en_us&docLocale=en_US

Feel free to open up a case with our support team if you feel that the performance/throughput is not where it should be for a thorough investigation would be needed for any performance related issues.

Please let me know if any further questions remain, thank you. 

[jneau]

- IWO/DRS = i have 2 node, DRS is not necessary for this customer

- View the backup throughput between clusters = no more backup policy scheduled

- SimpliVity - Network Troubleshooting Guide = script zeus is already successful on all port tests and diag, I tested an iperf between the two omnistacks with a result of 9.8 Gbits/s (ports Sto/Fed).

I've been using Simplivity for over 7 years now. Deployments, replace in place node, regular updates on a set of 2,4,8 and even 16 nodes clients at one time. And I've never found this case... I know the solution...while waiting for HPE VME & Simplivity

I have a doubt about the hardware side, especially the SFP+, maybe nothing to see but it costs nothing to try

I'm going to replace the 2nd node soon, we're going to migrate it to direct attach mode, we'll see.

Regards,

[Joshua610]

Thank you. I will follow up on Monday, take care.