Brock Benard
Thank you for the reply. 300Mbps is a low amount of bandwidth for any Nimble Storage array to fill using 8 streams (volumes) replicating at once.
There might be some confusion of Mbps and MiBps. 1MiBps = ~8.4Mbps. So using the math you have provided, 400Mbps = ~47MiBps. Academically, MebiByte is 1024 KebiBytes, however everyone in computer industry is used to thinking that MegaByte is 1024x1024, which it is not academically. Yet, all Operating Systems are now reporting storage in "Mega" while using base 2 (1024). (Simple confirmation is to look at the "Bytes" value of the "Page" file of your Windows PC - it will be base 2^x). This should explain why you see peak bandwidth on the array as number 400 while path device is 8 times less at the same time.
In regards to the replication streams; each stream is a volume snapshot which is being replicated. In version Nimble OS 3.x, there are 8 streams, as you point out. The reason why there are multiple streams is to allow system to read enough data from the disks to saturate the network link during the variety of customized scheduling. Each volume's snapshot has it's own challenge of reading data from disks, since it needs to find the blocks to replicate. However, this challenge comes into play with a lot higher bandwidth, such as 2Gbps and usually on Hybrid arrays, where HDD spindles are involved. Regardless of how randomly data laid out on RAID on the AFA's, 8 streams is definitely enough to saturate the link of 300Mbps.
All 8 streams are replicating over a TCP session. TCP session is dependent on the reliability of the network path. I believe the other "N" might be using multiple TCP sessions (regardless of streams). In a non-optimal network path, that would make the that replication to utilize more of the bandwidth from the advertised network link speed. Non-optimal network path does not just mean there are network errors or physical device issues. It can mean that there is other traffic (congestion) or some kind of filtering is involved (Proxy, Firewall, IPS/IDS, Optimizer). There have been quite few cases where a device, meant to improve some matter of network transmission (Proxy, Firewall, IPS/IDS, Optimizer) have interfered with the TCP transmission to the point where it cannot saturate the link. I cannot say this is the case for your situation, but what i can say is that Nimble Storage array replicating compressed data blocks, thus, there is no reason to add a "WAN optimizer" on a path.
What i would suggest in your situation, if possible, is to bypass the Riverbed device completely (not just set a "bypass" rule, but not route through it at all) as a test. And after you pause/resume replication (this re-initiates TCP session) you might be able to see the asked 300Mbps while there is no congestion (like from the "N" device). If not, then network packet captures are in order, which would help determine if there is something else on the path (look for retransmits, out-of-order packets, TCP resets, etc).
In the case of original poster, 1Gbps limit seems to be kind of interesting. I hope Nimble Storage Support will provide some guidance of why it appears limited to this specific value.
