iSCSI Latency is Not an Issue for Nimble Storage

[vilpatel86]

I am little bit confused with some of the latency numbers & need some clarification. What I read from internet resources is that iSCSI has more latency compared to FC & FCOE because of the TCP/IP overhead.

Looking at reference architecture for VDI  (http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/dc-partner-vmware/guide_c07-719522.pdf)

the average read & write latency numbers are below 12ms.

If anybody can shed light on this, I will appreciate.

Sincerely

Viral Patel

[andyk78]

Heya Viral.

I can't seem to access that VDI document there. Is there another location where it can be accessed?

What kinds of latency numbers are you seeing in your environment? (in VMWare, and on the array)

Also, indeed there is a little overhead with iSCSI due to network overhead: TCP/IP is a transport protocol has handshakes, resends, etc. And that's on top of the iSCSI "Ready to Transfer" (R2T) operations that occur (RFC 3720), where the first burst is 64k, followed by the rest of the payload.

On a relatively isolated/empty network (like many client iSCSI networks), it's not much at all. Perhaps a total of about 1-3ms depending on the environment and network hardware? (Host ---> One Switch ---> Array topology will usually operate at sub-milisecond speeds depending on payload size and the like. It's not as fast as 8 GBit FC in terms of protocol overhead, but only by a few microseconds here and there.

Not sure what you're alluding to re 12 ms: Most clean networks (again, assuming Host --> One/Two Switches --> Array) won't have latencies that high.

If you're experiencing a performance issues, slow performance etc, please feel free to open a Nimble support case as well, we'll look into that for you. Please be specific about what you're seeing, what data you're looking at, etc.

Let us know if you need any further help-

Andy K

[rocky36]

We see very low latencies, writes under 1ms a lot of the time. Here are some pictures. This is with brocade VDX switches and Nimble 240G. Let me know if you want any other information from our environment, I'd be more than happy to share.

[vilpatel86]

Hi Andy & Rocky

Thanks for the response .

the Link I provided (http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/dc-partner-vmware/guide_c07-719522.pdf)

I suspected this because of the report that I read through demarrtek.com ( see link below) which is trying to compare the results ( read latency) of FC,FCOE & iSCSI.

http://www.demartek.com/Reports_Free/Demartek_Cisco_Multi-protocol_Latency_Multi-switch_Evaluation_2012-07.pdf

In this report, Average Latency is higher for the iSCSI compare to FCOE ( P7 of above report). In my mind, I was trying to make the case for the iSCSI for the most of the Production workload.

This latency is from the SQLIO perspective . Information that Rocky Provided convince me that with 10G & Jumbo Frame iSCSI can cover the Most of the Workload.

Sincerely

Viral

[Nick_Dyer · Accepted solution]

Hi Viral,

It's true in "traditional" iSCSI storage implementations latency can be quite high. But this is due to how data is read and written randomly on the underlying storage itself.

Nimble Storage relies on CASL, which sequentialises all incoming writes. This enables us to deliver FC-like latency for reads and writes without the need for FC or FCoE.

If you have an hour spare I highly recommend watching the CASL Deep Dive video on Youtube by Devin Hamilton. By far our most popular and informative video uncovering the secret sauce.

[andyk78]

Heya Vital: Just a shout out to what Nick said.

Also, please note that the testing that Demartek did was on Netapp arrays for iSCSI and FC (I used to work at Netapp, so am very familiar with the hardware and performance parameters that Demartek is testing with). Also, interestingly enough, the array that they were using (3240 with 24 10k RPM disk drives) does not appear to be using Netapp's own answers to hybrid flash architecture (called "FlashPool" at Netapp, mixed aggregates of 10k or 15k RPM disk raidgroups with SSD raidgroups), nor are they using expandable flash memory ("FlashCache"), which would put it more in line with a Nimble array in terms of raw storage architecture/hardware (more of a hybrid array, like what we do out of the box)... but of course with a street price at least the very least four times higher than our costliest Nimble array, which because of our optimized OS and CASL, we would totally outperform anyway. :-)

I would love to see Demartek run the numbers again with a standard Nimble array at 1/4 the street price as their setup for that document. Again, we would blow those numbers away to the point that their test comparing iSCSI to FC would be meaningless. That's the benefit of iSCSI with an array that was designed, from its infancy, to not just be "some disks with SSDs serving as Flash", but a genuinely hybrid array that is aware of and utilizes SSDs and normal HDs together to the best of their performance: We can get the kinds of op and latency numbers that would make *anyone* rethink going with FC (unless they already had all the fabric hardware there already).

If you have a Sales Engineer near you or who works with you, please contact them, they can probably bring by a proof of concept that demonstrates exactly what you're looking for.

Thanks!
-Andy

[toffitomek105]

Rocky,

Did you ever run any tests using 1 Gbit links?

Do you have any idea how much results for 1Gbit links may be different from yours?

Cheers

[warkaj1]

Rocky,

We have Brocade VDX 10 Gig and a Nimble 260G and this is what I'm seeing.... it's not good. Might be a configuration issue from the VM hosts to the Nimble though.

[marktheblue45]

Blimey that's like my graphical output. Need to talk to Nimble to what I'm doing wrong perhaps?

[marktheblue45]

Our latency was lower for reads less than 0.5 ms but just under 1ms when copying 2TB OF FILES