How nimble storage deduplication ?

[hoangdv105]

Hi all,

I want to understand how deduplication of nimble storage work  but i can't see any document about that. Maybe i miss to find that. Please someone can tell me about that !

Thanks !!

[rugby0134 · Accepted solution]

Currently the Nimble OS only support LZ4 compression. Because the array has such a low cost per GB, we don't need to deploy De-duplication to save storage space.  Deduplication for primary storage only helps save storage in a couple specific workloads.  Deploying deduplication for VDI clones workloads for example don't save you much.

[hoangdv105]

Thank you, but can you explain for me how LZ4 compression work ?, and how much percent capacity space reduced when compressed ?

[rugby0134]

RealTime Data Compression: LZ4 explained here's a good explanation.

[rlabarre65]

Hoang DV.  Given the fact that we have Infosight, and thus an insane amount of detail about a customers data we can tell you that we average around 50% compression (as a total customer average).  However, as you know all workloads are different and your individual mileage may vary.  While databases tend to skew the graph higher, video files etc move it in the opposite direction (as they offer no compression). We can see file share data being highly compressable at times as well.  Exchange data, OS Volumes etc are compressable as well, to what extent will again, vary based on your particular data sampling.

Hope that helps.

[tmcmullin51]

Hoang,

If you really want to know the gory details, here are more some good articles:

LZ4 (compression algorithm) - Wikipedia, the free encyclopedia

lz4 - Extremely Fast Compression algorithm - Google Project Hosting

Wow, LZ4 is fast! - Adrien Grand

My point in sharing this is that while there are other algorithms that can provide slightly better compression ratios, Nimble chose LZ4 because of its speed - which is critical when you're designing an enterprise disk array.

Is there a particular type of workload in which you're interested?  I could probably look up some average compression ratios for you.

[hoangdv105]

Thanks about your support, this really helpful for me !

[jwagner137]

I know this has already been flagged as answered but aren't we talking two entirely different things here?

Compression and deduplication aren't the same thing at least not from my understanding of what each does and how it does it.

[katusavage119]

Hi John,

Thank you for bumping this thread. We actually have a great blog post discussing Nimble's entire data reduction approach following the launch of our all flash arrays. It goes into deduplication, inline compression, zero-pattern elimination, and zero-copy clones. There is also an in-depth technical blog post for our Nimble OS 3.0 blog that discusses dedupe from an engineering perspective.

I hope this provides a better picture of what you're asking about.

All Flash: Data Reduction Beyond Dedupe | Nimble Storage

Dedupe – An Engineer Looks Under the Hood

Cheers,

Kat Usavage - NimbleConnect Admin

[jwagner137]

Kat,

Thanks very much for this info. Nice to know that this is coming! 

[alucorp11]

Hi Hoang,

Deduplication, specially inline dedup is excellent, not just for saving storage space, but also performance. Less data to write, plus more data fitting in your cache, all adds up to better performance.

For VDI, I see 30:1 data reduction, though it is with a competitor's array and not Nimble, since Nimble does not have deduplication.

[rugby0134]

Nimble does have DeDup in the 3.1 code release.    Buy an All-Flash Nimble array and you're golden!