topic Re: Nimble Storage Deduplication? in Array Performance and Data Protection

Nimble Storage Deduplication?

hoangdv105 — Mon, 04 Aug 2014 14:06:33 GMT

Hi all,

I want to understand how deduplication of nimble storage work but i can't see any document about that. Maybe i miss to find that. Please someone can tell me about that !

Thanks !!

Re: How nimble storage deduplication ?

rugby0134 — Mon, 04 Aug 2014 14:32:16 GMT

Currently the Nimble OS only support LZ4 compression. Because the array has such a low cost per GB, we don't need to deploy De-duplication to save storage space. Deduplication for primary storage only helps save storage in a couple specific workloads. Deploying deduplication for VDI clones workloads for example don't save you much.

Re: How nimble storage deduplication ?

hoangdv105 — Mon, 04 Aug 2014 16:00:06 GMT

Thank you, but can you explain for me how LZ4 compression work ?, and how much percent capacity space reduced when compressed ?

Re: How nimble storage deduplication ?

rugby0134 — Mon, 04 Aug 2014 16:23:09 GMT

RealTime Data Compression: LZ4 explained here's a good explanation.

Re: How nimble storage deduplication ?

rlabarre65 — Mon, 04 Aug 2014 17:39:24 GMT

Hoang DV. Given the fact that we have Infosight, and thus an insane amount of detail about a customers data we can tell you that we average around 50% compression (as a total customer average). However, as you know all workloads are different and your individual mileage may vary. While databases tend to skew the graph higher, video files etc move it in the opposite direction (as they offer no compression). We can see file share data being highly compressable at times as well. Exchange data, OS Volumes etc are compressable as well, to what extent will again, vary based on your particular data sampling.

Hope that helps.

Re: Nimble Storage Deduplication?

tmcmullin51 — Mon, 04 Aug 2014 18:42:44 GMT

Hoang,

If you really want to know the gory details, here are more some good articles:

LZ4 (compression algorithm) - Wikipedia, the free encyclopedia

lz4 - Extremely Fast Compression algorithm - Google Project Hosting

Wow, LZ4 is fast! - Adrien Grand

My point in sharing this is that while there are other algorithms that can provide slightly better compression ratios, Nimble chose LZ4 because of its speed - which is critical when you're designing an enterprise disk array.

Is there a particular type of workload in which you're interested? I could probably look up some average compression ratios for you.

Re: Nimble Storage Deduplication?

hoangdv105 — Mon, 04 Aug 2014 22:54:12 GMT

Thanks about your support, this really helpful for me !

Re: Nimble Storage Deduplication?

jwagner137 — Thu, 03 Mar 2016 19:13:12 GMT

I know this has already been flagged as answered but aren't we talking two entirely different things here?

Compression and deduplication aren't the same thing at least not from my understanding of what each does and how it does it.

Re: Nimble Storage Deduplication?

katusavage119 — Wed, 09 Mar 2016 18:48:53 GMT

Hi John,

Thank you for bumping this thread. We actually have a great blog post discussing Nimble's entire data reduction approach following the launch of our all flash arrays. It goes into deduplication, inline compression, zero-pattern elimination, and zero-copy clones. There is also an in-depth technical blog post for our Nimble OS 3.0 blog that discusses dedupe from an engineering perspective.

I hope this provides a better picture of what you're asking about.

All Flash: Data Reduction Beyond Dedupe | Nimble Storage

Dedupe – An Engineer Looks Under the Hood

Cheers,

Kat Usavage - NimbleConnect Admin

Re: Nimble Storage Deduplication?

jwagner137 — Wed, 09 Mar 2016 18:59:36 GMT

Kat,

Thanks very much for this info. Nice to know that this is coming!

Re: Nimble Storage Deduplication?

alucorp11 — Tue, 19 Apr 2016 22:20:26 GMT

Hi Hoang,

Deduplication, specially inline dedup is excellent, not just for saving storage space, but also performance. Less data to write, plus more data fitting in your cache, all adds up to better performance.

For VDI, I see 30:1 data reduction, though it is with a competitor's array and not Nimble, since Nimble does not have deduplication.

Re: Nimble Storage Deduplication?

rugby0134 — Thu, 21 Apr 2016 15:11:34 GMT

Nimble does have DeDup in the 3.1 code release. Buy an All-Flash Nimble array and you're golden!