How bad does write latency get when the 80% capacity full threshold is crossed?

[db808]

Hi,

How bad does write latency get when the 80% capacity full threshold is crossed? I understand that the overhead to garbage collect reclaimable block increases and may become so resource-intensive that it can no longer be done in the "background" .... which results in the "find free space" function becoming synchronous and can end up pacing the rate which new incoming write data can be handled.

I know "it depends", and "your milage may vary" depending on the workload and the various combinations of CPU/memory/SSD and disk that are configured, but is this degradation in performance in the order of tens of percent ...or 10+ fold?

Customers have no guidance or understanding how steep the performance cliff is that we might fall over.

I guess another way to ask this question is ... if you are operating at 80% - 84% storage capacity and write latency increases to 50+ milliseconds, with peaks over 150 milliseconds ... is expected due the extra overhead of finding free space, or is something "broken"?

We were running with write throughput of 1060 MB/sec in March, and now can't go faster than 30-60 MB/sec sequential writes. ... but there are no "alerts" being reported from the Nimble storage.

Thank you for your help.

Dave B

[foodandbikes]

I was talking to support a few weeks ago and was advised writes only slowdown when it gets to 95% full.

80% or 85% was the slowness point on previous versions, like in 1.x and maybe 2.x.

[Nick_Dyer]

The recommended limit we tell customers is 90% full (hence why Infosight reports everything at 90%) - the true engineering limit where garbage collection becomes WAY more aggressive is 95%. YMMV, of course. It's definitely not 80%, and I don't believe it ever has been - it certainly wasn't when I joined around 1.4.4.0 timeframe :)