Grounded in the Cloud
cancel
Showing results for 
Search instead for 
Did you mean: 

Erasure Coding promises to reduce object storage costs in Openstack Swift

Stephen_Spector

Guest Post: Alistair Coles, HP Helion Engineering

 

There is a lot of excitement in the Openstack Swift community around the recent Kilo release, which saw the introduction of Erasure Coding as an alternative way to make stored objects  durable [1].  Whereas Swift has typically used triple-replication to ensure that an object is always available in the face of data loss or server failure, meaning that every object consumes three times its size on disk, Erasure Coding can achieve the same level of durability using approximately half the disk space. That results in your Swift cluster requiring less storage for the same capacity.

 

So how does it work? Put simply, Erasure Coding takes an object, splits it into a number of chunks, uses a clever algorithm to calculate some extra parity chunks and then spreads all the chunks across disks. Some of those chunks can be lost, but it is still possible to reassemble the original object from the remaining chunks. Furthermore, any chunks lost due to a disk failure can be recalculated and restored using the other chunks.

 

Taking a concrete example, a 1 GByte object might get transformed into fourteen 100 MByte chunks that collectively consume 1.4 GByte on disk - that compares very favorably to the 3 GByte consumed by triple-replication. Any four of those fourteen chunks could be lost or temporarily unavailable and the object can still be reassembled.

 

There is, inevitably, a price to pay for the reduced storage requirement – the Erasure Coding algorithm places an extra load on CPUs when objects are written and read. For data that is expected to be written once and read infrequently, this could well be a price worth paying. For other data, triple-replication may still be the best choice. So Swift allows both approaches to co-exist in the same cluster as different Storage Policies [2], allowing users to make the most appropriate choice for each of their object containers.

 

Swift developers from HP, Intel, RedHat, and SwiftStack have been working hard to include the ‘beta’ version of Erasure Coding in the Kilo release. Work is now continuing on testing at scale and performance characterization, bringing it closer to production deployment.

 

[1] http://docs.openstack.org/developer/swift/overview_erasure_code.html

[2] http://docs.openstack.org/developer/swift/overview_policies.html

Senior Manager, Cloud Online Marketing
0 Kudos
About the Author

Stephen_Spector

I manage the HPE Helion social media and website teams promoting the enterprise cloud solutions at HPE for hybrid, public, and private clouds. I was previously at Dell promoting their Cloud solutions and was the open source community manager for OpenStack and Xen.org at Rackspace and Citrix Systems. While at Citrix Systems, I founded the Citrix Developer Network, developed global alliance and licensing programs, and even once added audio to the DOS ICA client with assembler. Follow me at @SpectorID

Events
28-30 November
Madrid, Spain
Discover 2017 Madrid
Join us for Hewlett Packard Enterprise Discover 2017 Madrid, taking place 28-30 November at the Feria de Madrid Convention Center
Read more
HPE at Worldwide IT Conferences and Events -  2017
Learn about IT conferences and events  where Hewlett Packard Enterprise has a presence
Read more
View all