Grounded in the Cloud
Showing results for 
Search instead for 
Did you mean: 

Erasure Coding promises to reduce object storage costs in Openstack Swift


Guest Post: Alistair Coles, HP Helion Engineering


There is a lot of excitement in the Openstack Swift community around the recent Kilo release, which saw the introduction of Erasure Coding as an alternative way to make stored objects  durable [1].  Whereas Swift has typically used triple-replication to ensure that an object is always available in the face of data loss or server failure, meaning that every object consumes three times its size on disk, Erasure Coding can achieve the same level of durability using approximately half the disk space. That results in your Swift cluster requiring less storage for the same capacity.


So how does it work? Put simply, Erasure Coding takes an object, splits it into a number of chunks, uses a clever algorithm to calculate some extra parity chunks and then spreads all the chunks across disks. Some of those chunks can be lost, but it is still possible to reassemble the original object from the remaining chunks. Furthermore, any chunks lost due to a disk failure can be recalculated and restored using the other chunks.


Taking a concrete example, a 1 GByte object might get transformed into fourteen 100 MByte chunks that collectively consume 1.4 GByte on disk - that compares very favorably to the 3 GByte consumed by triple-replication. Any four of those fourteen chunks could be lost or temporarily unavailable and the object can still be reassembled.


There is, inevitably, a price to pay for the reduced storage requirement – the Erasure Coding algorithm places an extra load on CPUs when objects are written and read. For data that is expected to be written once and read infrequently, this could well be a price worth paying. For other data, triple-replication may still be the best choice. So Swift allows both approaches to co-exist in the same cluster as different Storage Policies [2], allowing users to make the most appropriate choice for each of their object containers.


Swift developers from HP, Intel, RedHat, and SwiftStack have been working hard to include the ‘beta’ version of Erasure Coding in the Kilo release. Work is now continuing on testing at scale and performance characterization, bringing it closer to production deployment.




Senior Manager, Cloud Online Marketing
0 Kudos
About the Author


I manage the HPE Helion social media and website teams promoting the enterprise cloud solutions at HPE for hybrid, public, and private clouds. I was previously at Dell promoting their Cloud solutions and was the open source community manager for OpenStack and at Rackspace and Citrix Systems. While at Citrix Systems, I founded the Citrix Developer Network, developed global alliance and licensing programs, and even once added audio to the DOS ICA client with assembler. Follow me at @SpectorID

Jan 30-31, 2018
Expert Days - 2018
Visit this forum and get the schedules for online HPE Expert Days where you can talk to HPE product experts, R&D and support team members and get answ...
Read more
See posts for dates
HPE Webinars - 2018
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all