On-premises object-based storage products are gaining popularity, expanding their role beyond secondary storage like backup and archive to cover newer workloads such as artificial intelligence (AI) , machine learning (ML), and analytics.
Why the growth now? IT organizations need storage solutions that can scale almost infinitely, support a wide range of
In this blog, I’ll help you understand how to use object-based storage to consolidate workloads in your data center and at edge sites.
Why IT organizations are looking for newer storage solutions
Enterprise digital transformation (DX) has been disruptive for IT organizations. The extended use of digital technology to improve customer experiences as well as the business and operational processes has produced the exponential growth of data. This is especially true when it comes to unstructured data, the proliferation of new workloads to support (especially AI and analytics), and the need for ubiquitous deployment (on-prem, cloud and edge) for the traditional and new enterprise applications. Each of these three areas has profound implications for enterprise storage solutions.
The exponential growth of data to store, protect, manage, and use
The proliferation of unstructured data has dramatically changed how data is persisted and consumed in enterprises. Keep this in mind: Data must always be accessible and it never ages. As a result, the data management paradigm has shifted from storage cost optimization to scalability, protection, and optimizing data accessibility. As a result, the new data-center storage solutions must be capable of the following:
- Infinite scalability and software-defined-storage architecture (SDS)
- A single global namespace capable of managing billions of objects across geographies and deployment sites (public clouds, on-premises, and edge).
- Distributed data protection
- Higher levels of data durability and reliability
- Easy system operability
- Optimized data and metadata management
- Multi-protocol access to data
The proliferation of new workloads
IoT, consumer usage data, and big data analytics, along with AI/ML, have become critical catalysts to propel organizations through their digital transformation journey. At the same time, this has introduced a new set of challenges for storage solutions, such as supporting higher-throughput and lower-latency workloads, providing multi-protocol access, and strong metadata management.
Ubiquitous deployment – on-prem, cloud, and edge – of the cloud-native applications
The transition to cloud-native architecture and development is another consequence of digital transformation. To support cloud-native applications, the storage solutions need to be:
- Containerized
- Multi-tenant
- Lightweight
How object-based storage is evolving
The hottest trend in object storage is the use of all-flash SSDs to address the performance needs of modern applications. To better leverage SSD, object storage vendors have re-architected or optimized their products to reduce the latency associated with the object-store metadata and to improve the throughput.
- Containerized object storage – The second big trend is the native support of Kubernetes. Object vendors are strengthening their support for Kubernetes and other container-oriented technologies to address development teams' needs of creating stateful data to support analytics, AI, and cloud-native applications. Lightweight object stores designed for container deployment and Kubernetes-based automation (e.g., Scality Artesca) are well suited to next-generation workloads that may span from edge to on-premises and cloud data centers.
- File and object support – Many object vendors add support for Network File Systems (NFS) or Server Message Block (SMB) in their products. Vice versa, many file system products now include S3 API support to make available file and object data into a single storage pool so that the IT team can use both file and object protocols as needed. Today, many object storage vendors have multi-protocol access, but not all of them offer truly concurrent access to the same data from different protocol stacks.
Object-based storage products are improving their support for hybrid and multi-cloud data and metadata management and monitoring to reduce the complexity of overseeing petabyte-scale storage environments.
Lastly, object vendors also prioritize environmental, social, and governance considerations to help customers reduce environmental impact and related costs. For more on this topic, please check out this ESG analyst report: The Digital Era Is Fueling Adoption of All-flash Object Storage.
Why object storage is a valid solution for a storage consolidation strategy
The combination of new and traditional capabilities has transformed object storage into an attractive storage solution capable of:
- Infinite scalability and software-defined-storage architecture (SDS). The software-defined-storage paradigm is based on commodity hardware and allows expanding capacity and throughput by adding new servers without reconfiguration or data re-balancing.
- A single global namespace capable of managing billions of objects across geographies and deployment sites. Using a single namespace avoids data silos and reduces the need for data copy from one storage system to another.
- Data protection distributed across the whole object-storage grid. The RAIN (Redundant Array of Independent Nodes) model, used by object storage, is much more efficient and resilient than the hardware RAID (Redundant Array of Independent Disks) approach used by traditional storage solutions. Data is accessible after losing multiple individual drives, nodes, and even locations.
- Higher levels of data durability. The data stored in the object storage has impressive durability (time between potential data corruption events).
- Throughput and latency. Object storage can achieve higher throughput by simply adding nodes, but their limit remains the access time (or latency). So, Object solutions can be used for high-throughput demanding applications, but not for millisecond latency application
- Multi-protocol access. A key aspect for workloads consolidation is the capability to support multiple and concurrent access protocols to the same data. Today, many object storage vendors have had multi-protocol access, but not all offer truly concurrent access to the same data from different protocol stacks.
- Rich metadata enrichment, search, and classification: Metadata – the information about the data – is critical for the successful use of the stored data. Efficient rich metadata management has always been object storage's main strength.
- Containerized storage - Having a CSI driver does not mean being a cloud-native storage solution. A truly cloud-native object storage system is packed in the application stack so that Kubernetes can orchestrate the storage nodes performing provisioning, placement, scaling, and upgrade operations.
Why now is the time for new object storage solutions
The availability of new and re-architected high-performance, S3-compatible, multi-protocols, flash-based object stores make on-premises object stores more attractive. Now multiple workloads can be consolidated on a single storage platform, which is especially useful for big data analytics, AI, and native-cloud services. IDC and Gartner research confirms[1] that new object storage products are rewriting the primary and secondary storage play books.
Obviously, backup and archive will continue to be essential use cases for object vendors. But the new product capabilities and the long-standing advantages of cost-effectiveness, ease of scaling to petabytes (and potentially exabytes) of data, and deployment flexibility across on-premises, edge, and hybrid cloud environments make software-defined object storage a strong contender against the block and file-based alternatives for the consolidation of next-generation workloads.
Learn more
Get more information on HPE Solutions for Objects. And stay tuned to Around the Storage Block to learn more about HPE data store solutions for AI and advanced analytics
[1] Examples of the analyst reports discussing the new object storage solutions:
- IDC - Object Storage: Evolution from Repository for Backups and Archives to Target for High-Performance Workloads
- IDC - Object Storage Moves Beyond Traditional Backup and Archival Use Cases to Target High-Performance Workloads
- Gartner - Market Guide for Hybrid Cloud Storage
- Gartner – 2022 Strategic roadmap for Storage
Andrea Fabrizi
Hewlett Packard Enterprise
twitter.com/HPE_Storage
linkedin.com/showcase/hpestorage/
hpe.com/storage
AndreaFabrizi1
Andrea Fabrizi is the Strategic Portfolio Manager for Big Data and Analytics at HPE.
