HPE Alletra MP X10000: a modern data platform for Kubernetes environments
The new HPE Alletra Storage MP X10000 object storage has been designed with large unstructured data sets in mind and optimized to enable enterprises to store and process object data faster. The exceptional performance of the X10000 is ideal for data lakes and modern data protection, where high processing performance and short data access times are crucial.
With this new offering, HPE further simplifies storage management based on a unified modular architecture for block, file, and object storage, along with intuitive management on the HPE GreenLake cloud platform.
Moreover, HPE is collaborating with NVIDIA to provide direct memory access (DMA) functionality between GPU memory, system memory, and the X10000 via the S3 object protocol, which is critical for AI-based applications. This direct access path will further increase system throughput, reduce latency, and enhance GPU utilization. This will be a unique solution on the market, as access is currently provided only through file protocols.
Unlike traditional storage solutions, the HPE Alletra Storage MP X10000 is an S3-compatible, all-flash object storage platform designed for the most demanding workloads, including active data lakes, digital repositories, and scalable data protection workflows. Its disaggregated architecture ensures independent scaling of performance and capacity, eliminating bottlenecks and maintaining fast performance.
Now HPE has integrated Container Object Storage Interface into HPE Alletra Storage MP X10000, enabling seamless object storage provisioning and management within Kubernetes. This integration allows developers and operators to automate storage tasks, improve scalability, and manage storage resources more efficiently. By leveraging COSI with HPE’s storage solutions, organizations can achieve a more robust and flexible Kubernetes environment, ultimately driving greater efficiency and performance in their cloud-native applications.
Kubernetes workloads dynamically scale based on application demand, and COSI bridges the gap between Kubernetes-native operations and high-performance, automated object storage. Here’s how:
- Seamless object storage integration: COSI enables fully automated bucket provisioning and lifecycle management within Kubernetes, reducing storage complexity.
- Scalability without bottlenecks: The X10000’s disaggregated design scales non-disruptively, ensuring Kubernetes applications never hit storage limits.
- Optimized AI/ML and analytics pipelines: High-performance, low-latency object storage ensures real-time data processing without compromise.
HPE CSI Driver
COSI originated from the Container Storage Interface (CSI) which is a standardized interface developed to enable storage providers to create plugins that can be used by container orchestration systems like Kubernetes. CSI abstracts the details of storage systems, allowing for the seamless integration and management of various storage solutions without requiring changes to the core orchestration platform. This standardization facilitates improved portability, flexibility, and scalability in managing storage resources across different environments, enhancing the overall efficiency of containerized applications.
The HPE Container Storage Interface (CSI) Driver for Kubernetes allows to use a Container Storage Provider (CSP) to perform data management operations on storage resources. The architecture of the CSI driver allows block storage vendors to implement a CSP. The CSI driver architecture allows a complete separation of concerns between upstream Kubernetes core, Storage Special-Interest-Group (CSI owners), CSI driver author (HPE) and the backend CSP developer.
HPE CSI driver architecture
For more detailed information on how HPE CSI driver works please watch this video.
Container Object Storage Interface COSI
The Container Object Storage Interface (COSI) is considered better suited for object storage compared to the Container Storage Interface (CSI) because COSI is specifically designed to handle the unique characteristics and requirements of object storage. Here are several reasons why COSI is better for object storage:
- Specialized for Object Storage: Unlike CSI, which is primarily focused on block and file storage, COSI is tailored to manage object storage, addressing its specific needs such as bucket creation, access policies, and object lifecycle management.
- Standardized Bucket Management: COSI provides standardized APIs for creating, managing, and deleting object storage buckets. This simplifies the integration and management of object storage within Kubernetes environments, making it easier for developers to work with object storage.
- Automated Provisioning and Access Control: COSI enables automated provisioning of object storage buckets and integrates access control mechanisms, reducing the need for manual configuration and minimizing the risk of misconfigurations.
- Enhanced Multi-Cloud Portability: COSI abstracts the underlying storage layers, ensuring consistent access to object storage across hybrid and multi-cloud Kubernetes clusters. This enhances portability and flexibility when deploying applications in diverse environments.
- Seamless Integration with Kubernetes: COSI is designed to work natively within Kubernetes, allowing object storage resources to be managed just like other Kubernetes resources using standard tools like kubectl. This results in a more unified and simplified management experience.
- Improved Scalability: COSI supports dynamic scaling of object storage based on workload demands, enabling efficient handling of varying data sizes and application requirements without manual intervention.
By focusing on the specific needs of object storage, COSI provides a more efficient, scalable, and user-friendly solution for managing object storage in Kubernetes environments compared to the more general-purpose CSI.
HPE Container Object Storage Interface (COSI) Driver for HPE Alletra Storage MP X10000
The HPE Container Object Storage Interface (COSI) Driver for HPE Alletra Storage MP X10000 is a Kubernetes-native solution that automates object storage provisioning while enhancing multi-cloud portability and operational efficiency. Developers no longer need to manually configure object storage, as COSI enables on-demand provisioning, automated access control, and standardized bucket management.
Key benefits of COSI with the HPE Alletra Storage MP X10000 include automated bucket provisioning, where storage is dynamically allocated and scaled as Kubernetes workloads grow, eliminating manual intervention. Seamless integration allows managing S3-compatible buckets as easily as other Kubernetes resources using kubectl. Multi-cluster data sharing is facilitated by COSI abstracting storage layers to ensure consistent object storage access across hybrid and multi-cloud Kubernetes clusters. Security is simplified through integrated access control and credential management, streamlining permissions and reducing misconfigurations. This enables a cloud-native infrastructure where object storage scales dynamically based on workload demands without administrative bottlenecks.
With the HPE Alletra Storage MP X10000 and COSI driver, enterprises can provision dedicated S3 buckets for applications in real-time, ensuring efficient data management for analytics, AI/ML, and backup workflows, and scale object storage non-disruptively to handle everything from small metadata files to massive data lakes. Additionally, they can optimize cost and compliance by leveraging automated bucket lifecycle policies, tagging, and multi-cloud replication, while enabling self-service storage for developers, eliminating delays, and improving operational efficiency.
The HPE COSI Driver for Kubernetes allows to use the HPE Object Storage Provider (OSP) to perform bucket management operations on storage resources. The COSI architecture allows to integrate HPE Alletra Storage MP X10000 Object Storage with a COSI-compatible containerized application running on the Kubernetes cluster. The COSI driver follows the gRPC (Remote Procedure Call) specification provided by Kubernetes.
HPE COSI driver architecture
Features and Capabilities
Below is the official table for COSI features that HPE has officially tested and validated against the platform matrix.
Refer to the official table of feature gates in the Kubernetes docs to determine the availability of alpha features. HPE strongly recommends to install the COSI driver with a Helm chart.
Compatibility and Support
HPE has tested the following combinations and included them as part of the official support services for the first HPE COSI Driver for Kubernetes v1.0.0 release.
Release highlights:
- Support for Kubernetes v1.25 to v1.31.
- Implementation of bucket creation, configuration (bucket tagging), lifecycle and access management.
- A log collector script that can be used to collect logs from any node.
Known issues
Known issues for HPE COSI Driver v1.0.0 release.
- The HPE COSI driver can be deployed only in the default namespace due to a bug in creating events in the COSI API objects when deployed in non-default namespaces.
- Creating BucketClaim or BucketAccess objects in parallel can cause failures in the COSI driver. A bug has been filed to address this issue.
- A warning event is created in the Bucket or BucketAccess resources when an error occurs, and has a life-span of one hour. During this period, if the error is resolved the Status will show Bucket Ready: true or Access Granted: true in the Bucket or BucketAccess respectively, but the warning event will persist till an hour lapses. A bug has been raised to resolve this ambiguity.
- Recreation of BucketClaim or BucketAccess objects doesn't work intermittently, as gRPC request is not sent to the COSI driver. This pull request will address the issue.
The benefits of bringing HPE COSI driver to HPE Alletra Storage MP X10000, such as simplified storage operations, better resource utilization, and increased agility in deploying and scaling applications. By leveraging COSI with HPE’s storage solutions, organizations can achieve a more robust and flexible Kubernetes environment, ultimately driving greater efficiency and performance in their cloud-native applications.
For more details about HPE Alletra Storage MP X10000 I recommend to read this article.
PiotrDrag
HPE Storage for Unstructured Data and AI Category & Business Development Manager for Central Europe. Passionate about primary storage, data protecion, Cloud Computing, scale out storage systems and Internet of Things.