HPE Alletra Storage MP X10000 - a high-performance object storage for data analytics

[PiotrDrag]

In early 2023, HPE introduced a modular data management solution under the HPE Alletra Storage MP brand. HPE Alletra Storage MP X10000 joined HPE portfolio half a year ago. This object storage solution, like its block and file counterparts, is supported by the standardized and modular hardware platform, HPE Alletra Storage MP, built from the ground up.

HPE Alletra MP X10000: a modern approach to data analytics

The new HPE Alletra Storage MP X10000 object storage has been designed with large unstructured data sets in mind and optimized to enable enterprises to store and process object data faster. The exceptional performance of the X10000 is ideal for data lakes and modern data protection, where high processing performance and short data access times are crucial.

One of the key advantages of HPE Alletra Storage MP X10000 is its flexibility and scalability. By employing an innovative approach to distributed file&object arrays based on a disaggregated or modular architecture, users can easily adjust storage capacity according to changing business needs. This flexibility helps avoid excessive infrastructure investments while ensuring the ability to quickly respond to changes in data storage requirements. Furthermore, this approach provides a level of performance unprecedented in previous systems. This is crucial for all companies and institutions that handle large volumes of data, require fast access to it, and need dynamic allocation of computing power.

HPE Alletra Storage MP – hardware foundation of HPE Alletra Storage MP X10000

HPE Alletra Storage MP is built on a disaggregated Software Defined Storage architecture. The term "disaggregated" in this context means the ability to independently scale compute and storage capacities, leading to more efficient resource utilization and eliminating the need for significant redundancy. You simply buy what you need now and expand as necessary in the future. This hardware platform can therefore be used for various applications. In this article, we focus on its object storage incarnation, the HPE Alletra Storage MP X10000.

The minimum configuration consists of two compute nodes equipped with 3 controllers, one storage node, and two switches. In terms of capacity, we are talking about a minimum of 68TB of usable capacity in the starting configuration. A detailed description of HPE Alletra Storage MP X10000 can be found here.

Software

The brain of the HPE Alletra Storage MP X10000 system is software built from the ground up, based on a key-value store model. In this model, the key-value store implements the storage of object fragments (called chunks) and their metadata independently of the data transfer protocol, forming the core data layer. It is optimized for flash memory access, reducing write amplification thanks to a log-structured and extent-based approach.

Within the key-value store, there are namespace layers specific to individual protocols, such as S3/Object. These protocol layers are optimized for the semantics of each specific protocol, treating each one as a first-class citizen. This allows the HPE Alletra Storage MP X10000 system to leverage the advantages of each protocol without inheriting the drawbacks of another or stacking protocols on top of each other (such as running an object protocol on top of a file protocol or vice versa).

From a software development perspective, the entire system is built using containers. This is beneficial in terms of flexibility and scalability and opens many possibilities for utilizing the system's computational power in the future. With high-performance compute nodes, it will be possible to deploy data analysis applications directly on them (e.g., Apache Iceberg, Kafka, etc.).

HPE Alletra Storage MP X10000 is designed to provide balanced read and write performance, both for high throughput and small transactional operations. This means that large clusters are not required for workloads with a high number of write operations. This results in optimized performance regardless of the workload, allowing performance goals to be met without waste. One of the design goals of the HPE Alletra Storage MP X10000 array was to ensure high initial performance even with a relatively small starting environment (i.e., 3 compute modules and one disk shelf).

Data Analytics use cases

Workloads associated with unstructured data are extremely diverse. Even within a specific category of applications, such as artificial intelligence, the nature of workloads can vary significantly. While many object storage systems prioritize throughput-oriented performance, HPE Alletra Storage MP X10000 provides high IOPS performance for small objects, as well as high throughput for larger objects and low latency for GET and PUT operations (< 2 ms).

In object storage, S3 buckets are a useful structure. However, in the case of certain object solutions, it is often necessary to use multiple S3 buckets to achieve maximum system performance. Typical unstructured workloads, such as analytics and data protection, often require a single bucket or a small number of buckets per application-specific unit, such as a single data store or backup chain.

For HPE Alletra Storage MP X10000, this is not a problem. Even a single bucket is sufficient to achieve maximum system performance. This eliminates the complexity associated with additional configuration to enhance system performance. This is particularly noticeable in write (PUT) operations. Compared to selected competing solutions, we can be up to 60 times faster in PUT operations for small objects using a single bucket. The ability of HPE Alletra Storage MP X10000 to linearly scale a single bucket means that individual applications benefit from the same scalability as many applications or users.

Given the above, HPE Alletra Storage MP X10000 is ideally suited to handle a variety of applications that require high performance and low data access times—ranging from traditional data analytics applications to modern cloud and container environments

Unveiling the Potential of HPE Alletra Storage MP X10000 for Modern AI and Analytics Workloads

In today's fast-paced digital era, businesses are inundated with massive datasets generated across diverse sources. Extracting insights from this data efficiently is critical for competitive advantage. HPE Alletra Storage MP X10000 is a purpose-built storage solution designed to tackle the demands of modern and constantly evolving AI and analytics workloads.

HPE Alletra Storage MP X10000 provides faster access to object data on a massive scale. With its high-performance all-flash object storage array utilizing a disaggregated, shared everything storage architecture, users of data analytics applications can leverage a single hardware platform to store data that can be shared across various applications (e.g., Apache Kafka, Spark, Iceberg, etc.). Moreover, the high-performance compute nodes and key-value store architecture enable the automatic addition of metadata to objects as required by specific applications or users.

A big picture of an exemplary modern analytical application landscape that X10000 can get integrated with.

Let’s explore how this innovative technology transforms data management, analytics, and storage for enterprises. 

Unparalleled Performance for AI and Analytics

• Acceleration for Retrieval-Augmented Generation (RAG) Pipelines
  By integrating seamlessly into RAG pipelines via Milvus vector database, the Alletra X10000 accelerates AI applications such as chatbots, summarization, and code pair generation. Its high-performance vector database capabilities drive faster query responses and semantic searches, essential for modern AI-driven enterprises.
• Lightning-Fast Active Data Lakes
  Businesses managing active data lakes for real-time analytics benefit from the X10000’s rapid data ingestion and retrieval speeds. For example, its performance in benchmarks demonstrated 9x faster data load times compared to AWS S3, showcasing its superiority for large-scale analytics.
• Support for Large Language Models (LLMs)
  The X10000 enables LLMs by providing high-speed, S3-compatible object storage. Its efficient embedding creation and synchronization capabilities reduce latency, ensuring seamless AI pipeline operations.

Key Data Analytics Use Cases

• OpenText Analytics DB (Vertica)
  Customer with 100T+ of active data lake using OpenText Analytics DB (previously known as Vertica), can leverage the HPE Alletra X10000’s all-flash performance in a modern implementation of OpenText Analytics DB (EON mode) that disaggregates computational resources from the storage layer (depot storage for local caching & communal object storage as a shared data repository).
• Splunk SmartStore Integration
  For enterprises like Splunk requiring cost-effective long-term storage for log monitoring and IT operations, the X10000 optimizes TCO by tiering cold data efficiently while maintaining rapid retrieval times.
• Elastic Stack Deployment
  The Elastic Stack benefits from the X10000’s ability to store infrequently accessed data in its frozen tier, reducing local storage requirements and enhancing query response times.

Compatibility with Modern Analytical Stacks

The Alletra Storage MP X10000 integrates seamlessly with a variety of tools, including:

• Starburst Enterprise: Enhancing distributed SQL querying and Data Lakehouse capabilities for analytics.
• OpenText Analytics DB (Vertica): Supporting Eon-mode disaggregation for large-scale data analytics.
• Apache Druid: Accelerating real-time OLAP queries and enabling cost-effective storage tiering.

Built for the AI Era

The X10000 is uniquely positioned to meet the demands of AI workloads by:

• Supporting NVIDIA accelerated computing for high-throughput AI processing via S3 over Remote Direct Memory Access (RDMA)
• Offering inline metadata services for faster insights.
• Ensuring seamless scalability with disaggregated storage architecture.

Summary

HPE continues to innovate with the Alletra Storage MP X10000, evolving its capabilities to support emerging technologies and growing data demands. From enhancing AI pipelines to integrating ISV solutions, the X10000 is set to remain a pivotal asset for enterprises navigating the AI and analytics landscape. The HPE Alletra Storage MP X10000 is more than a storage solution—it's a transformative platform that empowers organizations to unlock the true potential of their data. With unmatched performance, scalability, and AI-driven optimizations, it sets a new standard for modern data workloads.

For those interested in HPE Alletra Storage MP X10000, I invite you to watch the introductory video on this technology.