The rapid adoption of AI is breaking legacy storage architectures both architecturally and economically. Learn how organizations of all sizes are adapting and setting new records with HPC storage from HPE.
Gartner once has stated that “There will be no way to put the storage beast on a diet.” That was even before the convergence of artificial intelligence (AI) and classic modeling & simulation (mod/sim) demanded even higher storage performance with even higher capacities.
HPE saw this coming and took action. The acquisition of Cray not only added intellectual property on the software, compute and interconnect layer, but also has resulted in the creation of an HPC storage portfolio that enables users of AI and mod/sim workloads in organizations of all sizes to put the storage beast on an effective diet.
This blog shares 10 specific examples in 10 different categories where new benchmarks are being set for parallel HPC storage that is feeding data to supercomputers or clusters of rack servers running AI and mod/sim workloads.
1 & 2. The largest and fastest parallel file system
- Flash-based performance tier of 5,400 nonvolatile memory express (NVMe) drives providing 11.5 PB of capacity at peak read-write speeds of 10 TB/s
- Hard-disk-based capacity tier of 47,700 perpendicular magnetic recording drives providing 679 PB of capacity at peak read speeds of 5.5 TB/s and peak write speeds of 4.6 TB/s
- Flash-based metadata tier of 480 NVMe devices providing an additional capacity of 10 PB
This represents the new high watermark for external high performance file systems—both for the largest storage capacity and well has for the fastest performance.
3. The largest all-flash parallel file system
4. The largest twin parallel file systems
5. The fastest restore speed of parallel file system data
The solution at zensact requires the ability protect and restore data at very high (record) speeds in order to hit the business-critical simulation window should a restore of the data be necessary. HPE Data Management Framework 7 (DMF) running on HPE ProLiant DL rack servers could meet the requirement of restoring petabytes of data with about 200 GB/s.
6. The most secret parallel file system
HPE recently announced that it has been awarded a $2B contract with the National Security Agency (NSA) to deliver HPE’s HPC technology as a service through the HPE GreenLake edge-to-cloud platform. The new collaboration will enable the NSA to harness rapidly growing AI and data needs more efficiently to create insights and other forecasting and analysis with optimal performance. By using HPE’s HPC solutions through the HPE GreenLake platform, which provides fully managed, secure cloud services on-premises, the NSA will benefit from an agile, flexible, and secure platform to meet their growing data management requirements.
Due to the nature of the client no more details can be shared regarding parallel storage but it might be very large and very fast. The public most likely will never know…
7. The most "circular" parallel file system
The new supercomputer, delivered as a service through the HPE GreenLake edge-to-cloud platform, include the Cray ClusterStor E1000 storage systems and the HPE Data Management Framework 7 to support complex, image-intensive workloads in modeling and simulation.
It’s housed in ENI’s famous Green Data Center in Ferrera Erbognone, a province in Pavia, Italy and improves energy usage and reduce electronic waste by using HPE Asset Upcycling Services. This is part of the circular economy initiative from HPE Financial Services, which leverages asset longevity to reuse products, by recycling equipment like Cray ClusterStor L300 storage systems from its existing HPC4 system and replacing it with newer solutions.
8. The lowest carbon footprint parallel file system
The LUMI storage system consists of a total of 117 PB usable capacity:
- 7 PB all flash Cray ClusterStor E1000 storage systems
- 80 PB HDD-based Cray ClusterStor E1000 storage systems tiered with HPE DMF 7 to
- 30 PB of CEPH-based object storage running on HPE Apollo 4200 Gen10 systems
The LUMI supercomputer will use 100% hydroelectric energy, and the heat it generates will be captured and used to heat homes and commercial premises in the area, making LUMI one of the most environmentally efficient supercomputers in the world.
9. The longest-serving large scale parallel file system
The Blue Waters supercomputer and its Cray ClusterStor file system recently celebrated its 8th birthday!
But what about the AI users that do not want or cannot invest in large scale clusters or supercomputers?
For them we have the recently announced HPE Parallel File System Storage that delivers an IBM Spectrum Scale (FKA GPFS)-based parallel file system starting already with 12 (!) storage drives (HDD or NVMe SSD) in four HPE ProLiant DL325 Gen10 Plus-based storage servers.
10. The smallest parallel file system
HPE Parallel File System Storage in its entry configuration with just 12 NVMe SSD delivers about 35 GB/s throughput (read) while the high-end Dell EMC Isilon F800 model “just” delivers 15 GB/s from 60 SSDs (see datasheet). That is 57% less data throughput from 150% more SSDs.
Many organizations used enterprise scale-out NAS like Dell EMC Isilon or NetApp AFF to feed their HPC clusters with data when the clusters were small. Now as the clusters and their data are expanding fast with the growth of the AI and mod/sim workloads, for many organizations NFS-based NAS storage is either breaking economically ($ per terabyte) or architecturally (performance/scalability).
This most likely is why Hyperion Research has found in its 2020 special study that the use of NFS-based storage is shrinking while more and more organizations are going parallel to cope with the growth of AI and mod/sim workloads.
If you want to understand why these shifts are happening, please read this business white paper.
HPE has the right HPC storage for AI and mod/sim workloads for organizations of all sizes
We are the right partner for you to go parallel for storage—whether you want to start with 12 drives with HPE Parallel File System Storage, or if you are looking for a 50,000+ drive parallel storage system like ORNL with Orion.
- Compute options
Supercomputers like the HPE Cray EX, or ultra-dense CPU/GPU systems like the HPE Apollo 2000 Gen10 Plus and the HPE Apollo 6500 Gen10 Plus, or standard density rack servers like HPE ProLiant DL - Interconnect options
High speed, low latency networks like HPE Slingshot, InfiniBand HDR, or 100/200 Gigabit Ethernet - Parallel file system options
The leading parallel file system in research (Lustre) embedded in Cray ClusterStor E1000 storage systems, or the leading parallel file system in the enterprise (IBM Spectrum Scale) embedded in HPE Parallel File System Storage - Parallel data protection solutions
HDD- or Tape-based backup/archive/restore for parallel data—on-premise or off-premise (co-location or public cloud) with HPE Data Management Framework 7 - Consumption options
Purchasing, financing with HPE Financial Services, or fully managed as-a-service models with the HPE GreenLake edge-to-cloud platform
Put the storage beast on an effective diet. Go parallel with HPE. Contact your HPE representative today!
Do we have your attention? Check out these resources for more information!
Hear about the latest HPC and AI advancements: Visit the Discover More Network
View the infographic: Accelerate your innovation with HPC—with HPE as your end-to-end partner
Download the business paper: Spend less on HPC/AI storage and more on CPU/GPU compute
Visit the webpage: HPC & AI Storage
Related articles:
- When should old data be deleted? via HPE's Enterprise.nxt
- Managing storage: It's all about the data via HPE's Enterprise.nxt
- Stay current on top tech trends and expert advice. Sign up for the weekly newsletter.
Uli Plechschmidt
Hewlett Packard Enterprise
twitter.com/hpe_hpc
linkedin.com/showcase/hpe-ai/
hpe.com/us/en/solutions/hpc
UliPlechschmidt
Uli leads the product marketing function for high performance computing (HPC) storage. He joined HPE in January 2020 as part of the Cray acquisition. Prior to Cray, Uli held leadership roles in marketing, sales enablement, and sales at Seagate, Brocade Communications, and IBM.
