It's lonely at the top: Lustre continues to dominate top 100 fastest supercomputers

[UliPlechschmidt]

The Lustre parallel file system continues to dominate at the supercomputing high end where performance and scalability matter most. Take a look at the trends in storage and why Lustre is the leader. 

The November 2020 TOP500 list is out! Processing speeds grab the headlines of this twice-yearly list of the world’s fastest supercomputers. But dig deeper into the data and you find a whole range of HPC industry trends.

Take storage, for example. Within the top 100 fastest supercomputers, 77 feature the open source Lustre® file system. That number is up three since the last TOP500 list was issued in June. Those numbers show a clear and growing preference from the industry at the high end. No other technology is even close. 

To understand these storage realities, let’s first take a broader view at HPC storage in general.

One size does not fit all

When it comes to HPC storage technologies, it’s important to understand that one size doesn’t fit all. Choosing which storage technology to deploy depends on the required throughput performance (measured in gigabyte per second) and the size of the single namespace (measured in terabyte usable capacity).

For the following discussion, Hyperion Research offers a useful classification system. It segments the HPC market into different “bands” depending on the purchase price of the HPC server systems the HPC storage is attached to:

• Supercomputer band for systems priced at $500,000 and up
• Divisional band for systems priced between $250,000 and $499,000
• Departmental band for systems priced between $100,000 and $249,000
• Workgroup band for sub-$100,000 systems 

The following table gives a simplified view of the different bands and the typically deployed storage technologies for primary storage.

Band	Storage technology	File systems	Product example
Supercomputer	Parallel File System	Lustre, IBM Spectrum Scale	Cray ClusterStor E1000
Departmental/Divisional	Network Attached Storage (NAS)	NFS	HPE solutions for Qumulo
Workgroup	Node-local Storage	Linux (e.g. EXT4)	SSD/HDD options for HPE Apollo 2000

If you only have a handful of compute nodes in a workgroup cluster, internal SSDs or HDDs often are enough. But if you have dozens of compute nodes, data often needs to be shared between compute nodes. In these cases, NAS systems are typically deployed.

The need changes yet again when customers are growing to hundreds or even thousands of compute nodes. That’s when you need parallel file systems because NAS at some point runs into architectural scaling limitations.

Due to their huge number of compute nodes, the largest supercomputers all deploy parallel file systems for their external shared storage. The following graphic shows the distribution of parallel file systems of the top 100 supercomputers on the November 2020 TOP500 list.

Top 100 list by parallel file system

Conclusions from the TOP500 list

Analysis of the TOP500 list show the Lustre system’s continued and growing dominance. Here are the facts:

• 77 of the world’s top 100 supercomputing systems feature Lustre—up 3 from the June 2020 list
• IBM Spectrum® Scale holds the number two spot with 23 systems—down 3 from June
• No other parallel file systems have made it into the top 100—EVER

Lustre and IBM Spectrum Scale are the sole contenders in the HPC parallel file system market. Specifically, Lustre continues to grow at the high end where performance and scalability matter most. As for IBM Spectrum Scale, it continues to grow in the mid-range—boosted in part by the IBM Spectrum Scale Erasure Code Edition designed to run on secure off-the-shelf rack servers like the HPE ProLiant DL family.

Newer file systems like Weka Matrix and VAST Data struggle to get traction in HPC. That’s due in part to the extreme price/performance requirements as they are targeted at feature rich but expensive enterprise storage like Dell EMC PowerScale (formerly known as Isilon), clustered ONTAP from NetApp or Pure Storage FlashBlade.

However you interpret the trends, the future is exciting for high-performance storage. For more on this quickly changing subject, read the white paper “The New HPC Era Needs New HPC Storage.”

Uli Plechschmidt
Hewlett Packard Enterprise

twitter.com/hpe_hpc
linkedin.com/showcase/hpe-ai/
hpe.com/us/en/solutions/hpc