Lustre file system continues to gain luster in HPC

[UliPlechschmidt]

Learn how the HPC file system landscape is changing and what that means for Lustre, the leading parallel file system.

October 13 is the first day of the annual Lustre Administrators and Developers Workshop (LAD) when worldwide Lustre administrators and developers gather to exchange their experiences, developments, tools, and best practices.

It’s remarkable what the Lustre community has achieved since December 2003 when version 1.0 of the open source parallel file system was released.

This is why today is a good time to share the results of a recent study conducted by the independent analyst firm Hyperion Research— “Shifts Are Occurring in the File System Landscape”—with a broader audience.

Over the last five years, Hyperion Research has conducted several broad MultiClient Studies (MCS) encompassing all aspects of the HPC ecosystem including file system usage. This offers a clear view on how the use of file systems in HPC has shifted since 2015. The focus is on the top three files systems by usage:

1. NFS
2. Lustre
3. IBM Spectrum Scale

Here is what Hyperion Research found based on the responses of HPC users with 1,864 HPC systems.

• While NFS remains the most widely adopted file system, it has dropped from being utilized at 54% of the sites down to 46%. NFS was one of the first file systems that could handle the initial scale of early HPC systems. Its first mover status, coupled with the fact that it's still adequate in smaller scale HPC systems today, is reflected in its continued, albeit shrinking, wide adoption.

• Lustre utilization has grown from 21% to 32.5%, and Lustre's open source approach has enabled it to mature its feature set across a wide number of areas including performance, resiliency, reliability, and scalability. This maturity has also given it stability and the capability of scaling to meet the demands of petascale and emerging exascale configurations.

• GPFS/IBM Spectrum Scale adoption has grown at a slightly lower rate, from 23% to 26.8%. This modest growth in adoption can be attributed to a slower feature advancement pace due to its proprietary nature along with a market shift away from IBM's dominance in the HPC sector and a change in the pricing model away from user-license-based to capacity-based.

The study also found that Lustre was the only parallel file system that has demonstrated consistent growth over study period. Congratulations and a big thank you to all the Lustre developers out there. Your hard work and passion continue to pay off!

It also must be very rewarding for the developer community to see that all the top public cloud service providers like AWS, Google Cloud Platform or Microsoft Azure have decided to offer Lustre-based storage.

HPE recently has launched the Lustre-based Cray ClusterStor E1000 storage system—that was selected by all three of the U.S. Department of Energy’s exascale leadership sites—for which HPE now also can provide enterprise-grade Lustre support. Historically, only two Lustre organizations were able to provide enterprise-grade support for Lustre users. One is the Whamcloud team that was acquired in 2012 by Intel and in 2018 by DDN. The other is the ClusterStor team that was acquired by Xyratex in 2010, Seagate in 2014, Cray in 2017, and now finally has found its permanent home at HPE.

If you want to learn more about the Cray ClusterStor E1000 storage system, please view the infographic and read the business paper or the technical paper.

In closing, HPE would like to give a big shout out to all Lustre developers worldwide—independent of the organization they work for—for their impact on the HPC community in the last nearly 17 years. Here's to the next 17 years and to a bright future for the whole Lustre community!

Uli Plechschmidt
Hewlett Packard Enterprise

twitter.com/hpe_hpc
linkedin.com/showcase/hpe-ai/
hpe.com/us/en/solutions/hpc