OpenFAM: When fabric-attached memory opens up

[Curt_Hopkins]

What happens when a ground-breaking new architecture turns into not one product with an innovative approach to computing, but into a host of them? Hewlett Packard Labs answered this question when the Memory-Driven Computing program became a fertile ground which launched not just a new product (though that’s happened too – the HPE Superdome Flex is the first product that uses Memory-Driven Computing principles), but new ways of looking at innovation from every direction.

OpenFAM creates a new storage tier wherein memory is attached to the communications fabric and disaggregated from compute nodes, enabling efficient computations over much larger data sets than are otherwise practical. To support developers who wish to program fabric-attached memory (FAM), HPE has defined OpenFAM, an API (application programming interface) specification for programming FAM and an open source reference implementation of that specification written in C++.

According to Sharad Singhal, Distinguished Technologist at Hewlett Packard Labs, this is in keeping with both HPE’s long tradition of computing innovation and its equally long tradition of open source development. This wasn’t a matter of Labs or HPE deciding to try to sell OpenFAM as a product. It was instead HPE customers who came to Singhal’s team and told them “that they believed the architecture was very much alive,” especially in service of the specialized applications that have grown out of their HPC undertakings.

These customers wish to take advantage of huge data sets that arise in applications in areas such as life sciences and cyber security, sets so large they won’t fit into the local memory of processors. OpenFAM allows the users to employ FAM in large clusters they use as a really fast storage tier, at an order of magnitude larger and faster than current storage options.

On the open source side of this development, Singhal stresses that this is “not just a research project,” he said. The code sitting on Github lives in a pure public open source development environment. “If we’re going to make this real you can’t put out toy projects,” he said, “customers have to be able to use it.” And they can.  

OpenFAM is not just an altruistic gesture, it is most definitely a practical one as well. Since FAM represents a new architecture, existing programming APIs do not provide easy-to-use libraries for use by developers. Most existing libraries assume data is held either in persistent storage or in random access memory (RAM) during computation. It is important to provide developers libraries that are easy to use, and at the same time enable them to try out practical applications. While commercial systems supporting FAM are still being developed, it is also important that the software provide value to developers today. OpenFAM was designed to run on current high-performance clusters as well as future fabric-attached memory systems. Thus developers can take advantage of the library for their data needs today. 

“The commercial availability of storage class memory technology, HPE’s Slingshot technology, and industry support for the CXL interface have finally made fabric-attached persistent memory a reality,” says David Emberson, Distinguished Technologist, HPC Advanced Technology Group. According to him, the appearance of HPC systems with integrated FAM represents the first real architectural innovation in high performance computing since the emergence of HPC clusters in the mid-1990s. 

At HPE, new data-driven HPC architectures for shared memory databases, data analytics, and machine learning applications are in development. “OpenFAM is an example of the close collaboration between Labs researchers and the HPC and MCS product engineering organizations,” says Emberson.

OpenFAM is available to the community on the OpenFAM page on Github.

Curt Hopkins
Hewlett Packard Enterprise

twitter.com/hpe_labs
linkedin.com/showcase/hewlett-packard-labs/
labs.hpe.com