Build better insight faster: Advance your business by combining HPC simulations and AI technique

[BenjaminRobbins]

SmartSim gives businesses not only the ability to integrate modern AI methodology into their work but also leverages a new paradigm for rapid data communication at scale. Learn how SmartSim works and what opportunities it brings to businesses in every industry.

Recently, interest in applying machine learning (ML) algorithms to improve scientific simulations has been increasing. That’s exactly why we developed SmartSim. This open-source library enables the use of ML with existing traditional high-performance computing (HPC) simulations. Adding a couple of lines of code to the existing simulation enables a host of new ML capabilities.

Simply put, here’s how SmartSim works

SmartSim connects traditional models written in Fortran, C, and C++ to the modern data science stack. Users can write Machine Learning and AI models with PyTorch, TensorFlow, Scikit-Learn, etc. and use them immediately inside their scientific simulations.

For example, SmartSim allows users to add models that have been trained from data that the simulation has run. What’s more, SmartSim performs at HPC scale, so it can be used on simulations spanning thousands of processors—all without writing to files.

Watch this short video to learn more about what SmartSim is and how your organization can benefit from using this innovative new framework.

Opening a world of possibilities between simulation and ML

This question has been a challenging one to answer: What is the optimum way to add machine learning (ML) capabilities to traditional HPC simulations? On the surface, it seems that our answer should just focus on AI. However, we saw a deeper opportunity that enables further computational possibilities. 

We believe the true difficulty—and opportunity—in bridging the two worlds of simulation and ML needed to be reformulated not just in terms of adding AI and ML to simulations but also in terms of data exchange. This raised a different question for us: How can we efficiently exchange data between a simulation and ML methodologies at scale? SmartSim provides the answer.

As you can see in the schematic below, SmartSim allows users to exchange data between existing simulations and an in-memory database. This in-memory database leverages Redis while the simulation is running—and enables developers to take additional action on the fly that is not just limited to ML. 

Figure 1: SmartSim higher-level architecture.

While the convergence of simulations and AI provides many unique options to explore, we believe that the critical question of how data is exchanged at scale represents a paradigm shift that is more important. By focusing on data exchange rather than just adding ML to simulations, we’re bringing new capabilities alongside existing simulations, such as online inference, online learning, online analysis, reinforcement learning, computational steering, and interactive visualization, to name a few.

Figure 2: How can you use SmartSim? Key characteristic = data exchange between simulation and in-memory database.

For the many organizations wanting to leverage simulations for “what if” scenarios that model the real world, SmartSim opens the door to myriad new possibilities. That’s because it’s always much faster to model things like material design, drug molecules, and multi-physics interactions, in a computer simulation than in physical form.

Doing what’s never been done before

In a recent collaboration between HPE and National Center for Atmospheric Research (NCAR), SmartSim enabled NCAR to run a climate-scale oceanographic model simulation showing movement of energy in the ocean with a level of detail that previously was simply too computationally expensive for use in production decadal-to-century climate models. Using AI and SmartSim, HPE and NCAR were able to replace time-consuming calculations with AI. SmartSim’s remarkable time-saving capability and the resulting global scale simulation will help scientists better understand the impacts of climate change on our planet, businesses, and lives—and most importantly, arm global citizens with the knowledge to mitigate the impacts. 

To hear more concrete use cases you can apply to your business, catch this discussion I had with HPE climate science expert Ilene Carpenter as part of our recent HPE Discover session: How AI and Climate Science Can Help You Run Your Business Better.  (Note: You do need to first register for HPE Discover to view the on-demand session.)

A smart solution for HPC and AI whose time has come

Take a close look at SmartSim and you’ll see how this new open source software framework for HPC and AI gives you two exciting (and much-needed) capabilities. One, you can integrate modern AI methodology and ML into your simulation work. Two, you can leverage this new paradigm for rapid data communication as scale. And all can be done with minimal changes to code. The result? More insight faster which in turn opens the door to more opportunity for your business, whatever industry you’re in.

Looking for additional information on this topic? You'll find more talks, papers, and source code on our GitHub repository.

---

Benjamin Robbins
Hewlett Packard Enterprise

twitter.com/hpe_hpc
linkedin.com/showcase/hpe-ai/
hpe.com/info/hpc

[Lin Pengcheng]

This is the beginning of Cray's application of "warehouse/workshop model"

warehouse: redis

workshop: traditional models written in Fortran, C, and C++ to the modern data science stack