Train machine learning models on supercomputers

[TechExperts]

Learn about updates to the HPE Machine Learning Development Environment. See how it enables you to harness the power of supercomputers to train machine learning (ML) models by integrating with the Slurm workload manager and Singularity container runtime.    By Vishnu Mohan, Product Manager AI/ML, HPE

The HPE Machine Learning Development Environment already offered the fastest and easiest way to train machine learning models at scale. Now, it’s poised to make it even faster and easier for developers to bridge the AI/ML worlds and HPC worlds. Here’s how.

We’re announcing the integration of the HPE Machine Learning Development Environment with the HPE Cray Dispatcher for Slurm and Singularity support. This means we are enabling the HPE Machine Learning Development Environment to be deployed on supercomputers to bring you the best model training experience possible – complete with Slurm and Singularity support. As you no doubt know, Slurm is the leading open-source workload and resource manager used by many of our HPC customers today. Likewise, Singularity brings its own set of unique features as a popular tool for running software containers on HPC systems.

It’s excellent news for model developers – enabling you to tap into the power of HPC

With the release of this update to the HPE Machine Learning Development Environment, you can tap into HPC environments for machine learning. It’s like you’re on your laptop but you’re actually connecting to and working off the power of a supercomputer. Everyone always says it – but it’s truly the best of both worlds.

Slurm is a free, open-source job scheduler used on many of the world’s supercomputers and high-performance compute clusters. Singularity is also a free, open source, cross platform software that containerizes scientific computing and HPC applications. That fact that the HPE Machine Learning Development Environment supports these two key open source technologies means that you can train your AI/ML models in HPC or supercomputing environments to drive faster business results.

As can be seen from the diagram below, getting started with training machine learning models on your supercomputer is easier than ever. Deploy and configure the HPE Machine Learning Development Environment’s Master and the HPE Cray Dispatcher on a node that has the Slurm CLI installed – you’re all set.

 So get ready to break through the challenges of niche requirements and restraints that make it difficult for users to just “run” on an HPC system. You’ll also benefit from greater flexibility when it comes to choosing different hardware, interconnects, and accelerators.

What’s more, when you opt to purchase the HPE Machine Learning Development Environment, the new Slurm and Singularity integrations are part of the product – with no separate licensing needed.

We’re just getting started!

We’re committed to expanding our portfolio to support more accelerators and supercomputing platforms for AI and HPC, bringing greater scale wherever and whenever you need it most.

Stay tuned for more exciting announcements from the leaders in AI and HPC.

---

Meet Tech Experts blogger Vishnu Mohan, Product Manager AI/ML, HPE

Vishnu leads the product team for HPE Machine Learning Development Environment, a platform whose vision is to become the de facto standard for teams that are building machine learning models at scale. He was previously a director of product management at Mesosphere (now D2iQ) where he managed a product portfolio comprising container orchestration, data & analytics, and developer services. Vishnu holds a master’s degree in computer science from UT Dallas.

Insights Experts
Hewlett Packard Enterprise

twitter.com/HPE_AI
linkedin.com/showcase/hpe-ai/
hpe.com/us/en/solutions/artificial-intelligence.html