Critical role of HPC Storage in autonomous vehicle development

[UliPlechschmidt]

A main obstacle to developing fully autonomous vehicles is capturing and processing the data generated by fleets of training cars. This massive amount of data creates a storage challenge that autonomous vehicle developers cannot overcome using their existing enterprise storage architectures.

Self-driving cars were once only the stuff of science fiction. But thanks to recent computing advances, they are becoming more of a reality every day. Still, a number of issues need addressing before fully autonomous vehicles share our roadways. And one of the largest is capturing and processing the data generated by fleets of training cars. This massive amount of data creates an unprecedented storage challenge that autonomous vehicle development companies cannot overcome using their existing enterprise storage architectures.

Luckily, we’ve made it our mission to provide technology that solves the world’s most complex computing challenges, so this problem is right in our wheelhouse.

First, a little background. Driving a car is second nature to many of us, so it’s easy to forget that safely operating a vehicle requires lightning-quick reaction times and the ability to respond to a variety of changing conditions in just milliseconds. When you think about the endless variety of scenarios that could come your way behind the wheel, you begin to realize what a massive job it is to then simulate them all. The fastest and most efficient way to simulate these situations and train models for autonomous driving systems is do it on an extremely large scale using machine learning and AI techniques.

It starts with sensors placed on fleets of training cars. These sensors collect information about a variety of operational, situational, environmental, and weather-related conditions. And the datasets they generate are massive. To put it into perspective, a fleet of 100 cars where each vehicle generates 5 TB per hour and runs for 8 hours per day will generate a total of 4 PB of data per day, or up to 20 PB per week. That’s equivalent to uploading just over 133 billion photos to Facebook. On top of that, projects focused on delivering fully autonomous vehicles will likely take years to complete.

This is no small amount of data to ingest, store, process, and manage

So, companies are coming to us to help them implement larger and more powerful storage architectures. Our answer to them is simple: only an HPC storage architecture will be able to deliver the scalability, performance, and affordability needed to address these extreme storage requirements.

Storage architectures set up to support autonomous vehicle projects must be able to: manage massive amounts of data in a single namespace, ingest/process data with extreme performance, and do it all without becoming cost prohibitive. These primary considerations – affordability, scalability, and throughput—make up what I like to call the “autonomous driving storage challenge triangle”:

Most early adopters are learning the hard way that simply scaling up existing enterprise storage technologies will not address some or all of the requirements of the storage challenge triangle. HPC storage can address extreme storage requirements because it scales easily, cost effectively, and is generally more focused on extracting the maximum performance out of each infrastructure device.

Cray ClusterStor HPC storage is our answer to the storage challenge for autonomous driving. ClusterStor incorporates the industry’s highest-performing and most efficient data storage technologies in an engineered HPC storage system that delivers maximum storage performance at the lowest overall TCO.

The secret to ClusterStor is a zero-waste, totally balanced system architecture that deploys the right technology mix at exactly the right time. Each ClusterStor system is pre-integrated, tested, and tuned to enable rapid deployment right out of the box. And it provides system management tools for insight into system health and performance. With ClusterStor, you can achieve sustained linear performance and application predictability even when dramatically scaling up capacity, and get up and running faster than ever before.

We want to help autonomous vehicle development companies succeed by addressing their clear need for scalability, performance, and affordability. We’re a unique player in the marketplace with a strong track record of combining high-performance I/O, storage, and networking technologies in custom-built systems that help our customers arrive at results faster with larger datasets. Our ClusterStor solution is the only HPC storage product on the market today that can offer the right performance, density, and capacity for autonomous vehicle projects at a budget that won’t break the bank.

This blog originally published on cray.com and has been updated and published here on HPE’s Advantage EX blog.

---

Uli Plechschmidt
Hewlett Packard Enterprise

twitter.com/hpe_hpc
linkedin.com/showcase/hpe-ai/
hpe.com/us/en/solutions/hpc