AutoML Decathletes, on your marks!

[Curt_Hopkins]

In July, Prof. Ameet Talwalkar from Carnegie Mellon University announced the launch of the “AutoML Decathlon,” a NeurIPS 2022 competition that will evaluate the performance of participants' automated machine learning (AutoML) methods on a diverse set of tasks. 

Hewlett Packard Enterprise, and specifically the AI Research Lab of Hewlett Packard Labs, is co-sponsoring the effort and providing the technical support to manage the queue and run the AutoML jobs of the competition participants.  The competition aims to establish a benchmark for the current state of AutoML and help the field more rapidly move towards a truly democratized ML toolkit that can be used by researchers and practitioners alike to explore a large space of neural model architectures.

Hewlett Packard Enterprise, Carnegie Mellon University, University of Wisconsin-Madison, and Morgan Stanley have partnered to co-sponsor the competition and support the technical activities. The competition is based on earlier collaborative work, namely NAS-Bench-360, between CMU, UW–Madison, and HPE. With evaluations on ten diverse tasks, NAS-Bench-360 is the first Neural Architecture Search testbed that goes beyond traditional AI domains such as vision, text, and audio signals.

To implement the experiment code in the testbed, the team used  HPE Machine Learning Development Environment, based on the determined.ai open-source software.

The AutoML Decathlon expands previous competition to a broader vision of understanding what the best approach is for a practitioner when faced with a modern ML problem, and how to select the best ML model for a given task.

“The competition team has curated 20 datasets which represent a broad spectrum of practical applications in scientific, technological, and industrial domains," says Cong Xu, a Senior Research Scientist at Hewlett Packard Labs and leader of this project. "These tasks vary in data type (image, finance time series, audio, and natural sciences), problem type (regression, single-label, and multi-label classification), and scale (ranging from several thousands to hundreds of thousands of observations). Unlike most past competitions in the AutoML community, competitors in the AutoML Decathlon are encouraged to submit a wide range of approaches from traditional hyperparameter optimization and assembling methods to modern techniques such as NAS and large-scale transfer learning.” 

To ensure efficiency and fairness, participants will submit jobs through a CodaLab frontend and queue, and the backend evaluation will be executed under a fixed computational budget on a farm of HPE Apollo 6500 Gen10 Plus GPU accelerated servers an ideal platform for HPC and deep learning jobs. A team in Grenoble, France has adapted the execution farm to be available as a CodaLab backend, and will manage the servers for the duration of the competition.

Figure 1 shows the competition timeline. The top winner (which will be announced in November) will receive a $15K first prize from HPE.

Figure 1: Summary of the AutoML Decathlon competition timeline.

To learn more, or if you’re interesting in submitting your entry to the competition, check out the following links:

NAS-Bench360: You can find the instructions to run the experiment codes using the open source version of Determined from the GitHub page at https://github.com/rtu715/NAS-Bench-360 and learn more about the benchmark and our various insights from the paper “NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks”, by Tu et al., available at https://arxiv.org/abs/2110.05668 

AutoML Decathlon: The competition is live as of July 15^th. You can learn more about the details at the competition website and the associated CodaLab website (for job submission) .