294 research outputs found
Neural Architecture Search: Insights from 1000 Papers
In the past decade, advances in deep learning have resulted in breakthroughs
in a variety of areas, including computer vision, natural language
understanding, speech recognition, and reinforcement learning. Specialized,
high-performing neural architectures are crucial to the success of deep
learning in these areas. Neural architecture search (NAS), the process of
automating the design of neural architectures for a given task, is an
inevitable next step in automating machine learning and has already outpaced
the best human-designed architectures on many tasks. In the past few years,
research in NAS has been progressing rapidly, with over 1000 papers released
since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized
and comprehensive guide to neural architecture search. We give a taxonomy of
search spaces, algorithms, and speedup techniques, and we discuss resources
such as benchmarks, best practices, other surveys, and open-source libraries
Automatic machine learning:methods, systems, challenges
This open access book presents the first comprehensive overview of general methods in Automatic Machine Learning (AutoML), collects descriptions of existing systems based on these methods, and discusses the first international challenge of AutoML systems. The book serves as a point of entry into this quickly-developing field for researchers and advanced students alike, as well as providing a reference for practitioners aiming to use AutoML in their work. The recent success of commercial ML applications and the rapid growth of the field has created a high demand for off-the-shelf ML methods that can be used easily and without expert knowledge. Many of the recent machine learning successes crucially rely on human experts, who select appropriate ML architectures (deep learning architectures or more traditional ML workflows) and their hyperparameters; however the field of AutoML targets a progressive automation of machine learning, based on principles from optimization and machine learning itself
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
Over the past half-decade, many methods have been considered for neural
architecture search (NAS). Bayesian optimization (BO), which has long had
success in hyperparameter optimization, has recently emerged as a very
promising strategy for NAS when it is coupled with a neural predictor. Recent
work has proposed different instantiations of this framework, for example,
using Bayesian neural networks or graph convolutional networks as the
predictive model within BO. However, the analyses in these papers often focus
on the full-fledged NAS algorithm, so it is difficult to tell which individual
components of the framework lead to the best performance.
In this work, we give a thorough analysis of the "BO + neural predictor"
framework by identifying five main components: the architecture encoding,
neural predictor, uncertainty calibration method, acquisition function, and
acquisition optimization strategy. We test several different methods for each
component and also develop a novel path-based encoding scheme for neural
architectures, which we show theoretically and empirically scales better than
other encodings. Using all of our analyses, we develop a final algorithm called
BANANAS, which achieves state-of-the-art performance on NAS search spaces. We
adhere to the NAS research checklist (Lindauer and Hutter 2019) to facilitate
best practices, and our code is available at
https://github.com/naszilla/naszilla
- …