peer reviewedWe introduce AlphaD3M, an automatic machine learning (AutoML) system based on
meta reinforcement learning using sequence models with self play. AlphaD3M is
based on edit operations performed over machine learning pipeline primitives
providing explainability. We compare AlphaD3M with state-of-the-art AutoML
systems: Autosklearn, Autostacker, and TPOT, on OpenML datasets. AlphaD3M
achieves competitive performance while being an order of magnitude faster,
reducing computation time from hours to minutes, and is explainable by design

Cho, Kyunghyun

DE PAULA LOURENCO, Raoni

Drori, Iddo

Freire, Juliana

Krishnamurthy, Yamuna

Piazentin Ono, Jorge

Rampin, Remi

Silva, Claudio

arXiv

Open Repository and Bibliography - Luxembourg

ALPHAD3M: MACHINE LEARNING PIPELINE SYNTHESISIDDO DRORI idrori@nyu.eduYAMUNA KRISHNAMURTHY yamuna@nyu.eduREMI RAMPIN remi.rampin@nyu.eduRAONI DE PAULA LOURENCO raoni@nyu.eduJORGE PIAZENTIN ONO jorgehpo@nyu.eduKYUNGHYUN CHO kyunghyun.cho@nyu.eduCLAUDIO SILVA csilva@nyu.eduJULIANA FREIRE juliana.freire@nyu.eduABSTRACTWe introduce AlphaD3M, an automatic machine learning (AutoML) system based on meta rein-forcement learning using sequence models with self play. AlphaD3M is based on edit operationsperformed over machine learning pipeline primitives providing explainability. We compare Al-phaD3M with state-of-the-art AutoML systems: Autosklearn, Autostacker, and TPOT, on OpenMLdatasets. AlphaD3M achieves competitive performance while being an order of magnitude faster,reducing computation time from hours to minutes, and is explainable by design.1 IntroductionAutomatic machine learning (AutoML) aims to learn how to learn. Given a dataset, a well defined task, and performancecriteria, the goal is to solve the task with respect to the dataset while optimizing performance. Existing systems havefocused on a relatively small set of machine learning primitives, with a few tasks (Feurer et al., 2015), or on a small setof datasets (Chen et al., 2018), or on numerous datasets within specific domains (Olson and Moore, 2016).DARPA’s Data Driven Discovery of Models (D3M) program pushes this vision further and proposes to developinfrastructure to automate model discovery, i.e., solve any task on any dataset specified by the user. Using a broad setof computational primitives as building blocks, the D3M system should synthesize a pipeline and set the appropriatehyper-parameters to solve a previously unknown data and problem. The D3M system also has a user interface thatenables users to interact with and improve the automatically generated results (Blei and Myth, 2017).Inspired by AlphaZero (Silver et al., 2017), we frame the problem of pipeline synthesis for model discovery as asingle-player game (McAleer et al., 2018): the player iteratively builds a pipeline by selecting among a set of actionswhich are insertion, deletion, replacement of pipeline parts. An inherent advantage of this approach is that at the endof the process, once there is a working pipeline, it is completely explainable, including all the actions and decisionswhich led to its synthesis. Another advantage is that our approach leverages recent advances in deep reinforcementlearning using self play, specifically expert iteration (Anthony et al., 2017) and AlphaZero (Silver et al., 2017), by usinga neural network for predicting pipeline performance and action probabilities, along with a Monte-Carlo Tree Search(MCTS), as illustrated in Figure 1 (left), which takes strong decisions based on the network. The process progresses byself play with iterative self improvement, and is known to be highly efficient at finding a solution to search problemsin very high dimensional spaces. We evaluate our approach using the OpenML dataset on the tasks of classificationand regression, demonstrating competitive performance and computation times an order of magnitude faster than otherAutoML systems.Each of the existing AutoML systems uses any one of the following key elements individually: differentiable program-ming, tree search, evolutionary algorithms, and Bayesian optimization, to find the best machine learning pipelines for agiven task and dataset. Differentiable programming, of which neural network backpropagation is a special case, is usedfor learning feature extraction and estimation (Ganin and Lempitsky, 2015) and for end-to-end learning of machinelearning pipelines with differentiable primitives (Milutinovic et al., 2017). Bayesian optimization methods are used forarXiv:2111.02508v1  [cs.LG]  3 Nov 2021ALPHAD3M: MACHINE LEARNING PIPELINE SYNTHESISAlphaZero AlphaD3MGame Go, chess AutoMLUnit piece pipeline primitiveState configuration meta data, task, pipelineAction move insert, delete, replaceReward win, lose, draw pipeline performanceFigure 1: AlphaD3M iterative improvement (left); AlphaD3M game representation (right).hyper-parameter tuning (Bergstra and Bengio, 2012). Both AutoWEKA (Kotthoff et al., 2017) and Autosklearn (Feureret al., 2015) extend the application of these techniques to the selection of the model in addition to the hyper-parametervalues, solving the combined algorithm selection and hyper-parameter optimization problem by fitting probabilisticmodels capturing the relationship between parameter values and performance measures using a Gaussian Process,Random Forest, or tree-structured Parzen estimator (Bergstra et al., 2011). Auto-Tuned Models (Swearingen et al., 2017)represent the search space as a tree with nodes being algorithms or hyperparameters and searches for the best branchusing a multi-armed bandit. TPOT (Olson and Moore, 2016) and Autostacker (Chen et al., 2018) uses evolutionaryalgorithms to generate machine learning pipelines while optimizing their hyperparameters. TPOT represents machinelearning pipelines as trees, whereas Autostacker represents them as stacked layers.Our goal is to search within a large space for the machine learning, and pre and post processing primitives and parameterswhich together constitute a pipeline for solving a task on a given dataset. The problem is that of high dimensionalsearch. Although the datasets differ, the solution pipelines contain recurring patterns. Just as a data scientist developsintuition and patterns about the pipeline components, we use a neural network along with a Monte-Carlo tree search inan iterative process. This combination results in the network learning these patterns while the search splits the probleminto components and looks ahead for solutions. By self play and evaluations the network improves, incorporating abetter intuition. An advantage of this iterative dual process is that it is computationally efficient in high dimensionalsearch (Silver et al., 2017).2 MethodsFollowing dual process theory, we solve the meta learning problem by sequence modeling using a deep neural networkand Monte Carlo tree search (MCTS) (Silver et al., 2017; Anthony et al., 2017). This section describes our representation,followed by details of the neural network and MCTS.2.1 RepresentationFigure 1 (right) illustrates a high level analogy between a two player competitive game and our single player pipelinesynthesis game, including state, action, and reward. A pipeline is a data mining work flow, of pre-processing, featureextraction, feature selection, estimation, and post-processing primitives. Algorithm 1 describes our pipeline staterepresentation. Our architecture models meta data and an entire pipeline chain as state rather than individual primitives.A pipeline, together with the meta data and problem definition is analogous to an entire game board configuration. Theactions are transitions from one state (pipeline) to another.2.2 Neural NetworkAlphaD3M uses a recurrent neural network, specifically an LSTM. Let fθ(s) = (P (s, a), v(s)), where P (s, a) is theaction probabilities and v(s) the evaluation score of the model predicted by the network f with parameters θ, for agiven dataset D and task T , for a given state s. The neural network predicts the probabilities over actions a which leadto sequences S that describe a pipeline, which in turn solves the given task on the dataset. The network inputs aretraining examples (st, πt, et) from games of self play, where st is the state at time t, πt the policy estimated by MCTS,and et the actual pipeline evaluation at the end of the game. The state st is composed of a vector encoded as describedin Algorithm 1. The network outputs are probabilities over actions P (s, a), and an estimate of pipeline performance v.2ALPHAD3M: MACHINE LEARNING PIPELINE SYNTHESISAlgorithm 1 Pipeline EncodingGiven datasets D, tasks T , and a set of possible pipeline sequences S1, . . . , Sn, from the available machine learning,and data pre and post processing primitives.• For each dataset Di and task Tj :1. Encode dataset Di as meta data features f(Di).2. Encode task Tj .3. Encode the current pipeline at time t by a vector St.4. Encode action fa(St), so policy π maps (f(Di), Tj , St) to fa(S1), . . . , fa(Sn).We optimize the network parameters θ by making the predicted model S match the real world modelR and the predictedevaluation results v match the real world evaluation e, by minimizing the cross entropy loss between S and R, andthe mean squared error between v and e. We add an `2 regularization term for the network parameters θ to avoidover-fitting and an `1 regularization term which prefers simple pipelines. Thus our network fθ is trained by minimizingthe following non-linear loss function using stochastic gradient descent:L(θ) = S logR+ (v − e)2 + α‖θ‖2 + β‖S‖1. (1)2.3 Monte Carlo Tree SearchOur algorithm takes the predictions (P (s, a), v(s)) of the neural network and uses them in a MTCS by running multiplesimulations to search for a pipeline sequence R with a better evaluation. The search result R improves upon thepredicted result S given by the network by improving the network policy using the update rule:U(s, a) = Q(s, a) + cP (s, a)√N(s)1 +N(s, a), (2)where Q(s, a) is the expected reward for action a from state s, N(s, a) is the number of times action a was taken fromstate s, N(s) the number of times state s was visited, P (s, a) is the estimate of the neural network for the probabilityof taking action a from state s, and c is a constant which determines the amount of exploration. At each step of thesimulation, we find the action a and state s which maximize U(s, a) and add the new state to the tree if it does not existwith the neural network estimates (P (s, a), v(s)) or call the search recursively otherwise. Next, the model representedby R is realized and applied to the data to solve the task, resulting in a better evaluation e which is the result of runningthe generated pipeline R on the data and task. Thus the real world search provides us with (R, e), where R is the realworld model, consisting of machine learning primitives, and e the real world evaluation of the model and pipeline usingthose primitives on the data and task.The neural network predictions, the MCTS model, and the real world evaluation, together, define a loss function shownin Equation 1, which is minimized to improve the neural network parameters. This process continues iteratively untilthe best model, which automatically solves the task, is found.Inspired by the neural editor (Guu et al., 2018) we use edit operations that make the pipeline generation explainable bydesign. For each iteration of self play the MCTS searches the possible valid pipelines. For each state or pipeline thenext possible states or pipelines are limited to those derived from the edit operations of the current state.3 ResultsThe data consists of 313 different tabular datasets, of which 296 are from OpenML (Vanschoren et al., 2014). Weconsidered classification, both binary (121 datasets) and multi-class (108 datasets), and univariate regression tasks (84datasets). Baseline pipelines were constructed using sklearn SGD estimators for classification and regression, and anannotated tabular feature extractor which uses linear SVC, Lasso, percentile classification or regression estimators fromsklearn.Figure 2 compares performance between AlphaD3M and SGD which is the baseline pipeline. Each of the 180 pointsrepresents a classification task on a different OpenML dataset. The datasets for which AlphaD3M performs better thanSGD are shown by green circles and those for which SGD performs better are shown by red crosses. Figure 2 showsthat AlphaD3M performs better than baseline for 75% of the datasets, both are comparable for 18% of the datasets, andperforms worse for only 7% of the datasets. Figure 3 shows the normalized difference in cross validation performance3ALPHAD3M: MACHINE LEARNING PIPELINE SYNTHESISFigure 2: AlphaD3M vs. SGD performance for 180 classification tasks on OpenML datasets.Figure 3: Comparison of normalized AlphaD3M performance t with SGD baseline performance b by estimator.of AlphaD3M t and SGD baseline b for a classification task for 180 datasets, split according to the estimators used byAlphaD3M, demonstrating better performance across diverse estimators.Figure 4 compares performance between different AutoML methods: Autosklearn, TPOT, and Autostacker, and ourmethod AlphaD3M, for a number of common OpenML datasets, which serve as representative benchmark datasetsfor AutoML systems (Olson et al., 2017; Olson and Moore, 2016; Chen et al., 2018). For each method and dataset,we compute the performance mean and standard deviation by repeated evaluation. As shown in Figure 4 our method,AlphaD3M, is competitive with other approaches. All four methods are competitive and on par, as their performanceincluding confidence intervals intersect; whereas SGD and Random Forest are not competitive with the leading AutoMLmethods.AlphaD3M is implemented using PyTorch. Our implementation takes advantage of GPUs while training the neuralnetwork and uses CPUs for the MCTS. Table 1 compares the running time of TPOT, Autostacker, and AlphaD3M onthe same datasets, along with the corresponding speedup factors. Table 1 shows that AlphaD3M performs on averagean order of magnitude faster, reducing computation time from hours to minutes.4 ConclusionsWe introduced AlphaD3M, an automatic machine learning system with competitive performance, which is an order ofmagnitude faster than existing state-of-the-art AutoML methods, reducing computation time from hours to minutes. We4ALPHAD3M: MACHINE LEARNING PIPELINE SYNTHESISFigure 4: Comparing between performance of AutoML methods on OpenML datasets.Table 1: Running time comparison (in seconds and speedup factors).Dataset/Method TPOT Autostacker AlphaD3M Speedup vs TPOT Speedup vs ASbreast cancer 3366 1883 460 7.3 4hill valley 17951 8411 556 32.2 15.1monks 1517 1532 348 4.3 4.3pima 5305 1940 619 8.5 3.1spectf 4191 1673 522 8 3.2vehicle 16795 4010 531 31.6 7.5presented the first single player AlphaZero game representation applied to meta learning by modeling meta-data, task,and entire pipelines as state.ReferencesT. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. In Conference on NeuralInformation Processing Systems, 2017.J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research,2012.J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. Conference on NeuralInformation Processing Systems, 2011.D. M. Blei and P. Myth. Science and data science. In Proceedings of National Academy of Sciences, 2017.B. Chen, H. Wu, W. Mo, I. Chattopadhyay, and H. Lipson. Autostacker: A compositional evolutionary learning system.The Genetic and Evolutionary Computation Conference, 2018.M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter. Efficient and robust automatedmachine learning. In Conference on Neural Information Processing Systems, 2015.Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. International Conference on MachineLearning, 2015.K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang. Generating sentences by editing prototypes. Transactions of theAssociation for Computational Linguistics, 2018.L. Kotthoff, C. Thornton, H. H. Hoos, F. Hutter, and K. Leyton-Brown. Auto-WEKA 2.0: Automatic model selectionand hyperparameter optimization in WEKA. The Journal of Machine Learning Research, 18(1), 2017.S. McAleer, F. Agostinelli, A. Shmakov, and P. Baldi. Solving the Rubik’s cube without human knowledge. arXivpreprint arXiv:1805.07470, 2018.M. Milutinovic, A. G. Baydi, R. Zinkov, W. Harvey, D. Song, and F. Wood. End-to-end training of differentiablepipelines across machine learning frameworks. In International Conference on Learning Representations, 2017.5ALPHAD3M: MACHINE LEARNING PIPELINE SYNTHESISR. Olson and J. Moore. TPOT: A tree-based pipeline optimization tool for automating machine learning. In InternationalConference on Machine Learning, 2016.R. S. Olson, W. La Cava, P. Orzechowski, R. J. Urbanowicz, and J. H. Moore. Pmlb: a large benchmark suite formachine learning evaluation and comparison. BioData mining, 2017.D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel,et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. Conference on NeuralInformation Processing Systems, 2017.T. Swearingen, W. Drevo, B. Cyphers, A. Cuesta-Infante, A. Ross, and K. Veeramachaneni. ATM: A distributed,collaborative, scalable system for automated machine learning. In IEEE International Conference on Big Data, 2017.J. Vanschoren, J. N. Van Rijn, B. Bischl, and L. Torgo. OpenML: networked science in machine learning. ACMSIGKDD Explorations Newsletter, 15(2), 2014.6

AlphaD3M: Machine Learning Pipeline Synthesis

https://orbilu.uni.lu/bitstream/10993/57634/1/2111.02508.pdf

AlphaD3M: Machine Learning Pipeline Synthesis

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg