Metalearning: a survey of trends and technologies

Metalearning attracted considerable interest in the machine learning community in the last years. Yet, some disagreement remains on what does or what does not constitute a metalearning problem and in which contexts the term is used in. This survey aims at giving an all-encompassing overview of the research directions pursued under the umbrella of metalearning, reconciling different definitions given in scientific literature, listing the choices involved when designing a metalearning system and identifying some of the future research challenges in this domain. © 2013 The Author(s)

Une approche par dissimilarité pour la caractérisation de jeux de données

La caractérisation de jeu de données reste un verrou majeur de l'analyse de données intelligente. Une majorité d'approches à ce problème agrègent les informations décrivant les attributs individuels des jeux de données, ce qui représente une perte d'information. Nous proposons une approche par dissimilarité permettant d'éviter cette agrégation, et étudions son intérêt dans la caractérisation des performances d'algorithmes de classifications, et dans la résolution de problèmes de méta-apprentissage

Caractérisation d'instances d'apprentissage pour méta-mining évolutionnaire.

Les nombreuses techniques d'apprentissage et de fouille de données mises au point au cours des dernières décennies peuvent se révéler d'importants atouts dans divers domaines, mais choisir la technique la plus appropriée pour une application précise est une tâche très complexe pour un non-expert. Notre objectif est ainsi de produire un assistant de modélisation répondant à ce besoin, par une approche à la frontière du méta-apprentissage et des heuristiques évolutionnaires. Nous présentons ici le fonctionnement prévu de cet assistant, suivi d'une discussion de notre approche du problème de caractérisation des instances d'apprentissage, qui reste un verrou majeur du méta-apprentissage et méta-mining

On the predictive power of meta-features in OpenML

The demand for performing data analysis is steadily rising. As a consequence, people of different profiles (i.e., non-experienced users) have started to analyze their data. However, this is challenging for them. A key step that poses difficulties and determines the success of the analysis is data mining (model/algorithm selection problem). Meta-learning is a technique used for assisting non-expert users in this step. The effectiveness of meta-learning is, however, largely dependent on the description/characterization of datasets (i.e., meta-features used for meta-learning). There is a need for improving the effectiveness of meta-learning by identifying and designing more predictive meta-features. In this work, we use a method from exploratory factor analysis to study the predictive power of different meta-features collected in OpenML, which is a collaborative machine learning platform that is designed to store and organize meta-data about datasets, data mining algorithms, models and their evaluations. We first use the method to extract latent features, which are abstract concepts that group together meta-features with common characteristics. Then, we study and visualize the relationship of the latent features with three different performance measures of four classification algorithms on hundreds of datasets available in OpenML, and we select the latent features with the highest predictive power. Finally, we use the selected latent features to perform meta-learning and we show that our method improves the meta-learning process. Furthermore, we design an easy to use application for retrieving different meta-data from OpenML as the biggest source of data in this domain.Peer ReviewedPostprint (published version

A Research on Automatic Hyperparameter Recommendation via Meta-Learning

The performance of classification algorithms is mainly governed by the hyperparameter configurations deployed. Traditional search-based algorithms tend to require extensive hyperparameter evaluations to select the desirable configurations during the process, and they are often very inefficient for implementations on large-scale tasks. In this dissertation, we resort to solving the problem of hyperparameter selection via meta-learning which provides a mechanism that automatically recommends the promising ones without any inefficient evaluations. In its approach, a meta-learner is constructed on the metadata extracted from historical classification problems which directly determines the success of recommendations. Designing fine meta-learners to recommend effective hyperparameter configurations efficiently is of practical importance. This dissertation divides into six chapters: the first chapter presents the research background and related work, the second to the fifth chapters detail our main work and contributions, and the sixth chapter concludes the dissertation and pictures our possible future work. In the second and third chapters, we propose two (kernel) multivariate sparse-group Lasso (SGLasso) approaches for automatic meta-feature selection. Previously, meta-features were usually picked by researchers manually based on their preferences and experience or by wrapper method, which is either less effective or time-consuming. SGLasso, as an embedded feature selection model, can select the most effective meta-features during the meta-learner training and thus guarantee the optimality of both meta-features and meta-learner which are essential for successful recommendations. In the fourth chapter, we formulate the problem of hyperparameter recommendation as a problem of low-rank tensor completion. The hyperparameter search space was often stretched to a one-dimensional vector, which removes the spatial structure of the search space and ignores the correlations that existed between the adjacent hyperparameters and these characteristics are crucial in meta-learning. Our contributions are to instantiate the search space of hyperparameters as a multi-dimensional tensor and develop a novel kernel tensor completion algorithm that is applied to estimate the performance of hyperparameter configurations. In the fifth chapter, we propose to learn the latent features of performance space via denoising autoencoders. Although the search space is usually high-dimensional, the performance of hyperparameter configurations is usually correlated to each other to a certain degree and its main structure lies in a much lower-dimensional manifold that describes the performance distribution of the search space. Denoising autoencoders are applied to extract the latent features on which two effective recommendation strategies are built. Extensive experiments are conducted to verify the effectiveness of our proposed approaches, and various empirical outcomes have shown that our approaches can recommend promising hyperparameters for real problems and significantly outperform the state-of-the-art meta-learning-based methods as well as search algorithms such as random search, Bayesian optimization, and Hyperband

Data Mining 4 Dummies: A Web application for automatic selection of data mining algorithms for new problems

O interesse na área de classificação e previsão está a crescer rapidamente na indústria e no comércio e uma série de ferramentas de data mining já estão disponíveis. No entanto essas ferramentas ainda são de utilidade limitada para os utilizadores finais que não sejam especialistas. Isto é devido ao facto de os sistemas de aprendizagem não serem triviais. Como resultado, utilizadores de machine learning/data mining são confrontados com dois desafios: escolher qual o algoritmo mais adequado para usar num determinado conjunto de dados, e combiná-los com transformações úteis e eficazes aos dados. Tradicionalmente, este tipo de problemas é resolvido através de tentativa-e-erro ou consultando especialistas. A primeira solução é demorada e pouco fiável; enquanto que a segunda é dispendiosa e depende das preferências do perito. Claramente, nenhuma das soluções é completamente satisfatória para utilizadores finais não-especialistas. Portanto, é necessária uma orientação automática e sistemática é necessária.Ao analisar o estado da arte, podemos ver como foram desenvolvidas diversas tentativas para abordar este problema e, apesar de algumas dessas tentativas já demonstrarem resultados muito interessantes, as mesmas ainda são dependentes de ferramentas específicas e apresentam alguma falta de orientação, simplicidade e transparência no processo. O foco desta dissertação é trazer uma nova abordagem para os utilizadores finais, através da criação de um novo sistema que permitirá a recomendação e o uso dos modelos/algoritmos mais promissores de uma forma distribuída e colaborativa.The interest in the area of classification and prediction is growing rapidly in industry and commerce. A large number of data mining tools are already available. However, such tools are still of limited use to end-users who are not experts. This is due to the fact that machine learning systems are non-trivial. As a result, users of machine learning/data mining systems are faced with two major problems: selecting the most suitable algorithm to use on a given dataset, and combining this algorithm with useful and effective transformations of the data. Traditionally, these problems are solved by trial-and-error or consulting experts. The first solution is time consuming and unreliable, while the second is expensive and based on preferences of the experts. Clearly, neither solution is completely satisfactory for the non-expert end-users. Therefore automatic and systematic guidance is required.By analysing the state of the art we can see how different attempts have been made to address this problem, and although some of them have shown very interesting results, they are still tool restricted and present a lack of satisfactory user guidance, simplicity and process transparency. The focus of this dissertation is to improve support to machine learning/data mining end-users, by creating a new system that will allow the recommendation and use of the most promising algorithms in a distributed and collaborative way

Metalearning

This open access book as one of the fastest-growing areas of research in machine learning, metalearning studies principled methods to obtain efficient models and solutions by adapting machine learning and data mining processes. This adaptation usually exploits information from past experience on other tasks and the adaptive processes can involve machine learning approaches. As a related area to metalearning and a hot topic currently, automated machine learning (AutoML) is concerned with automating the machine learning processes. Metalearning and AutoML can help AI learn to control the application of different learning methods and acquire new solutions faster without unnecessary interventions from the user. This book offers a comprehensive and thorough introduction to almost all aspects of metalearning and AutoML, covering the basic concepts and architecture, evaluation, datasets, hyperparameter optimization, ensembles and workflows, and also how this knowledge can be used to select, combine, compose, adapt and configure both algorithms and models to yield faster and better solutions to data mining and data science problems. It can thus help developers to develop systems that can improve themselves through experience. This book is a substantial update of the first edition published in 2009. It includes 18 chapters, more than twice as much as the previous version. This enabled the authors to cover the most relevant topics in more depth and incorporate the overview of recent research in the respective area. The book will be of interest to researchers and graduate students in the areas of machine learning, data mining, data science and artificial intelligence. ; Metalearning is the study of principled methods that exploit metaknowledge to obtain efficient models and solutions by adapting machine learning and data mining processes. While the variety of machine learning and data mining techniques now available can, in principle, provide good model solutions, a methodology is still needed to guide the search for the most appropriate model in an efficient way. Metalearning provides one such methodology that allows systems to become more effective through experience. This book discusses several approaches to obtaining knowledge concerning the performance of machine learning and data mining algorithms. It shows how this knowledge can be reused to select, combine, compose and adapt both algorithms and models to yield faster, more effective solutions to data mining problems. It can thus help developers improve their algorithms and also develop learning systems that can improve themselves. The book will be of interest to researchers and graduate students in the areas of machine learning, data mining and artificial intelligence