OBOE: Collaborative Filtering for AutoML Model Selection

Algorithm selection and hyperparameter tuning remain two of the most challenging tasks in machine learning. Automated machine learning (AutoML) seeks to automate these tasks to enable widespread use of machine learning by non-experts. This paper introduces OBOE, a collaborative filtering method for time-constrained model selection and hyperparameter tuning. OBOE forms a matrix of the cross-validated errors of a large number of supervised learning models (algorithms together with hyperparameters) on a large number of datasets, and fits a low rank model to learn the low-dimensional feature vectors for the models and datasets that best predict the cross-validated errors. To find promising models for a new dataset, OBOE runs a set of fast but informative algorithms on the new dataset and uses their cross-validated errors to infer the feature vector for the new dataset. OBOE can find good models under constraints on the number of models fit or the total time budget. To this end, this paper develops a new heuristic for active learning in time-constrained matrix completion based on optimal experiment design. Our experiments demonstrate that OBOE delivers state-of-the-art performance faster than competing approaches on a test bed of supervised learning problems. Moreover, the success of the bilinear model used by OBOE suggests that AutoML may be simpler than was previously understood

A Bayesian Approach toward Active Learning for Collaborative Filtering

Collaborative filtering is a useful technique for exploiting the preference patterns of a group of users to predict the utility of items for the active user. In general, the performance of collaborative filtering depends on the number of rated examples given by the active user. The more the number of rated examples given by the active user, the more accurate the predicted ratings will be. Active learning provides an effective way to acquire the most informative rated examples from active users. Previous work on active learning for collaborative filtering only considers the expected loss function based on the estimated model, which can be misleading when the estimated model is inaccurate. This paper takes one step further by taking into account of the posterior distribution of the estimated model, which results in more robust active learning algorithm. Empirical studies with datasets of movie ratings show that when the number of ratings from the active user is restricted to be small, active learning methods only based on the estimated model don't perform well while the active learning method using the model distribution achieves substantially better performance.Comment: Appears in Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence (UAI2004

Ensembles of choice-based models for recommender systems

In this thesis, we focused on three main paradigms: Recommender Systems, Decision Making, and Ensembles. The work is structured as follows. First, the thesis analyzes the potential of choice-based models. The motivation behind this was based on the idea of applying sound decisionmaking paradigms, such as choice and utility theory, in the field of Recommender Systems. Second, this research analyzes the cognitive process underlying choice behavior. On the one hand, neural and gaze activity were recorded experimentally from different subjects performing a choice task in a Web Interface. On the other hand, cognitive were fitted using rational, emotional, and attentional features. Finally, the work explores the hybridization of choice-based models with ensembles. The goal is to take the best of the two worlds: transparency and performance. Two main methods were analyzed to build optimal choice-based ensembles: uninformed and informed. First one, two strategies were evaluated: 1-Learner and N-Learners ensembles. Second one, we relied on three types of prior information: (1) High diversity, (2) Low error prediction (MSE), (3) and Low crowd error

Hybrid Recommender Systems for Next Purchase Prediction Based on Optimal Combination Weights

Recommender systems (RS) play a key role in e-commerce by preselecting presumably interesting products for customers. Hybrid RSs using a weighted average of individual RSs’ predictions have been widely adopted for improving accuracy and robustness over individual RSs. While for regression tasks, approaches to estimate optimal weighting schemes based on individual RSs’ out-of-sample errors exist, there is scant literature in classification settings. Class prediction is important for RSs in e-commerce, as here item purchases are to be predicted. We propose a method for estimating weighting schemes to combine classifying RSs based on the variance-covariance structures of the errors of individual models' probability scores. We evaluate the approach on a large real-world ecommerce data set from a European telecommunications provider, where it shows superior accuracy compared to the best individual model as well as a weighting scheme that averages the predictions using equal weights