Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation

This article presents Individual Conditional Expectation (ICE) plots, a tool for visualizing the model estimated by any supervised learning algorithm. Classical partial dependence plots (PDPs) help visualize the average partial relationship between the predicted response and one or more features. In the presence of substantial interaction effects, the partial response relationship can be heterogeneous. Thus, an average curve, such as the PDP, can obfuscate the complexity of the modeled relationship. Accordingly, ICE plots refine the partial dependence plot by graphing the functional relationship between the predicted response and the feature for individual observations. Specifically, ICE plots highlight the variation in the fitted values across the range of a covariate, suggesting where and to what extent heterogeneities might exist. In addition to providing a plotting suite for exploratory analysis, we include a visual test for additive structure in the data generating model. Through simulated examples and real data sets, we demonstrate how ICE plots can shed light on estimated models in ways PDPs cannot. Procedures outlined are available in the R package ICEbox.Comment: 22 pages, 14 figures, 2 algorithm

Allocating in the Presence of Dominance: A Mean-Variance Portfolio Choice Economic Experiment

I conduct a mean-variance portfolio choice economic experiment to evaluate how individuals’ portfolio choices deviate from what modern portfolio theory considers optimal. The experimental framework is comprised of three treatments. In each treatment the portfolio selection task involves choosing between two risky assets with zero correlation among their payoffs and one risk free asset. Participants are tasked with completing thirty choice rounds in which they must allocate a constant experimental capital amount to the available asset options after which they are shown period-by-period state-realizations. I utilize the definition of dominance as described in Neugebauer (2004), and Baltussen and Post (2011), that states an asset is dominant if it is attractive in isolation – the asset with the higher Sharpe-ratio. The risky asset, A or B, that is dominant, and the return characteristics of the dominant asset vary over treatments 1, 2, and 3. I find that, relative to theoretically optimal allocation, subjects disproportionately allocate their experimental capital to asset A, the asset with higher expected return and variance, in all treatments, and forgo the benefits to diversification that asset B provides. In order to analyze subjects’ allocation decisions across treatments, I utilize Robust OLS and Fixed Effects regression frameworks

(Psycho-)Analysis of Benchmark Experiments

It is common knowledge that certain characteristics of data sets -- such as linear separability or sample size -- determine the performance of learning algorithms. In this paper we propose a formal framework for investigations on this relationship. The framework combines three, in their respective scientific discipline well-established, methods. Benchmark experiments are the method of choice in machine and statistical learning to compare algorithms with respect to a certain performance measure on particular data sets. To realize the interaction between data sets and algorithms, the data sets are characterized using statistical and information-theoretic measures; a common approach in the field of meta learning to decide which algorithms are suited to particular data sets. Finally, the performance ranking of algorithms on groups of data sets with similar characteristics is determined by means of recursively partitioning Bradley-Terry models, that are commonly used in psychology to study the preferences of human subjects. The result is a tree with splits in data set characteristics which significantly change the performances of the algorithms. The main advantage is the automatic detection of these important characteristics. The framework is introduced using a simple artificial example. Its real-word usage is demonstrated by means of an application example consisting of thirteen well-known data sets and six common learning algorithms. All resources to replicate the examples are available online