Reliability and validity in comparative studies of software prediction models

Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest that it is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models

Which heuristics can aid financial-decision-making?

© 2015 Elsevier Inc. We evaluate the contribution of Nobel Prize-winner Daniel Kahneman, often in association with his late co-author Amos Tversky, to the development of our understanding of financial decision-making and the evolution of behavioural finance as a school of thought within Finance. Whilst a general evaluation of the work of Kahneman would be a massive task, we constrain ourselves to a more narrow discussion of his vision of financial-decision making compared to a possible alternative advanced by Gerd Gigerenzer along with numerous co-authors. Both Kahneman and Gigerenzer agree on the centrality of heuristics in decision making. However, for Kahneman heuristics often appear as a fall back when the standard von-Neumann-Morgenstern axioms of rational decision-making do not describe investors' choices. In contrast, for Gigerenzer heuristics are simply a more effective way of evaluating choices in the rich and changing decision making environment investors must face. Gigerenzer challenges Kahneman to move beyond substantiating the presence of heuristics towards a more tangible, testable, description of their use and disposal within the ever changing decision-making environment financial agents inhabit. Here we see the emphasis placed by Gigerenzer on how context and cognition interact to form new schemata for fast and frugal reasoning as offering a productive vein of new research. We illustrate how the interaction between cognition and context already characterises much empirical research and it appears the fast and frugal reasoning perspective of Gigerenzer can provide a framework to enhance our understanding of how financial decisions are made

Predicting SMT solver performance for software verification

The approach Why3 takes to interfacing with a wide variety of interactive and automatic theorem provers works well: it is designed to overcome limitations on what can be proved by a system which relies on a single tightly-integrated solver. In common with other systems, however, the degree to which proof obligations (or “goals”) are proved depends as much on the SMT solver as the properties of the goal itself. In this work, we present a method to use syntactic analysis to characterise goals and predict the most appropriate solver via machine-learning techniques. Combining solvers in this way - a portfolio-solving approach - maximises the number of goals which can be proved. The driver-based architecture of Why3 presents a unique opportunity to use a portfolio of SMT solvers for software verification. The intelligent scheduling of solvers minimises the time it takes to prove these goals by avoiding solvers which return Timeout and Unknown responses. We assess the suitability of a number of machinelearning algorithms for this scheduling task. The performance of our tool Where4 is evaluated on a dataset of proof obligations. We compare Where4 to a range of SMT solvers and theoretical scheduling strategies. We find that Where4 can out-perform individual solvers by proving a greater number of goals in a shorter average time. Furthermore, Where4 can integrate into a Why3 user’s normal workflow - simplifying and automating the non-expert use of SMT solvers for software verification

Predicting SMT solver performance for software verification

The approach Why3 takes to interfacing with a wide variety of interactive and automatic theorem provers works well: it is designed to overcome limitations on what can be proved by a system which relies on a single tightly-integrated solver. In common with other systems, however, the degree to which proof obligations (or “goals”) are proved depends as much on the SMT solver as the properties of the goal itself. In this work, we present a method to use syntactic analysis to characterise goals and predict the most appropriate solver via machine-learning techniques. Combining solvers in this way - a portfolio-solving approach - maximises the number of goals which can be proved. The driver-based architecture of Why3 presents a unique opportunity to use a portfolio of SMT solvers for software verification. The intelligent scheduling of solvers minimises the time it takes to prove these goals by avoiding solvers which return Timeout and Unknown responses. We assess the suitability of a number of machinelearning algorithms for this scheduling task. The performance of our tool Where4 is evaluated on a dataset of proof obligations. We compare Where4 to a range of SMT solvers and theoretical scheduling strategies. We find that Where4 can out-perform individual solvers by proving a greater number of goals in a shorter average time. Furthermore, Where4 can integrate into a Why3 user’s normal workflow - simplifying and automating the non-expert use of SMT solvers for software verification