Learning From Mistakes: Machine Learning Enhanced Human Expert Effort Estimates

In this paper, we introduce a novel approach to predictive modeling for software engineering, named Learning From Mistakes (LFM). The core idea underlying our proposal is to automatically learn from past estimation errors made by human experts, in order to predict the characteristics of their future misestimates, therefore resulting in improved future estimates. We show the feasibility of LFM by investigating whether it is possible to predict the type, severity and magnitude of errors made by human experts when estimating the development effort of software projects, and whether it is possible to use these predictions to enhance future estimations. To this end we conduct a thorough empirical study investigating 402 maintenance and new development industrial software projects. The results of our study reveal that the type, severity and magnitude of errors are all, indeed, predictable. Moreover, we find that by exploiting these predictions, we can obtain significantly better estimates than those provided by random guessing, human experts and traditional machine learners in 31 out of the 36 cases considered (86%), with large and very large effect sizes in the majority of these cases (81%). This empirical evidence opens the door to the development of techniques that use the power of machine learning, coupled with the observation that human errors are predictable, to support engineers in estimation tasks rather than replacing them with machine-provided estimates

Effects of timing on users' perceived control when interacting with intelligent systems

This research relates to the usability of mixed-initiative interaction systems, in which actions can be initiated either through a choice by the user or through intelligent decisions taken by the system. The key issue addressed here is how to preserve the user's perceived control ("sense of agency'') when the control of the interaction is being transferred between the system and the user in a back-and-forth manner. Previous research in social psychology and cognitive neuroscience suggests timing is a factor that can influence perceived control in such back-and-forth interactions. This dissertation explores the hypothesis that in mixed-initiative interaction, a predictable interaction rhythm can preserve the user's sense of control and enhance their experience during a task (e.g. higher confidence in task performance, stronger temporal alignment, lower perceived levels of stress and effort), whereas irregular interaction timing can have the opposite effect. Three controlled experiments compare alternative rhythmic strategies when users interact with simple visual stimuli, simple auditory stimuli, and a more realistic assisted text labelling task. The results of all three experiments support the hypothesis that a predictable interaction rhythm is beneficial in a range of interaction modalities and applications. This research contributes to the field of human-computer interaction (HCI) in four ways. Firstly, it builds novel connections between existing theories in cognitive neuroscience, social psychology and HCI, highlighting how rhythmic temporal structures can be beneficial to the user's experience: particularly, their sense of control. Secondly, it establishes timing as a crucial design resource for mixed-initiative interaction, and provides empirical evidence of how the user's perceived control and other task experiences (such as reported levels of confidence, stress and effort) can be influenced by the manipulation of timing. Thirdly, it provides quantitative measures for the user's entrainment behaviours that are applicable to a wide range of interaction timescales. Lastly, it contextualises the design of timing in a realistic application scenario and offers insights to the design of general end-user automation and decision support tools.Cambridge Commonwealth European and International Trust Scholarship Cambridge Philosophical Society research studentship Gonville & Caius College hardship grant China Scholarship Council (CSC) (The agreement with CSC has been officially cancelled on 28 December 2017.

Guidelines for Statistical Testing

This document provides an introduction to statistical testing. Statistical testing of software is here defined as testing in which the test cases are produced by a random process meant to produce different test cases with the same probabilities with which they would arise in actual use of the software. Statistical testing of software has these main advantages: for the purpose of reliability assessment and product acceptance, it supports directly estimates of reliability, and thus decisions on whether the software is ready for delivery or for use in a specific system. This feature is unique to statistical testing; for the purpose of improving the software, it tends to discover defects which would cause failures with the higher frequencies before those that would cause less frequent failures, thus focusing correction efforts in the most cost-effective way and delivering better software for a given debugging effort. Statistical testing has been reported to achieve dramatic improvements; from the point of view of costs, it facilitates the automation of the test process, thus allowing more testing at acceptable cost than manual testing would allow. This document explains the basic theory underlying statistical testing and provides guidance for its application. The material is organised to facilitate use both as an introduction for software engineers who are new to this approach to testing, and as a reference source during application. Statistical testing is applicable to practically all kinds of software, so this document is not markedly specialised for space applications, though the examples are mostly space-related and the discussion of the software lifecycle is meant to apply to common practice among ESA suppliers

Construction and evaluation of a tool for quantifying uncertainty of software cost estimates

Software development effort estimation is a continuous challenge in the software industry. The inherent uncertainty of effort estimates, which is due to factors such as evolving technology and significant elements of creativity in software development, is an important challenge for software project management. The specific management challenge addressed in this thesis is to assess the uncertainty of effort required for a new software release in the context of incremental software development. The evaluated approach combines task-level estimates with historical data on the estimation accuracy of past tasks for this assessment, by creating effort prediction intervals. The approach was implemented in a web-based tool, and evaluated in the context of a large Norwegian software project with estimation data from three contracted software development companies. In the evaluation we compared the approach to a simpler baseline method, and we found that our suggested approach more consistently produced reasonably accurate prediction intervals. Several variants of the basic approach were investigated. Fitting the historical data to a parametric distribution consistently improved the efficiency of the produced prediction intervals, but the accuracy suffered in cases where the parametric distribution could not reflect the historical distribution of estimation accuracy. Clustering tasks based on size had a positive effect on the produced effort intervals, both in terms of accuracy and efficiency. We believe the suggested approach and tool can be useful in software development project planning and estimation processes providing useful information to support planning, budgeting and resource allocation

Uncertainty in Quantitative Risk Analysis - Characterisation and Methods of Treatment

The fundamental problems related to uncertainty in quantitative risk analyses, used in decision making in safety-related issues (for instance, in land use planning and licensing procedures for hazardous establishments and activities) are presented and discussed, together with the different types of uncertainty that are introduced in the various stages of an analysis. A survey of methods for the practical treatment of uncertainty, with emphasis on the kind of information that is needed for the different methods, and the kind of results they produce, is also presented. Furthermore, a thorough discussion of the arguments for and against each of the methods is given, and of different levels of treatment based on the problem under consideration. Recommendations for future research and standardisation efforts are proposed

A Transparency Index Framework for Machine Learning powered AI in Education

The increase in the use of AI systems in our daily lives, brings calls for more ethical AI development from different sectors including, finance, the judiciary and to an increasing extent education. A number of AI ethics checklists and frameworks have been proposed focusing on different dimensions of ethical AI, such as fairness, explainability and safety. However, the abstract nature of these existing ethical AI guidelines often makes them difficult to operationalise in real-world contexts. The inadequacy of the existing situation with respect to ethical guidance is further complicated by the paucity of work to develop transparent machine learning powered AI systems for real-world. This is particularly true for AI applied in education and training. In this thesis, a Transparency Index Framework is presented as a tool to forefront the importance of transparency and aid the contextualisation of ethical guidance for the education and training sector. The transparency index framework presented here has been developed in three iterative phases. In phase one, an extensive literature review of the real-world AI development pipelines was conducted. In phase two, an AI-powered tool for use in an educational and training setting was developed. The initial version of the Transparency Index Framework was prepared after phase two. And in phase three, a revised version of the Transparency Index Framework was co- designed that integrates learning from phases one and two. The co-design process engaged a range of different AI in education stakeholders, including educators, ed-tech experts and AI practitioners. The Transparency Index Framework presented in this thesis maps the requirements of transparency for different categories of AI in education stakeholders, and shows how transparency considerations can be ingrained throughout the AI development process, from initial data collection to deployment in the world, including continuing iterative improvements. Transparency is shown to enable the implementation of other ethical AI dimensions, such as interpretability, accountability and safety. The 3 optimisation of transparency from the perspective of end-users and ed-tech companies who are developing AI systems is discussed and the importance of conceptualising transparency in developing AI powered ed-tech products is highlighted. In particular, the potential for transparency to bridge the gap between the machine learning and learning science communities is noted. For example, through the use of datasheets, model cards and factsheets adapted and contextualised for education through a range of stakeholder perspectives, including educators, ed-tech experts and AI practitioners