A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algorithms

Abstract syntax tree (AST) mapping algorithms are widely used to analyze changes in source code. Despite the foundational role of AST mapping algorithms, little effort has been made to evaluate the accuracy of AST mapping algorithms, i.e., the extent to which an algorihtm captures the evolution of code. We observe that a program element often has only one best-mapped program element. Based on this observation, we propose a hierarchical approach to automatically compare the similarity of mapped statements and tokens by different algorithms. By performing the comparison, we determine if each of the compared algorithms generates inaccurate mappings for a statement or its tokens. We invite 12 external experts to determine if three commonly used AST mapping algorithms generate accurate mappings for a statement and its tokens for 200 statements. Based on the experts' feedback,we observe that our approach achieves a precision of 0.98--1.00 and a recall of 0.65--0.75. Furthermore, we conduct a large-scale study with a dataset of ten Java projects, containing a total of 263,165 file revisions. Our approach determines that GumTree, MTDiff and IJM generate inaccurate mappings for 20%--29%, 25%--36% and 21%--30% of the file revisions, respectively. Our experimental results show that state-of-art AST mapping agorithms still need improvements

An approach and benchmark to detect behavioral changes of commits in continuous integration

International audienceWhen a developer pushes a change to an application’s codebase, a good practice is to have a test case specifying this behavioral change. Thanks to continuous integration (CI), the test is run on subsequent commits to check that they do no introduce a regression for that behavior. In this paper, we propose an approach that detects behavioral changes in commits. As input, it takes a program, its test suite, and a commit. Its output is a set of test methods that capture the behavioral difference between the pre-commit and postcommit versions of the program. We call our approach DCI (Detecting behavioral changes in CI). It works by generating variations of the existing test cases through (i) assertion amplification and (ii) a search-based exploration of the input space. We evaluate our approach on a curated set of 60 commits from 6 open source Java projects. To our knowledge, this is the first ever curated dataset of real-world behavioral changes. Our evaluation shows that DCI is able to generate test methods that detect behavioral changes. Our approach is fully automated and can be integrated into current development processes. The main limitations are that it targets unit tests and works on a relatively small fraction of commits. More specifically, DCI works on commits that have a unit test that already executes the modified code. In practice, from our benchmark projects, we found 15.29% of commits to meet the conditions required by DCI

Considerations for the interdisciplinary development of environmental system models

Effective decision making and policy development requires holistic consideration of the modelling context. This thesis explores how consideration of multiple disciplinary perspectives and concerns lead to an integrative model development process for the purpose of socio-environmental systems (SES) management. The research is presented through two frames: (1) Integrated Environmental Model (IEM) development through a System-of-Systems (SoS) approach, and (2) the socio-technical considerations within an interdisciplinary modelling process. The presented research incorporates the perspectives of the modelling, systems engineering, and software development paradigms. IEMs are developed for the purpose of integrating knowledge across the various disciplines involved, whereas traditional approaches focus on single systems within the SES, such as hydrology, economics, social dynamics, or climatic drivers. Use of IEMs allows for the consideration of the flow-on effects due to system changes and interaction, and how these may affect long-term SES behaviour. Pathways that are robust - i.e., lead to beneficial or desirable outcomes - under a range of plausible but uncertain conditions can then be identified and assessed. An interconnected network of system models thus makes up an SoS model allowing consideration of higher-order effects. In practice, however, the decisions and approaches taken in developing constituent models may influence integrated system behaviour once coupled. The socio-technical modelling concerns within the SoS/SES modelling context, including the methods to assess and manage model validity, complexity, and uncertainty, with respect to model purpose and intended outcomes are explored through a series of publications. This thesis contributes to the growing body of knowledge through: 1. An expansive overview of the currently available software for model uncertainty and sensitivity analysis, and the techniques they encompass 2. An integrated environmental model for the Lower Campaspe catchment in North-Central Victoria, Australia. The model explores long-term implications of water management decisions and potential policy changes (primarily through an agricultural lens), including conjunctive use of surface and groundwater under a range of uncertain futures. 3. Demonstration of a property-based sensitivity analysis approach to model diagnostics that combines software testing and sensitivity analysis to validate model behaviour. The approach is useful as a first-pass screening tool. Failure to reproduce expected model behaviour indicates issues with the model to be corrected and avoids the necessity of more computationally demanding diagnostics. 4. A pragmatic step-by-step framework for the sensitivity analysis of spatially distributed environmental models 5. Exploration and discussion of the modelling practices, issues and challenges that arise when dealing with the various influences and effects of scale within the interdisciplinary SoS context through a socio-technical lens. The discussion leads to a call for a grander vision for SoS-IEM modelling (and commensurate funding) to better enable interdisciplinary, and integrative, socio-environmental research to occur. 6. A shared reflexive account of two case studies that draws out the considerations and decisions regarding scale to arrive at five shared lessons learnt to foster an effective interdisciplinary modelling process. The key conclusion is the need for researchers involved in SoS modelling of SESs to actively consider and address cross-disciplinary concerns through improved interdisciplinary communication, documentation practices, and explicit consideration of the interplay between defined scales and resulting influence on uncertainty. Integrative consideration of these would then lower or avoid barriers that hamper the development and application of integrated environmental system models