Evaluating Maintainability Prejudices with a Large-Scale Study of Open-Source Projects

Exaggeration or context changes can render maintainability experience into prejudice. For example, JavaScript is often seen as least elegant language and hence of lowest maintainability. Such prejudice should not guide decisions without prior empirical validation. We formulated 10 hypotheses about maintainability based on prejudices and test them in a large set of open-source projects (6,897 GitHub repositories, 402 million lines, 5 programming languages). We operationalize maintainability with five static analysis metrics. We found that JavaScript code is not worse than other code, Java code shows higher maintainability than C# code and C code has longer methods than other code. The quality of interface documentation is better in Java code than in other code. Code developed by teams is not of higher and large code bases not of lower maintainability. Projects with high maintainability are not more popular or more often forked. Overall, most hypotheses are not supported by open-source data.Comment: 20 page

A methodology for validating cloud models using metamorphic testing

Cloud computing is a paradigm that provides access to a flexible, elastic and on-demand computing infrastructure, allowing users to dynamically request virtual resources. However, researchers typically cannot experiment with critical parts of cloud systems such as the underlying cloud architecture, resource-provisioning policies and the configuration of resource virtualisation. This problem can be partially addressed through using simulations of cloud systems. Unfortunately, the problem of testing cloud systems is still challenging due to the many parameters that such systems typically have and the difficulty in determining whether an observed behaviour is correct. In order to alleviate these issues, we propose a methodology to semi-automatically test and validate cloud models by integrating simulation techniques and metamorphic testing.This research was partially supported by the Spanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTuDIo (TIN2012-36812-C02-01)

Refinement type contracts for verification of scientific investigative software

Our scientific knowledge is increasingly built on software output. User code which defines data analysis pipelines and computational models is essential for research in the natural and social sciences, but little is known about how to ensure its correctness. The structure of this code and the development process used to build it limit the utility of traditional testing methodology. Formal methods for software verification have seen great success in ensuring code correctness but generally require more specialized training, development time, and funding than is available in the natural and social sciences. Here, we present a Python library which uses lightweight formal methods to provide correctness guarantees without the need for specialized knowledge or substantial time investment. Our package provides runtime verification of function entry and exit condition contracts using refinement types. It allows checking hyperproperties within contracts and offers automated test case generation to supplement online checking. We co-developed our tool with a medium-sized ($\approx$3000 LOC) software package which simulates decision-making in cognitive neuroscience. In addition to helping us locate trivial bugs earlier on in the development cycle, our tool was able to locate four bugs which may have been difficult to find using traditional testing methods. It was also able to find bugs in user code which did not contain contracts or refinement type annotations. This demonstrates how formal methods can be used to verify the correctness of scientific software which is difficult to test with mainstream approaches

A partial oracle for uniformity statistics

This paper investigates the problem of testing implementations of uniformity statistics. In this paper we used Metamorphic Testing to address the oracle problem, of checking the output of one or more test executions, for uniformity statistics. We defined a partial oracle that uses regression analysis (a Regression Model based Metamorphic Relation). We investigated the effectiveness of our partial oracle. We found that the technique can achieve mutation scores ranging from 77.78% to 100%, and tends towards higher mutation scores in this range. These results are promising, and suggest that the Regression Model based Metamorphic Relation approach is a viable method of alleviating the oracle problem in implementations of uniformity statistics, and potentially other classes of statistics e.g. correlation statistics

Data mining for software engineering and humans in the loop

The field of data mining for software engineering has been growing over the last decade. This field is concerned with the use of data mining to provide useful insights into how to improve software engineering processes and software itself, supporting decision-making. For that, data produced by software engineering processes and products during and after software development are used. Despite promising results, there is frequently a lack of discussion on the role of software engineering practitioners amidst the data mining approaches. This makes adoption of data mining by software engineering practitioners difficult. Moreover, the fact that experts’ knowledge is frequently ignored by data mining approaches, together with the lack of transparency of such approaches, can hinder the acceptability of data mining by software engineering practitioners. To overcome these problems, this position paper provides a discussion of the role of software engineering experts when adopting data mining approaches. It also argues that this role can be extended to increase experts’ involvement in the process of building data mining models. We believe that such extended involvement is not only likely to increase software engineers’ acceptability of the resulting models, but also improve the models themselves. We also provide some recommendations aimed at increasing the success of experts involvement and model acceptability

Syntactic Complexity Metrics and the Readability of Programs in a Functional Computer Language

This article reports on the defintion and the measutement of the software complexity metrics of Halstead and McCabe for programs written in the functional programming language Miranda. An automated measurement of these metrics is described. In a case study, the correlation is established between the complexity metrics and the expert assessment of the readability of programs in Miranda, and compared with those for programs in Pascal