Data sets and data quality in software engineering

OBJECTIVE - to assess the extent and types of techniques used to manage quality within software engineering data sets. We consider this a particularly interesting question in the context of initiatives to promote sharing and secondary analysis of data sets. METHOD - we perform a systematic review of available empirical software engineering studies. RESULTS - only 23 out of the many hundreds of studies assessed, explicitly considered data quality. CONCLUSIONS - first, the community needs to consider the quality and appropriateness of the data set being utilised; not all data sets are equal. Second, we need more research into means of identifying, and ideally repairing, noisy cases. Third, it should become routine to use sensitivity analysis to assess conclusion stability with respect to the assumptions that must be made concerning noise levels

On the Inducibility of Stackelberg Equilibrium for Security Games

Strong Stackelberg equilibrium (SSE) is the standard solution concept of Stackelberg security games. As opposed to the weak Stackelberg equilibrium (WSE), the SSE assumes that the follower breaks ties in favor of the leader and this is widely acknowledged and justified by the assertion that the defender can often induce the attacker to choose a preferred action by making an infinitesimal adjustment to her strategy. Unfortunately, in security games with resource assignment constraints, the assertion might not be valid; it is possible that the defender cannot induce the desired outcome. As a result, many results claimed in the literature may be overly optimistic. To remedy, we first formally define the utility guarantee of a defender strategy and provide examples to show that the utility of SSE can be higher than its utility guarantee. Second, inspired by the analysis of leader's payoff by Von Stengel and Zamir (2004), we provide the solution concept called the inducible Stackelberg equilibrium (ISE), which owns the highest utility guarantee and always exists. Third, we show the conditions when ISE coincides with SSE and the fact that in general case, SSE can be extremely worse with respect to utility guarantee. Moreover, introducing the ISE does not invalidate existing algorithmic results as the problem of computing an ISE polynomially reduces to that of computing an SSE. We also provide an algorithmic implementation for computing ISE, with which our experiments unveil the empirical advantage of the ISE over the SSE.Comment: The Thirty-Third AAAI Conference on Artificial Intelligenc

Shifted Science Revisited: Percolation Delays and the Persistence of Wrongful Convictions Based on Outdated Science

We previously wrote about the phenomenon of convictions based on science that is credible at the time of trial, but later comes to be repudiated. Such post-conviction shifts in science were most obvious and reprehensible in the very old cases, the example being a 1986 arson prosecution, whose scientific underpinnings are exposed in a post-conviction motion filed in 2011. Immediately upon completing that article, we came to realize that it told only half the story. We seek in this Article to build on that foundational idea of shifted science by discussing at length a harder question: the perception, percolation, and continued evolution of shifts in science. We address here cases that arise on the cusp of a shift, identify the process of the shift in various forensic science disciplines, and analyze how difficult it can be to perceive and address a shift in science, even when it occurs concurrently with, or even some time prior to, trial. Taking a step-by-step route through the process of significant shifts in several different forensic disciplines, we hope to clarify the many stages involved in these shifts and the important consequences of misperceiving shifts in science as they occur. Finally, we also lay a foundation for a later piece addressing the difficult question of legal avenues for relief in shifted science cases that arise on the cusp of a revolution, such as those we address here

Syntactic strategies of exclamatives

The study presented in this paper has two aims. First, it establishes pragmasemantic features of exclamations and exclamatives relying on three formulated approaches – a constructional approach, a presupposition approach, and a scalarity approach, and suggests distinguishing proper exclamatives, the syntactic structures of which are conventionally associated with an illocutionary force of expressivity, from improper ones that do not have such an association. Second, involving the data of 45 languages, the paper reveals and describes 5 syntactic strategies of exclamatives, which are as follows: subject-verb inversion, subordinate clauses, noun phrases, anaphoric adverbs and adjectives, and wh-phrases. The latter three are further divided into several sub-strategies

Learning Interpretable Rules for Multi-label Classification

Multi-label classification (MLC) is a supervised learning problem in which, contrary to standard multiclass classification, an instance can be associated with several class labels simultaneously. In this chapter, we advocate a rule-based approach to multi-label classification. Rule learning algorithms are often employed when one is not only interested in accurate predictions, but also requires an interpretable theory that can be understood, analyzed, and qualitatively evaluated by domain experts. Ideally, by revealing patterns and regularities contained in the data, a rule-based theory yields new insights in the application domain. Recently, several authors have started to investigate how rule-based models can be used for modeling multi-label data. Discussing this task in detail, we highlight some of the problems that make rule learning considerably more challenging for MLC than for conventional classification. While mainly focusing on our own previous work, we also provide a short overview of related work in this area.Comment: Preprint version. To appear in: Explainable and Interpretable Models in Computer Vision and Machine Learning. The Springer Series on Challenges in Machine Learning. Springer (2018). See http://www.ke.tu-darmstadt.de/bibtex/publications/show/3077 for further informatio