Are Delayed Issues Harder to Resolve? Revisiting Cost-to-Fix of Defects throughout the Lifecycle

Many practitioners and academics believe in a delayed issue effect (DIE); i.e. the longer an issue lingers in the system, the more effort it requires to resolve. This belief is often used to justify major investments in new development processes that promise to retire more issues sooner. This paper tests for the delayed issue effect in 171 software projects conducted around the world in the period from 2006--2014. To the best of our knowledge, this is the largest study yet published on this effect. We found no evidence for the delayed issue effect; i.e. the effort to resolve issues in a later phase was not consistently or substantially greater than when issues were resolved soon after their introduction. This paper documents the above study and explores reasons for this mismatch between this common rule of thumb and empirical data. In summary, DIE is not some constant across all projects. Rather, DIE might be an historical relic that occurs intermittently only in certain kinds of projects. This is a significant result since it predicts that new development processes that promise to faster retire more issues will not have a guaranteed return on investment (depending on the context where applied), and that a long-held truth in software engineering should not be considered a global truism.Comment: 31 pages. Accepted with minor revisions to Journal of Empirical Software Engineering. Keywords: software economics, phase delay, cost to fi

Exploring Software Testing Strategies Used on Software Applications in the Government

Developing a defect-free software application is a challenging task. Despite many years of experience, the intense development of reliable software remains a challenge. For this reason, software defects identified at the end of the testing phase are more expensive than those detected sooner. The purpose of this multiple case study is to explore the testing strategies software developers use to ensure the reliability of software applications in the government contracting industry. The target population consisted of software developers from 3 government contracting organizations located along the East Coast region of the United States. Lehman’s laws of software evolution was the conceptual framework. The data collection process included semistructured interviews with software developers (n = 10), including a review of organizational documents (n = 77). Thematic analysis was used to identify patterns and codes from the interviews. Member checking activities were triangulated with organizational documents to produce 4 major themes: (a) communication and collaboration with all stakeholders, (b) development of well-defined requirements, (c) focus on thorough documentation, and (d) focus on automation testing. The results of this study may contribute to information about testing strategies that may help organizations improve or enhance their testing practices. The results of this study may serve as a foundation for positive social change by potentially improving citizens’ experience with government software applications as a result of potential improvement in software testing practice

Evaluating the Likelihood of Bug Inducing Commits Using Metrics Trend Analysis

Continuous software engineering principles advocate a release-small, release-often process model, where new functionality is added to a system, in small increments and very frequently. In such a process model, every time a change is introduced it is important to identify as early as possible, whether the system has entered a state where faults are more likely to occur. In this paper, we present a method that is based on process, quality, and source code metrics to evaluate the likelihood that an imminent bug-inducing commit is highly probable. More specifically, the method analyzes the correlations and the rate of change of selected metrics. The findings from the technical debt dataset extracted data from SonarQube indicate that before bug-inducing commits, metrics that otherwise are not correlated, suddenly exhibit a high correlation or a high rate of change. This metric behavior can then be used for assessing an impending bug-inducing commit. The technique is programing language agnostic, based on metrics that can be extracted without the use of specialized parsers and can be applied to forewarn developers that a file, or a collection of files, has entered a state where faults are highly probable

On the Challenges of Implementing Machine Learning Systems in Industry

RÉSUMÉ : Dans l’optique de ce mémoire, nous nous concentrons sur les défis de l’implantation de sys-tèmes d’apprentissage automatique dans le contexte de l’industrie. Notre travail est réparti sur deux volets: dans un premier temps, nous explorons des considérations fondamentales sur le processus d’ingénierie de systèmes d’apprentissage automatique et dans un second temps, nous explorons l’aspect pratique de l’ingénierie de tels systèmes dans un cadre industriel. Pour le premier volet, nous explorons un des défis récemment mis en évidence par la com-munauté scientifique: la reproducibilité. Nous expliquons les défis qui s’y rattachent et, à la lueur de cette nouvelle compréhension, nous explorons un des effets rattachés, omniprésent dans l’ingénierie logicielle: la présence de défaut logiciels. À l’aide d’une méthodologie rigoureuse nous cherchons à savoir si la présence de défauts logiciels, parmis un échantillon de taille fixe, dans un cadriciel d’apprentissage automatique impacte le résultat d’un processus d’apprentissage.----------ABSTRACT : Software engineering projects face a number of challenges, ranging from managing their life-cycle to ensuring proper testing methodologies, dealing with defects, building, deploying, among others. As machine learning is becoming more prominent, introducing machine learn-ing in new environments requires skills and considerations from software engineering, machine learning and computer engineering, while also sharing their challenges from these disciplines. As democratization of machine learning has increased by the presence of open-source projects led by both academia and industry, industry practitioners and researchers share one thing in common: the tools they use. In machine learning, tools are represented by libraries and frameworks used as software for the various steps necessary in a machine learning project. In this work, we investigate the challenges in implementing machine learning systems in the industry

System Qualities Ontology, Tradespace and Affordability (SQOTA) Project – Phase 4

This task was proposed and established as a result of a pair of 2012 workshops sponsored by the DoD Engineered Resilient Systems technology priority area and by the SERC. The workshops focused on how best to strengthen DoD’s capabilities in dealing with its systems’ non-functional requirements, often also called system qualities, properties, levels of service, and –ilities. The term –ilities was often used during the workshops, and became the title of the resulting SERC research task: “ilities Tradespace and Affordability Project (iTAP).” As the project progressed, the term “ilities” often became a source of confusion, as in “Do your results include considerations of safety, security, resilience, etc., which don’t have “ility” in their names?” Also, as our ontology, methods, processes, and tools became of interest across the DoD and across international and standards communities, we found that the term “System Qualities” was most often used. As a result, we are changing the name of the project to “System Qualities Ontology, Tradespace, and Affordability (SQOTA).” Some of this year’s university reports still refer to the project as “iTAP.”This material is based upon work supported, in whole or in part, by the U.S. Department of Defense through the Office of the Assistant of Defense for Research and Engineering (ASD(R&E)) under Contract HQ0034-13-D-0004.This material is based upon work supported, in whole or in part, by the U.S. Department of Defense through the Office of the Assistant of Defense for Research and Engineering (ASD(R&E)) under Contract HQ0034-13-D-0004

A strategic approach to failure mitigation: A study of project and quality management in five projects

The causes of operational failure remain unclear to those tasked with both delivering projects and managing operational assets. Greater awareness of the owner and their supply network capabilities to mitigate failure could reduce significant quality costs that can amount to many millions of pounds. This thesis investigates why assets handed over to the owner have failed during operation, and proposes new ways that capabilities can be integrated to reduce and prevent potential operational failure from arising. An abductive reasoning with a grounded theory approach was used over a three-year period, and involved quarterly expert research steering group meetings to validate the iteration between literature and empirical observation to obtain new insights. The first workshop and questionnaire phase of the study created a Cost of Quality (COQ) framework; this was then tested on five multi-case study and subsequently developed within a single expert owner organisation using semi-structured interviews, card sorting and a Delphi review. The results show that the owner and the multi-organisational supply network capabilities are fragmented in addressing operational failure. By identifying and measuring quality cost failure, owners and their supply network will learn and be able to procure more integrated capabilities in failure mitigation for reducing quality cost failure. This will be achieved with better understanding of the relationship between owner’s strategic requirement, technical project delivery and functional operations management capabilities, which is summarised in a capabilities cycle model. The model illustrates the need for a strategic project and quality management approach to integrate capabilities within each phase of a project’s lifecycle. An integrated capabilities approach is proposed for the owner and its multi-organisational supply and operator network to integrate and collaborate in relation to the capabilities required to equally share project risk and quality cost in mitigating the failures

Information systems failure: A business-led knowledge requirements framework for modelling business requirements

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Our work will be mainly concerned with improving the crucial first stage (the requirements stage) of any system development methodology in order to improve requirements. A framework has been developed, called "knowledge requirements framework (KRF)" to help customers and system developers bridge the knowledge and understanding gaps at the initial requirements stage of the Information Technology System (ITS) development process. Unclear business requirements, mismatch of knowledge and understanding are among the major factors that contributes to some ITS failures worldwide. The aim is to capture functional requirements at the initial stage of the system development process and to integrate systems and people use them in the development process. Multi-surveys are conducted, capture and highlight the criteria of initial requirements exactness and executability. Knowledge and understanding gaps, which occur in the development process, are described. These gaps constitute the problem at the invisible architecture in the initial requirements stage, as they expose mismatch of both knowledge and understanding problems (Requirements/Specifications). A notation to describe this framework is elaborated, novel techniques and tools for the construction and application of customer requirements in systems development are developed and used in KRF to facilitate bridging these gaps. The resulting prototype KRF is developed and used against some example problems in retail organisations, and so shown to be sufficient in principle of handling all the negotiation problems at the initial requirements stage, singly and in combination. Also, it is shown how KRF sub-process can be combined and used to elicit information and knowledge mining between both the customer and the system developer using human communication and interaction capture as an example. Systems these days are living systems, changeable, in business and the human factor in developing them cannot be excluded. It is further shown how these techniques and tools can be augmented with established methodologies rather than inventing new ones and to enable management to react as quickly as possible to global changing market conditions. This proposed framework is also evaluated and tested against the original criteria of initial requirements, exactness and executability