A Unified Checklist for Observational and Experimental Research in Software Engineering (Version 1)

Current checklists for empirical software engineering cover either experimental research or case study research but ignore the many commonalities that exist across all kinds of empirical research. Identifying these commonalities, and explaining why they exist, would enhance our understanding of empirical research in general and of the differences between experimental and case study research in particular. In this report we design a unified checklist for empirical research, and identify commonalities and differences between experimental and case study research. We design the unified checklist as a specialization of the general engineering cycle, which itself is a special case of the rational choice cycle. We then compare the resulting empirical research cycle with two checklists for experimental research, and with one checklist for case study research. The resulting checklist identifies important questions to be answered in experimental and case study research design and reports. The checklist provides insights in two different types of empirical research design and their relationships. Its limitations are that it ignores other research methods such as meta-research or surveys. It has been tested so far only in our own research designs and in teaching empirical methods. Future work includes expanding the comparison with other methods and application in more cases, by others than ourselves

The Scalability-Efficiency/Maintainability-Portability Trade-off in Simulation Software Engineering: Examples and a Preliminary Systematic Literature Review

Large-scale simulations play a central role in science and the industry. Several challenges occur when building simulation software, because simulations require complex software developed in a dynamic construction process. That is why simulation software engineering (SSE) is emerging lately as a research focus. The dichotomous trade-off between scalability and efficiency (SE) on the one hand and maintainability and portability (MP) on the other hand is one of the core challenges. We report on the SE/MP trade-off in the context of an ongoing systematic literature review (SLR). After characterizing the issue of the SE/MP trade-off using two examples from our own research, we (1) review the 33 identified articles that assess the trade-off, (2) summarize the proposed solutions for the trade-off, and (3) discuss the findings for SSE and future work. Overall, we see evidence for the SE/MP trade-off and first solution approaches. However, a strong empirical foundation has yet to be established; general quantitative metrics and methods supporting software developers in addressing the trade-off have to be developed. We foresee considerable future work in SSE across scientific communities.Comment: 9 pages, 2 figures. Accepted for presentation at the Fourth International Workshop on Software Engineering for High Performance Computing in Computational Science and Engineering (SEHPCCSE 2016

AN EMERGING THEORY ON THE INTERACTION BETWEEN REQUIREMENTS ENGINEERING AND SYSTEMS ARCHITECTING BASED ON A SUITE OF EXPLORATORY EMPIRICAL STUDIES

Requirements Engineering and Systems Architecting are often considered the most important phases of the software development lifecycle. Because of their close proximity in the software development lifecycle, there is a high degree of interaction between these two processes. While such interaction has been recognized and researched in terms of new technology (particularly methods and tools), there is a distinct lack of empirical understanding regarding the scientific properties of this interaction. Furthermore, in Requirements Engineering and Systems Architecting, not only technical but human aspects are considered critical for the success of these processes due to these processes lying at the front-end of the development cycle and therefore being more aligned with real-world issues. Thus, the scientific properties of the interactions between Requirements Engineering and Systems Architecting can be broken down into these two key aspects. For instance, the following example research questions relate to such scientific properties: What is the impact of an existing system’s architecture on requirements decision-making? What kinds of requirements-oriented problems are encountered during architecting? What is the impact of an existing systems architecture on new requirements being elicited? What is the impact of requirements engineering knowledge on systems architecting? There is little in the literature addressing such questions. This thesis explores such issues through a suite of six exploratory empirical studies that were conducted over the last five years. Based on the observations from these studies, an emerging theory is proposed that describes the impact of human and process factors in the interaction between Requirements Engineering and Systems Architecting. The impact of this emerging body of knowledge is deemed to be on the following: technology development for Requirements Engineering and Software Architecting (methods, tools, processes, etc.); hiring and training personnel for Requirements Engineering and Systems Architecture processes in industry; Requirements Engineering and Systems Architecture project planning; curriculum improvement in academia; and future empirical research in Requirements Engineering and Systems Architecting

Variability Management in an unaware software product line company: An experience report

Software product line adoption is a challenging task in software development organisations. There are some reports in the literature of how software product line engineering has been adopted in several companies using di erent variabil-ity management techniques and patterns. However, to the best of our knowledge, there are no empirical reports on how variability management is handled in companies that do not know about software product line methods and tools. In this paper we present an experience report observing variability management practices in a software development company that was unaware of software product line approaches. We brie y report how variability management is performed in di erent areas ranging from business architecture to software assets management. From the observation we report some open research opportunities for the future and foster further similar and more structured empirical studies on unaware software product line companies.Ministerio de Economía y Competitividad TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía P12-TIC-186

Error Detection and Recovery in Software Development

Software rarely works as intended when it is first written. Software engineering research has long been concerned with assessing why software fails and who is to blame, or why a piece of software is flawed and how to prevent such faults in the future. Errors are examined in the context of bugs, elements of source code that produce undesirable, unexpected and unintended deviations in behaviour. Though error is a prevalent, mature topic within software engineering, error detection and recovery are less well understood. This research uses rich qualitative methods to study error detection and recovery in professional software development practice. It has considered conceptual representations of error in software engineering research and trade literature. Using ethnographic principles, it has gathered accounts given by professional developers in interviews and in video-recorded paired interaction. Developers performing a range of tasks were observed, and findings were compared to theories of human error formed in psychology and safety science. Three empirical studies investigated error from the perspective of developers, recon- structing the view they hold when errors arise, to build a catalogue of active encounters with error in conceptual design, at the desk and after the fact. Analyses were structured to consider development holistically over time, rather than in terms of discrete tasks. By placing emphasis on “local rationality”, analytical focus was redirected from outcomes toward factors that influence performance. The resultant observations are assembled in an account of error handling in software development as personal and situated (in time and the developer’s environment), with implications for the changing nature of expertise

Exploring the Software Verification and Validation Process with Focus on Efficient Fault Detection

Quality is an aspect of high importance in software development projects. The software organizations have to ensure that the quality of their developed products is what the customers expect. Thus, the organizations have to verify that the product is functioning as expected and validate that the product is what the customers expect. Empirical studies have shown that in many software development projects as much as half of the projected schedule is spent on the verification and validation activities. The research in this thesis focuses on exploring the state of practice of the verification and validation process and investigating methods for achieving efficient fault detection during the software development. The thesis aims at increasing the understanding of the activities conducted to verify and validate the software products, by the means of empirical research in the software engineering domain. A survey of eleven Swedish software development organizations investigates the current state of practice of the verification and validation activities, and how these activities are managed today. The need for communicating and visualising the verification and validation process was expressed during the survey. Therefore the usefulness of process simulations was evaluated in the thesis. The simulations increased the understanding of the relationships between different activities among the involved participants. In addition, an experiment was conducted to compare the performance of the two verification and validation activities, inspection and testing. In the future work, empirical research, including experiment results, will be used for calibration and validation of simulation models, with focus on using simulation as a method for decision support in the verification and validation process

Programming language trends : an empirical study

Predicting the evolution of software engineering technology trends is a dubious proposition. The recent evolution of software technology is a prime example; it is fast paced and affected by many factors, which are themselves driven by a wide range of sources. This dissertation is part of a long term project intended to analyze software engineering technology trends and how they evolve. Basically, the following questions will be answered: How to watch, predict, adapt to, and affect software engineering trends? In this dissertation, one field of software engineering, programming languages, will be discussed. After reviewing the history of a group of programming languages, it shows that two kinds of factors, intrinsic factors and extrinsic factors, could affect the evolution of a programming language. Intrinsic factors are the factors that can be used to describe the general desigu criteria of programming languages. Extrinsic factors are the factors that are not directly related to the general attributes of programming languages, but still can affect their evolution. In order to describe the relationship of these factors and how they affect programming language trends, these factors need to be quantified. A score has been assigued to each factor for every programming language. By collecting historical data, a data warehouse has been established, which stores the value of each factor for every programming language. The programming language trends are described and evaluated by using these data. Empirical research attempts to capture observed behaviors by empirical laws. In this dissertation, statistical methods are used to describe historical programming language trends and predict the evolution of the future trends. Several statistics models are constructed to describe the relationships among these factors. Canonical correlation is used to do the factor analysis. Multivariate multiple regression method has been used to construct the statistics models for programming language trends. After statistics models are constructed to describe the historical programming language trends, they are extended to do tentative prediction for future trends. The models are validated by comparing the predictive data and the actual data

Requirements Engineering that Balances Agility of Teams and System-level Information Needs at Scale

Context: Motivated by their success in software development, large-scale systems development companies are increasingly adopting agile methods and their practices. Such companies need to accommodate different development cycles of hardware and software and are usually subject to regulation and safety concerns. Also, for such companies, requirements engineering is an essential activity that involves upfront and detailed analysis which can be at odds with agile development methods. Objective: The overall aim of this thesis is to investigate the challenges and solution candidates of performing effective requirements engineering in an agile environment, based on empirical evidence. Illustrated with studies on safety and system-level information needs, we explore RE challenges and solutions in large-scale agile development, both in general and from the teams’ perspectives. Method: To meet our aim, we performed a secondary study and a series of empirical studies based on case studies. We collected qualitative data using interviews, focus groups and workshops to derive challenges and potential solutions from industry. Findings: Our findings show that there are numerous challenges of conducting requirements engineering in agile development especially where systems development is concerned. The challenges discovered sprout from an integration problem of working with agile methods while relying on established plan-driven processes for the overall system. We highlight the communication challenge of crossing the boundary of agile methods and system-level (or plan-driven) development, which also proves the coexistence of both methods. Conclusions: Our results highlight the painful areas of requirements engineering in agile development and propose solutions that can be explored further. This thesis contributes to future research, by establishing a holistic map of challenges and candidate solutions that can be further developed to make RE more efficient within agile environments