454,100 research outputs found
Ethical Issues in Empirical Studies of Software Engineering
The popularity of empirical methods in software engineering research is on the rise. Surveys,
experiments, metrics, case studies, and field studies are examples of empirical methods used to
investigate both software engineering processes and products. The increased application of
empirical methods has also brought about an increase in discussions about adapting these
methods to the peculiarities of software engineering. In contrast, the ethical issues raised by
empirical methods have received little, if any, attention in the software engineering literature. This
article is intended to introduce the ethical issues raised by empirical research to the software
engineering research community, and to stimulate discussion of how best to deal with these ethical
issues. Through a review of the ethical codes of several fields that commonly employ humans and
artifacts as research subjects, we have identified major ethical issues relevant to empirical studies
of software engineering. These issues are illustrated with real empirical studies of software
engineering
Standards of Validity and the Validity of Standards in Behavioral Software Engineering Research: The Perspective of Psychological Test Theory
Background. There are some publications in software engineering research that
aim at guiding researchers in assessing validity threats to their studies.
Still, many researchers fail to address many aspects of validity that are
essential to quantitative research on human factors. Goal. This paper has the
goal of triggering a change of mindset in what types of studies are the most
valuable to the behavioral software engineering field, and also provide more
details of what construct validity is. Method. The approach is based on
psychological test theory and draws upon methods used in psychology in relation
to construct validity. Results. In this paper, I suggest a different approach
to validity threats than what is commonplace in behavioral software engineering
research. Conclusions. While this paper focuses on behavioral software
engineering, I believe other types of software engineering research might also
benefit from an increased focus on construct validity.Comment: ACM/IEEE International Symposium on Empirical Software Engineering
and Measurement (ESEM), Oulu, Finland, October 11-12, 2018. 4 page
A Unified Checklist for Observational and Experimental Research in Software Engineering (Version 1)
Current checklists for empirical software engineering cover either experimental research or case study research but ignore the many commonalities that exist across all kinds of empirical research. Identifying these commonalities, and explaining why they exist, would enhance our understanding of empirical research in general and of the differences between experimental and case study research in particular. In this report we design a unified checklist for empirical research, and identify commonalities and differences between experimental and case study research. We design the unified checklist as a specialization of the general engineering cycle, which itself is a special case of the rational choice cycle. We then compare the resulting empirical research cycle with two checklists for experimental research, and with one checklist for case study research. The resulting checklist identifies important questions to be answered in experimental and case study research design and reports. The checklist provides insights in two different types of empirical research design and their relationships. Its limitations are that it ignores other research methods such as meta-research or surveys. It has been tested so far only in our own research designs and in teaching empirical methods. Future work includes expanding the comparison with other methods and application in more cases, by others than ourselves
Evolution of statistical analysis in empirical software engineering research: Current state and steps forward
Software engineering research is evolving and papers are increasingly based
on empirical data from a multitude of sources, using statistical tests to
determine if and to what degree empirical evidence supports their hypotheses.
To investigate the practices and trends of statistical analysis in empirical
software engineering (ESE), this paper presents a review of a large pool of
papers from top-ranked software engineering journals. First, we manually
reviewed 161 papers and in the second phase of our method, we conducted a more
extensive semi-automatic classification of papers spanning the years 2001--2015
and 5,196 papers. Results from both review steps was used to: i) identify and
analyze the predominant practices in ESE (e.g., using t-test or ANOVA), as well
as relevant trends in usage of specific statistical methods (e.g.,
nonparametric tests and effect size measures) and, ii) develop a conceptual
model for a statistical analysis workflow with suggestions on how to apply
different statistical methods as well as guidelines to avoid pitfalls. Lastly,
we confirm existing claims that current ESE practices lack a standard to report
practical significance of results. We illustrate how practical significance can
be discussed in terms of both the statistical analysis and in the
practitioner's context.Comment: journal submission, 34 pages, 8 figure
Challenges in Survey Research
While being an important and often used research method, survey research has
been less often discussed on a methodological level in empirical software
engineering than other types of research. This chapter compiles a set of
important and challenging issues in survey research based on experiences with
several large-scale international surveys. The chapter covers theory building,
sampling, invitation and follow-up, statistical as well as qualitative analysis
of survey data and the usage of psychometrics in software engineering surveys.Comment: Accepted version of chapter in the upcoming book on Contemporary
Empirical Methods in Software Engineering. Update includes revision of typos
and additional figures. Last update includes fixing two small issues and
typo
Empirical Evidence of Large-Scale Diversity in API Usage of Object-Oriented Software
In this paper, we study how object-oriented classes are used across thousands
of software packages. We concentrate on "usage diversity'", defined as the
different statically observable combinations of methods called on the same
object. We present empirical evidence that there is a significant usage
diversity for many classes. For instance, we observe in our dataset that Java's
String is used in 2460 manners. We discuss the reasons of this observed
diversity and the consequences on software engineering knowledge and research
The Scalability-Efficiency/Maintainability-Portability Trade-off in Simulation Software Engineering: Examples and a Preliminary Systematic Literature Review
Large-scale simulations play a central role in science and the industry.
Several challenges occur when building simulation software, because simulations
require complex software developed in a dynamic construction process. That is
why simulation software engineering (SSE) is emerging lately as a research
focus. The dichotomous trade-off between scalability and efficiency (SE) on the
one hand and maintainability and portability (MP) on the other hand is one of
the core challenges. We report on the SE/MP trade-off in the context of an
ongoing systematic literature review (SLR). After characterizing the issue of
the SE/MP trade-off using two examples from our own research, we (1) review the
33 identified articles that assess the trade-off, (2) summarize the proposed
solutions for the trade-off, and (3) discuss the findings for SSE and future
work. Overall, we see evidence for the SE/MP trade-off and first solution
approaches. However, a strong empirical foundation has yet to be established;
general quantitative metrics and methods supporting software developers in
addressing the trade-off have to be developed. We foresee considerable future
work in SSE across scientific communities.Comment: 9 pages, 2 figures. Accepted for presentation at the Fourth
International Workshop on Software Engineering for High Performance Computing
in Computational Science and Engineering (SEHPCCSE 2016
- …