Ethical Issues in Empirical Studies of Software Engineering

The popularity of empirical methods in software engineering research is on the rise. Surveys, experiments, metrics, case studies, and field studies are examples of empirical methods used to investigate both software engineering processes and products. The increased application of empirical methods has also brought about an increase in discussions about adapting these methods to the peculiarities of software engineering. In contrast, the ethical issues raised by empirical methods have received little, if any, attention in the software engineering literature. This article is intended to introduce the ethical issues raised by empirical research to the software engineering research community, and to stimulate discussion of how best to deal with these ethical issues. Through a review of the ethical codes of several fields that commonly employ humans and artifacts as research subjects, we have identified major ethical issues relevant to empirical studies of software engineering. These issues are illustrated with real empirical studies of software engineering

The consistency of empirical comparisons of regression and analogy-based software project cost prediction

OBJECTIVE - to determine the consistency within and between results in empirical studies of software engineering cost estimation. We focus on regression and analogy techniques as these are commonly used. METHOD – we conducted an exhaustive search using predefined inclusion and exclusion criteria and identified 67 journal papers and 104 conference papers. From this sample we identified 11 journal papers and 9 conference papers that used both methods. RESULTS – our analysis found that about 25% of studies were internally inconclusive. We also found that there is approximately equal evidence in favour of, and against analogy-based methods. CONCLUSIONS – we confirm the lack of consistency in the findings and argue that this inconsistent pattern from 20 different studies comparing regression and analogy is somewhat disturbing. It suggests that we need to ask more detailed questions than just: “What is the best prediction system?

Experimental software engineering in educational context

Empirical studies are important in software engineering to evaluate new tools, techniques, methods and technologies in a structured way before they are introduced in the industrial (real) software process. Within this PhD thesis we will develop a framework of a consistent process for involving students as subjects of empirical studies of software engineering. In concrete, our experiences with software development teams composed of students will analyze how RUP (Rational Unified Process) processes can be compliant with the CMMI (Capability Maturity Model Integration), namely in the context of MLs (maturity levels) 2 and 3. Additionally, we will also analyze the influence of project management tools to improve the process maturity of the teams. Our final goal of carrying out empirical studies with students is to understand its validity when compared with the corresponding studies in real industrial settings

Building on CHASM: A Study of Using Counts for the Analysis of Static Models of Processes

Process modelling is gaining increasing acceptance by software engineers as a useful discipline to facilitate both process understanding and improvement activities. This position paper builds upon previous work reported at the 1997 ICSE workshop on process models and empirical studies of software engineering (Phalp and Counsell 1997). In the previous paper, we argued that simple counts could be used to support analysis of static process models. We also illustrated the idea with a coupling measure for Role Activity Diagrams, a graphical process modelling notation adapted from Petri Nets. At that time only limited empirical work had been carried out, based upon a single industrial study, where we found high levels of coupling in an inefficient process (a more thorough description may be found in (Phalp and Shepperd 1999)). We now summarise a more recent study, which uses a similar analysis of process coupling again based on simple counts. In the study, we compared ten software prototyping processes drawn from eight different organisations. We found that this approach does yield insights into process problems, which could potentially be missed by qualitative analysis alone. This is particularly so when analysing real world processes, which are frequently more complex than their text book counterparts. One notable finding was that despite differences in size and domain, role types across the organisations exhibited similar levels of coupling. Furthermore, where there were deviations in one particular role type, this led the authors to discover a relationship between project size and the coupling levels within that type of role. Given the simplicity of our approach and the complexity of many real world processes we argue that quantitative analysis of process models should be considered as a process analysis technique

Consent Issues Raised by Observational Research in Organisations

non

Using Counts as Heuristics for the Analysis of Static Models

The upstream activities of software development are often viewed as both the most important, in terms of cost, and the yet the least understood, and most problematic, particularly in terms of satisfying customer requirements. Business process modelling is one solution that is being increasingly used in conjunction with traditional software development, often feeding in to requirements and analysis activities. In addition, research in Systems Engineering for Business Process Change, highlights the importance of modelling business processes in evolving and maintaining the legacy systems that support those processes. However, the major use of business process modelling, is to attempt to restructure the business process, in order to improve some given aspect, e.g., cost or time. This restructuring may be seen either as separate activity or as a pre-cursor to the development of systems to support the new or improved process. Hence, the analysis of these business models is vital to the improvement of the process, and as a consequence to the development of supporting software systems. Supporting this analysis is the focus of this paper. Business processes are typically described with static (diagrammatic) models. This paper proposes the use of measures (counts) to aid analysis and comparison of these static process descriptions. The proposition is illustrated by showing how measures can be applied to a commonly used process-modelling notation, Role Activity Diagrams (RADs). Heuristics for RADs are described and measures suggested which support those heuristics. An example process is used to show how a coupling measure can be used to highlight features in RADs useful to the process modeller. To fully illustrate the proposition the paper describes and applies a framework for the theoretical validation of the coupling measure. An empirical evaluation follows. This is illustrated by two case studies; the first based on the bidding process of a large telecommunications systems supplier, and the second a study of ten prototyping processes across a number of organisations. These studies found that roles of the same type exhibited similar levels of coupling across processes. Where roles did not adhere to tentative threshold values, further investigation revealed unusual circumstances or hidden behaviour. Notably, study of the prototyping roles, which exhibited the greatest variation in coupling, found that coupling was highly correlated with the size of the development team. This suggests that prototyping in large projects had a different process to that for small projects, using more mechanisms for communication. Hence, the empirical studies support the view that counts (measures) may be useful in the analysis of static process models

Some Findings Concerning Requirements in Agile Methodologies

gile methods have appeared as an attractive alternative to conventional methodologies. These methods try to reduce the time to market and, indirectly, the cost of the product through flexible development and deep customer involvement. The processes related to requirements have been extensively studied in literature, in most cases in the frame of conventional methods. However, conclusions of conventional methodologies could not be necessarily valid for Agile; in some issues, conventional and Agile processes are radically different. As recent surveys report, inadequate project requirements is one of the most conflictive issues in agile approaches and better understanding about this is needed. This paper describes some findings concerning requirements activities in a project developed under an agile methodology. The project intended to evolve an existing product and, therefore, some background information was available. The major difficulties encountered were related to non-functional needs and management of requirements dependencies

Preliminary Results in a Multi-site Empirical Study on Cross-organizational ERP Size and Effort Estimation

This paper reports on initial findings in an empirical study carried out with representatives of two ERP vendors, six ERP adopting organizations, four ERP implementation consulting companies, and two ERP research and advisory services firms. Our study’s goal was to gain understanding of the state-of-the practice in size and effort estimation of cross-organizational ERP projects. Based on key size and effort estimation challenges identified in a previously published literature survey, we explored some difficulties, fallacies and pitfalls these organizations face. We focused on collecting empirical evidence from the participating ERP market players to assess specific facts about the state-of-the-art ERP size and effort estimation practices. Our study adopted a qualitative research method based on an asynchronous online focus group