Quality of Design, Analysis and Reporting of Software Engineering Experiments:A Systematic Review

Background: Like any research discipline, software engineering research must be of a certain quality to be valuable. High quality research in software engineering ensures that knowledge is accumulated and helpful advice is given to the industry. One way of assessing research quality is to conduct systematic reviews of the published research literature. Objective: The purpose of this work was to assess the quality of published experiments in software engineering with respect to the validity of inference and the quality of reporting. More specifically, the aim was to investigate the level of statistical power, the analysis of effect size, the handling of selection bias in quasi-experiments, and the completeness and consistency of the reporting of information regarding subjects, experimental settings, design, analysis, and validity. Furthermore, the work aimed at providing suggestions for improvements, using the potential deficiencies detected as a basis. Method: The quality was assessed by conducting a systematic review of the 113 experiments published in nine major software engineering journals and three conference proceedings in the decade 1993-2002. Results: The review revealed that software engineering experiments were generally designed with unacceptably low power and that inadequate attention was paid to issues of statistical power. Effect sizes were sparsely reported and not interpreted with respect to their practical importance for the particular context. There seemed to be little awareness of the importance of controlling for selection bias in quasi-experiments. Moreover, the review revealed a need for more complete and standardized reporting of information, which is crucial for understanding software engineering experiments and judging their results. Implications: The consequence of low power is that the actual effects of software engineering technologies will not be detected to an acceptable extent. The lack of reporting of effect sizes and the improper interpretation of effect sizes result in ignorance of the practical importance, and thereby the relevance to industry, of experimental results. The lack of control for selection bias in quasi-experiments may make these experiments less credible than randomized experiments. This is an unsatisfactory situation, because quasi-experiments serve an important role in investigating cause-effect relationships in software engineering, for example, in industrial settings. Finally, the incomplete and unstandardized reporting makes it difficult for the reader to understand an experiment and judge its results. Conclusions: Insufficient quality was revealed in the reviewed experiments. This has implications for inferences drawn from the experiments and might in turn lead to the accumulation of erroneous information and the offering of misleading advice to the industry. Ways to improve this situation are suggested

Auto-organisation de fragments pour la conception de processus de développement

Dans le domaine de l'ingénierie du logiciel, plusieurs méthodes ont été conçues, chacune avec ses avantages et ses inconvénients. Même s'il en existe à ce jour de nombreuses, l'expérience a montré que ces méthodes ne sont pas universelles et ne peuvent prévoir toutes les situations possibles. L'évolution actuelle des systèmes impose des modifications de plus en plus rapides et fréquentes de leurs méthodes. Certaines tentatives ont essayé d'unifier des processus de développement et leurs méta-modèles mais se sont révélées infaisables jusqu'à présent. Une autre voie explorée a été de profiter de ces nombreuses méthodes en essayant de combiner leurs caractéristiques particulières. Les travaux sur l'ingénierie des méthodes situationnelles ont cherché plus précisément à permettre la réutilisation de fragments de démarches existantes pour en construire de nouvelles mieux adaptées aux spécificités d'une situation. Cette famille d'approches a l'avantage de préserver les acquis et d'apporter également une certaine flexibilité en fournissant les moyens d'adapter une méthode aux besoins spécifiques d'une situation particulière. C'est dans ce contexte que s'est inscrit mon travail de thèse. Ma contribution dans ce domaine vise à apporter des éléments de réponses pour la conception automatique de processus composé de fragments dont les verrous sont les suivants : (i) comment sélectionner les fragments ; (ii) comment les composer pour obtenir un processus de méthode ; (iii) comment évaluer le résultat de la composition. Une première contribution essentielle de cette thèse est axée sur l'analyse des contraintes et besoins nécessaires à la composition automatique de fragments à savoir la standardisation de fragments, leur sélection et assemblage ainsi que rôle de l'utilisateur avant et pendant la composition. Ces travaux m'ont permis de proposer le système SCoRe (\textit{Self-Combining method fRagments}) qui auto-compose des fragments de méthodes à l'aide d'un système multi-agents adaptatif afin de déterminer un processus adapté et adaptable à des situations spécifiques. Son objectif est, d'une part, d'assister au maximum l'équipe de développement dans la réalisation de son projet, et, d'autre part, de s'adapter aux potentielles évolutions pendant l'exécution du processus composé. Une deuxième contribution réside dans la définition de critères et méthodes pour l'évaluation de processus existants ou résultants de la production du système multi-agent adaptatif SCoRe. L'objectif de ses évaluations est de mesurer les caractéristiques structurelles et les performances des méthodes évaluées.Software systems are becoming more and more complex. A common dilemma faced by software engineers in building complex systems is the lack of method adaptability. In the Software Engineering field, several methods have been created with advantages and drawbacks but the experience shows that existing methods are not universal and cannot answer all situations. The system evolution requires method modifications more rapid and frequent. Some researchers tried to unify metamodel. Another way explored was to take advantage of the different methods by combining their characteristics. The situational method engineering (SME) approaches focus on the in-house construction of methods for systems development which are tuned to specific situations of development projects. My work of thesis is included in this context. My main contribution aims at bringing first answers to the automatic process composition. The issues of the creation of tailored processes are the following: (i) selecting appropriate method components from a repository of reusable method components; (ii) tailoring these method components as appropriate; (iii) integrating these tailored method components to form the new situation-specific method. A first main contribution of this thesis aims at studying the ability to design process and adjust the proposed process according to the characteristics of application domain and users profile. I propose an original system called SCoRe (Self-Combined method fRagments) to automatically build a self-adaptive design process where each fragment is encapsulated in an autonomous agent. On one hand, the goal of SCoRe is to assist the designer in the choice of fragment and therefore in the definition of a process most suitable to the current situation. On the other, its goal is to react dynamically to potential modifications. A second contribution resides in the definition of criteria and methods for evaluating existing processes or processes assembled by SCoRe. The aim of these evaluations is to measure structural characteristics and performance of evaluated methods