Quality of Design, Analysis and Reporting of Software Engineering Experiments:A Systematic Review

Background: Like any research discipline, software engineering research must be of a certain quality to be valuable. High quality research in software engineering ensures that knowledge is accumulated and helpful advice is given to the industry. One way of assessing research quality is to conduct systematic reviews of the published research literature. Objective: The purpose of this work was to assess the quality of published experiments in software engineering with respect to the validity of inference and the quality of reporting. More specifically, the aim was to investigate the level of statistical power, the analysis of effect size, the handling of selection bias in quasi-experiments, and the completeness and consistency of the reporting of information regarding subjects, experimental settings, design, analysis, and validity. Furthermore, the work aimed at providing suggestions for improvements, using the potential deficiencies detected as a basis. Method: The quality was assessed by conducting a systematic review of the 113 experiments published in nine major software engineering journals and three conference proceedings in the decade 1993-2002. Results: The review revealed that software engineering experiments were generally designed with unacceptably low power and that inadequate attention was paid to issues of statistical power. Effect sizes were sparsely reported and not interpreted with respect to their practical importance for the particular context. There seemed to be little awareness of the importance of controlling for selection bias in quasi-experiments. Moreover, the review revealed a need for more complete and standardized reporting of information, which is crucial for understanding software engineering experiments and judging their results. Implications: The consequence of low power is that the actual effects of software engineering technologies will not be detected to an acceptable extent. The lack of reporting of effect sizes and the improper interpretation of effect sizes result in ignorance of the practical importance, and thereby the relevance to industry, of experimental results. The lack of control for selection bias in quasi-experiments may make these experiments less credible than randomized experiments. This is an unsatisfactory situation, because quasi-experiments serve an important role in investigating cause-effect relationships in software engineering, for example, in industrial settings. Finally, the incomplete and unstandardized reporting makes it difficult for the reader to understand an experiment and judge its results. Conclusions: Insufficient quality was revealed in the reviewed experiments. This has implications for inferences drawn from the experiments and might in turn lead to the accumulation of erroneous information and the offering of misleading advice to the industry. Ways to improve this situation are suggested

An i*-based Reengineering Framework for Requirements Engineering

Avui en dia, els sistemes d'informació són un actiu clau en les organitzacions i sovint els proporcionen un avantatges competitiu. Per a que això segueixi així, han de ser mantinguts i evolucionats d'acord amb els objectius estratègics de la organització. Aquesta evolució inclou els requeriments del sistema d'informació, la tecnologia emprada i els processos suportats. L'impacte dels canvis pot anar des de petites modificacions al desenvolupament d'un nou sistema d'informació i, per aquest motiu, l'evolució dels sistemes d'informació s'analitza durant la fase de requeriments, on es possible avaluar-ne la magnitud utilitzant menys recursos. Des d'aquest punt de vista, els mètodes de l'enginyeria de requeriments i els de la reenginyeria de processos sovint comparteixen els mateixos objectius i es pot considerar que la reenginyeria de processos es adequada tant per al desenvolupament com per al manteniment dels sistemes d'informació. El llenguatge i* està orientat a objectius i permet modelar els sistemes d'informació en termes d'actors i dependencies entre ells. El llenguatge i* s'utilitza en l'enginyeria de requeriments i en la reenginyeria de processos de negoci, però no existeixen gaires propostes comunes a ambdues disciplines. Amb l'objectiu d'utilitzar el llenguatge i* en la reenginyeria de processos, s'ha definit PRiM, un mètode basat en i* per a la reenginyeria de processos (Proces Reenginieering i*-based Method). PRiM assumeix que ja existeix un procés que s'utilitzarà com a punt de partida per l'especificació o l'evolució del nou sistema d'informació. El mètode PRiM consta de sis fases: 1) l'anàlisi dels processos i dels sistemes d'informació actuals, 2) la construcció del model i*, 3) la reenginyeria dels processos actuals, 4) la generació de models i* representant les diferents alternatives, 5) l'avaluació de les alternatives utilitzant mètriques estructurals i 6) l'especificació del nou sistema d'informació a partir del model i* escollit. En les sis fases de PRiM, s'utilitzen diferents mètodes i tècniques algunes creades expressament pel mètode i d'altres provinents de l'enginyeria de requeriments i la reenginyeria de processos. Tot i això, hi ha altres mètodes i tècniques que poden ser utilitzades enlloc d'aquestes i que poden ser mes convenients quan les condicions d'aplicació del mètode canvien. Per tal de permetre la selecció i inclusió d'altres tècniques, es proposa l'aplicació de l'enginyeria de mètodes (Method Engineering). Aquesta disciplina permet construir nous mètodes a partir de parts de mètodes ja existents, i s'ha utilitzat per definir un mètode marc per a la reenginyeria anomenat ReeF (Reengineering Framework). A ReeF, les sis fases de PRiM es presenten de forma genèrica per tal de permetre la selecció de la tècnica més apropiada per cada una de les fases, a partir de l'experiència de l'usuari com dels seus coneixements de l'aplicació. Com a exemple d'aplicació de ReeF, s'ha definit el mètode SARiM.Les contribucions principals de la tesis son dues. En primer lloc, els dos mètodes basats en i* definits (PRiM per a la reenginyeria de processos, i SARiM, per a la reenginyeria d'arquitectures software). En segon lloc, les diferents tècniques i* definides en PRiM i que poden ser utilitzades per construir models i*, generar alternatives i avaluar-les amb mètriques estructurals. Aquestes tècniques i mètodes s'han obtingut a partir de l'estudi de l'estat de l'art i s'han validat en diferents casos d'estudi formatius i en un cas d'estudi industrial. Com a suport, s'han desenvolupat dues eines: 1) REDEPEND-REACT, que permet la modelització gràfica de models i*, la generació d'alternatives i la definició de mètriques estructurals, i 2) J-PRiM, que dóna suport a les fases de PRiM mitjançant el tractament textual dels models i*.Information Systems are a crucial asset of the organizations and can provide competitive advantages to them. However, once the Information System is built, it has to be maintained and evolved, which includes changes on the requirements, the technology used, or the business processes supported. All these changes are diverse in nature and may require different treatments according to their impact, ranging from small improvements to the deployment of a new Information System. In both situations, changes are addressed at the requirements level, where decisions are analysed involving less resources. Because Requirements Engineering and Business Process Reengineering methods share common activities, and the design of the Information System with the business strategy has to be maintained during its evolution, a Business Process Reengineering approach is adequate for addressing Information Systems Development when there is an existing Information System to be used as starting point. The i* framework is a well-consolidated goal-oriented approach that allows to model Information Systems in a graphical way, in terms of actors and dependencies among them. The i* framework addresses Requirements Engineering and Business Process Reengineering but none of the i*-based existing approaches provides a complete framework for reengineering. In order to explore the applicability of i* for a reengineering framework, we have defined PRiM: a Process Reengineering i* Method, which assumes that there is an existing process that is the basis for the specification of the new Information System. PRiM is a six-phase method that combines techniques from the fields of Business Process Reengineering and Requirements Engineering and defines new techniques when needed. As a result PRiM addresses: 1) the analysis of the current process using socio-technical analysis techniques; 2) the construction of the i* model by differentiating the operationalization of the process form the strategic intentionality behind it; 3) the reengineering of the current process based on its analysis for improvements using goal acquisition techniques; 4) the generation of alternatives based on heuristics and patterns; 5) the evaluation of alternatives by defining structural metrics; and, 6) the specification of the new Information System from the selected i* model.There are several techniques from the Requirements Engineering and Business Process Reengineering fields, that can be used instead the ones selected in PRiM. Therefore, in order to not enforce the application of a certain technique we propose a more generic framework where to use and combine them. Method Engineering is the discipline that constructs new methods from parts of existing ones and, so, it is the approach adopted to define ReeF: a Reengineering Framework. In ReeF the six phases of PRiM are abstracted and generalized in order to allow selecting the most appropriate techniques for each of the phases, depending on the user expertise and the domain of application. As an example of the applicability of ReeF, the new method SARiM is defined. The main contributions of this work are twofold. On the one hand, two i*-based methods are defined: the PRiM method, which addresses process reengineering, and SARiM, which addresses software architecture reengineering. On the other hand, we provide several i*-based techniques to be used for constructing i* models, generating alternatives, and evaluating them using Structural Metrics. These methods and techniques are based on exhaustive review of existing work and their validation is done by means of several formative case studies and an industrial case study. Tool support has been developed for the approach: REDEPEND-REACT supporting the graphical modelling of i*, the generation of alternatives and the definition of Structural Metrics; and J-PRiM supporting all the phases of the PRiM method using a textual visualization of the i* models