Quality of Design, Analysis and Reporting of Software Engineering Experiments:A Systematic Review

Background: Like any research discipline, software engineering research must be of a certain quality to be valuable. High quality research in software engineering ensures that knowledge is accumulated and helpful advice is given to the industry. One way of assessing research quality is to conduct systematic reviews of the published research literature. Objective: The purpose of this work was to assess the quality of published experiments in software engineering with respect to the validity of inference and the quality of reporting. More specifically, the aim was to investigate the level of statistical power, the analysis of effect size, the handling of selection bias in quasi-experiments, and the completeness and consistency of the reporting of information regarding subjects, experimental settings, design, analysis, and validity. Furthermore, the work aimed at providing suggestions for improvements, using the potential deficiencies detected as a basis. Method: The quality was assessed by conducting a systematic review of the 113 experiments published in nine major software engineering journals and three conference proceedings in the decade 1993-2002. Results: The review revealed that software engineering experiments were generally designed with unacceptably low power and that inadequate attention was paid to issues of statistical power. Effect sizes were sparsely reported and not interpreted with respect to their practical importance for the particular context. There seemed to be little awareness of the importance of controlling for selection bias in quasi-experiments. Moreover, the review revealed a need for more complete and standardized reporting of information, which is crucial for understanding software engineering experiments and judging their results. Implications: The consequence of low power is that the actual effects of software engineering technologies will not be detected to an acceptable extent. The lack of reporting of effect sizes and the improper interpretation of effect sizes result in ignorance of the practical importance, and thereby the relevance to industry, of experimental results. The lack of control for selection bias in quasi-experiments may make these experiments less credible than randomized experiments. This is an unsatisfactory situation, because quasi-experiments serve an important role in investigating cause-effect relationships in software engineering, for example, in industrial settings. Finally, the incomplete and unstandardized reporting makes it difficult for the reader to understand an experiment and judge its results. Conclusions: Insufficient quality was revealed in the reviewed experiments. This has implications for inferences drawn from the experiments and might in turn lead to the accumulation of erroneous information and the offering of misleading advice to the industry. Ways to improve this situation are suggested

Empirical validation of a usability inspection method for model-driven Web development

Web applications should be usable in order to be accepted by users and to improve their success probability. Despite the fact that this requirement has promoted the emergence of several usability evaluation methods, there is a need for empirically validated methods that provide evidence about their effectiveness and that can be properly integrated into early stages of Web development processes. Model-driven Web development processes have grown in popularity over the last few years, and offer a suitable context in which to perform early usability evaluations due to their intrinsic traceability mechanisms. These issues have motivated us to propose a Web Usability Evaluation Process (WUEP) which can be integrated into model-driven Web development processes. This paper presents a family of experiments that we have carried out to empirically validate WUEP. The family of experiments was carried out by 64 participants, including PhD and Master¿s computer science students. The objective of the experiments was to evaluate the participants¿ effectiveness, efficiency, perceived ease of use and perceived satisfaction when using WUEP in comparison to an industrial widely used inspection method: Heuristic Evaluation (HE). The statistical analysis and meta-analysis of the data obtained separately from each experiment indicated that WUEP is more effective and efficient than HE in the detection of usability problems. The evaluators were also more satisfied when applying WUEP, and found it easier to use than HE. Although further experiments must be carried out to strengthen these results, WUEP has proved to be a promising usability inspection method for Web applications which have been developed by using model-driven development processes.The authors would like to thank all the participants in the experiments, along with the usability experts that supported certain tasks of the evaluation design stage, and of which the control group was composed. This research work is funded by the MULTIPLE project (TIN2009-13838) and the FPU program (AP2007-03731) from the Spanish Ministry of Science and Education.Fernández Martínez, A.; Abrahao Gonzales, SM.; Insfrán Pelozo, CE. (2013). Empirical validation of a usability inspection method for model-driven Web development. Journal of Systems and Software. 86(1):161-186. https://doi.org/10.1016/j.jss.2012.07.043S16118686

Long-Span Brettstapel Roof Structures: A Parametric Design Approach

Bakgrunn: Tre har økt i popularitet som et konstruksjonsmateriale de siste årene, på grunn av et økt fokus på klimaendringer og materialets status som miljøvennlig. I en tid hvor digitalisering gjennomsyrer byggebransjen er kunnskap om hvordan det komplekse materialet kan modelleres avgjørende, og forskning på dette området kreves fremdeles. Massivtre åpner for bruk av tre til flere konstruktive formål. Men, de sammensatte tre-elementene kompliserer den digitale modelleringsprosessen ytterligere. Formål: Denne studien er en todelt undersøkelse av (1) hvordan Norsk Massivtres kantstilte massivtre-element, "Brettstapel", kan utnyttes til å skape takkonstruksjoner med lange spenn over 20 meter, (2) hvordan tre generelt, og massivtre spesielt, kan bli modellert og utforsket digitalt. Metode: Norsk Massivtres Brettstapel element, og ulike takkonstruksjoner bestående av denne, er modellert og analysert ved hjelp av parametrisk design-verktøy og tradisjonelle CAD-programmer. De utforskede takkonstruksjonene er som følger: skrå og flate underspente tak, foldede W-tak og skråtak med Brettstapel-bjelker. En teoretisk sammenligning av Brettstapel og kryss-laminert tre (CLT) er gjennomført for å få kunnskap om Brettstapel's potensiale som materiale for takplater. Resultat: En digital modell av Brettstapel med FEM 3D solide elementer gir en tilfredsstillende simulering av responsen. Forenklede parametriske modeller, ved bruk av FEM skall-elementer, indikerer et potensiale for takkonstruksjonene med lange spenn. Ved å ta begrensninger og feilkilder med i betraktningen, er det de underspente takkonstruksjonene som virker mest lovende. Sammenlignet med CLT har Brettstapel fordeler i forhold til annet arealmoment og rulleskjær. Konklusjon: Resultatene indikerer at massivtres anisotropi og geometriske kompleksitet må tas i betraktning gjennom volumetrisk FEM-modellering for å gi nøyaktige resultater. Hjelpemidler innen parametrisk design kan utnyttes til å gi informasjon om geometri og konstruktivt potensiale. Men, de parametriske modellene er for forenklede til å gi detaljert konstruktiv informasjon. Resultatene i denne studien kan være nyttige for både fremtidig forskning og for den kommersielle industrien