Evolution of statistical analysis in empirical software engineering research: Current state and steps forward

Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001--2015 and 5,196 papers. Results from both review steps was used to: i) identify and analyze the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.Comment: journal submission, 34 pages, 8 figure

COOPER-framework: A Unified Standard Process for Non-parametric Projects

Practitioners assess performance of entities in increasingly large and complicated datasets. If non-parametric models, such as Data Envelopment Analysis, were ever considered as simple push-button technologies, this is impossible when many variables are available or when data have to be compiled from several sources. This paper introduces by the ‘COOPER-framework’ a comprehensive model for carrying out non-parametric projects. The framework consists of six interrelated phases: Concepts and objectives, On structuring data, Operational models, Performance comparison model, Evaluation, and Result and deployment. Each of the phases describes some necessary steps a researcher should examine for a well defined and repeatable analysis. The COOPER-framework provides for the novice analyst guidance, structure and advice for a sound non-parametric analysis. The more experienced analyst benefits from a check list such that important issues are not forgotten. In addition, by the use of a standardized framework non-parametric assessments will be more reliable, more repeatable, more manageable, faster and less costly.DEA, non-parametric efficiency, unified standard process, COOPER-framework.

On the Role of Theory and Evidence in Macroeconomics

This paper, which is prepared for the Inagural Conference of the Institute for New Economic Thinking in King's College, Cambridge, 8-11 April 2010, questions the preeminence of theory over empirics in economics and argues that empirical econometrics needs to be given a more important and inde- pendent role in economic analysis, not only to have some confidence in the soundness of our empirical inferences, but to uncover empirical regularities that can serve as a basis for new economic thinking.

Did inflation really soar after the euro cash changeover? Indirect evidence from ATM withdrawals

The introduction of the euro notes and coins in the first two months of 2002 was followed by a lively debate on the alleged inflationary effects of the new currency. In Italy, as in the rest of the euro area, survey-based measures signaled a much sharper rise in inflation than measured by the official price indices, whose quality was called into question. In this paper we gather indirect evidence on the behavior of prices from the analysis of cash withdrawals from ATM and their determinants. Since these data do not rely on official inflation statistics, they provide an independent check for the latter. We present a model in which the relationship between aggregate ATM withdrawals and aggregate expenditure is not homogenous of degree one in the price level, a prediction which is strongly supported by the data. This feature allows us to test the hypothesis that, after the introduction of the euro notes and coins, consumer prices underwent an increase not recorded by official inflation statistics. We do not find evidence in support of this hypothesis.banknotes, currency, euro, inflation.

Time to reject the privileging of economic theory over empirical evidence? A Reply to Lawson (2009)

The present financial and economic crisis has revealed a systemic failure of academic economics and emphasized the need to re-think how to model economic phenomena. Lawson (2009) seems concerned that critics of standard models now will fill academic journals with contributions that make the same methodological mistakes, albeit in slightly different guise. In particular, he is rather sceptical to use of mathematical statistical models, such as the CVAR approach, as a way of learning about economic mechanisms. In this paper I discuss whether this is a relevant claim and argue that it is likely to be based on a misunderstanding of what a proper statistical analysis is and can offer. In particular, I argue that the strong evidence of (near) unit roots and (structural) breaks in economic variables suggests that standard economic models need to be modified or changed to incorporate these strong features of the data. Furthermore, I argue that a strong empirical methodology that allows data to speak freely about economic mechanisms, such as the CVAR, would ensure that important information in the data is not over heard when needed. Adequately applied such models would provide us with an early warnings system signalling that the economy is moving seriously out of equilibrium.economic crisis; Dahlem report; CVAR approach; Theory-first; Reality-first; Imperfect Knowledge Expectations; non-stationary data