research
Are we heading towards a replicability crisis in energy efficiency research? A toolkit for improving the quality, transparency and replicability of energy efficiency impact evaluations
- Publication date
- Publisher
- eceee
Abstract
Several high-profile replication failures have called into question the reproducibility of results in medicine,
neuroscience, genetics, psychology and economics (Camerer et al. 2016). A paper published in Science found
that just one third of psychology studies could be replicated when the study was run for a second time (OSC
2015). To our knowledge, there have been no attempted replications of energy efficiency studies; so can we be
confident that the estimated energy savings from policy initiatives like the European roll out of smart meters will
be realised? Or that electric vehicles will reduce carbon emissions by predicted levels? Or is energy heading
towards its own reproducibility crisis? Researchers call for the increased use of randomised control trials (RCTs)
to evaluate energy efficiency policy and the introduction of protocols or guidelines for conducting experiments
(Vine et al. 2014; Frederiks et al. 2016). However, no guidelines for increasing reproducibility have been
proposed. Moreover, RCTs are just one method for causal analysis and RCTs cannot answer all important causal
questions. This paper will outline research methods for improved impact assessment of energy efficiency policy,
including RCTs, but also quasi-experiments and systematic reviews that go beyond the conclusions of single
experiments. It will then present tools for increasing replicability: pre-registration of trials; pre-analysis plans;
reporting standards; synthesis tools and; publication of datasets with computer code in data repositories. Based
on work by our research group at the UCL Energy Institute, we recognize that not all of these tools (mostly from medical trials) provide ‘off-the-shelf’ models for energy efficiency evaluations, and so consider adaptations for
energy research. Our aim is to stimulate discussion and get feedback from the research community at ECEEE so
the toolkit can be developed and potentially adopted more widely