Applying the ROBINS-I tool to natural experiments: an example from public health

Background: A new tool to assess Risk of Bias In Non-randomised Studies of Interventions (ROBINS-I) was published in Autumn 2016. ROBINS-I uses the Cochrane-approved risk of bias (RoB) approach and focusses on internal validity. As such, ROBINS-I represents an important development for those conducting systematic reviews which include non-randomised studies (NRS), including public health researchers. We aimed to establish the applicability of ROBINS-I using a group of NRS which have evaluated non-clinical public health natural experiments. Methods: Five researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to independently assess risk of bias in five studies which had assessed the health impacts of a domestic energy efficiency intervention. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons underpinning lack of consensus for specific questions and bias domains. Results: ROBINS-I helped to systematically articulate sources of bias in NRS. However, the lack of consensus in assessments for all seven bias domains raised questions about ROBINS-I’s reliability and applicability for natural experiment studies. The two RoB domains with least consensus were selection (Domain 2) and performance (Domain 4). Underlying the lack of consensus were difficulties in applying an intention to treat or per protocol effect of interest to the studies. This was linked to difficulties in determining whether the intervention status was classified retrospectively at follow-up, i.e. post hoc. The overall risk of bias ranged from moderate to critical; this was most closely linked to the assessment of confounders. Conclusion: The ROBINS-I tool is a conceptually rigorous tool which focusses on risk of bias due to the counterfactual. Difficulties in applying ROBINS-I may be due to poor design and reporting of evaluations of natural experiments. While the quality of reporting may improve in the future, improved guidance on applying ROBINS-I is needed to enable existing evidence from natural experiments to be assessed appropriately and consistently. We hope future refinements to ROBINS-I will address some of the issues raised here to allow wider use of the tool

Fertility Regulation

In the past two centuries the proportion of couples using some form of conscious pregnancy-prevention has risen from close to zero to about two-thirds. In European populations this radical change in behaviour occurred largely between 1870 and 1930 without the benefit of highly effective methods. In Asia, Africa and Latin America, the change took place after 1950 since when the global fertility rate has halved from 5.0 births to 2.5 births per woman. In this chapter we describe the controversies surrounding the idea of birth control and the role of early pioneers such as Margaret Sanger; the advances in contraceptive and abortion technologies; the ways in which family planning has been promoted by many governments, particularly in Asia; trends in use of specific methods; the problems of discontinuation of use; and the incidence of unintended pregnancies and abortions

Analytical framework and findings

The introduction sets the stage for the book by describing how the global multilateral architecture is cracking under the tectonic power shifts that characterize the world economy. It sketches the debate about the necessity and feasibility of adapting international institutions to these new realities. The introduction explains that the main purpose of the book is to examine how various multilateral institutions come under pressure to adapt to shifting power configurations, how key actors respond to these pressures, and the observed patterns can be interpreted. The chapter further defines key concepts and puts forward a typology of multilateral institutions, before outlining a set of key questions that guide the analysis in each of the subsequent chapters. Finally, the structure of the book and the findings of each chapter are summarized