Book review: creating the market university: how academic science became an economic engine by Elizabeth Popp Berman

Contributing to debates about the relationship between universities, government, and industry, Elizabeth Popp Berman sheds light on how knowledge and politics intersect to structure the economy. Arnaud Vaganay finds the book to be extremely topical: UK universities are now catching up with their American rivals to spur on the growth the country so badly needs

Book review: exceptional people: how migration shaped our world and will define our future

Arnaud Vaganay finds Ian Goldin’s recent book to be a great contribution to migration studies, packed with statistics, maps, and important discussions on migratory pressure in Europe

Book review: the tyranny of utility: behavioral social science and the rise of paternalism.

Arnaud Vaganay finds a courageous and original contribution to the field of behavioural economics in The Tyranny of Utility, but is not convinced by the author’s call for value-based policies in the context of the current debt crisis.

Cluster sampling bias in government-sponsored evaluations: a correlational study of employment and welfare pilots in England

For pilot or experimental employment programme results to apply beyond their test bed, researchers must select ‘clusters’ (i.e. the job centres delivering the new intervention) that are reasonably representative of the whole territory. More specifically, this requirement must account for conditions that could artificially inflate the effect of a programme, such as the fluidity of the local labour market or the performance of the local job centre. Failure to achieve representativeness results in Cluster Sampling Bias (CSB). This paper makes three contributions to the literature. Theoretically, it approaches the notion of CSB as a human behaviour. It offers a comprehensive theory, whereby researchers with limited resources and conflicting priorities tend to oversample ‘effect-enhancing’ clusters when piloting a new intervention. Methodologically, it advocates for a ‘narrow and deep’ scope, as opposed to the ‘wide and shallow’ scope, which has prevailed so far. The PILOT-2 dataset was developed to test this idea. Empirically, it provides evidence on the prevalence of CSB. In conditions similar to the PILOT-2 case study, investigators (1) do not sample clusters with a view to maximise generalisability; (2) do not oversample ‘effect-enhancing’ clusters; (3) consistently oversample some clusters, including those with higher-than-average client caseloads; and (4) report their sampling decisions in an inconsistent and generally poor manner. In conclusion, although CSB is prevalent, it is still unclear whether it is intentional and meant to mislead stakeholders about the expected effect of the intervention or due to higher-level constraints or other considerations

Outcome reporting bias in government-sponsored policy evaluations: a qualitative content analysis of 13 Studies

The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light–especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or ‘spin’) in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government’s commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform’s champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended

An Evaluation of the Hand Hygiene Behaviour and Compliance of the General Public When Using Public Restrooms in Northern Ireland (NI) during the Initial Weeks of the Novel Coronavirus (COVID-19) Pandemic.

Background: The ongoing novel coronavirus (COVID-19) global pandemic has resulted in significant levels of morbidity and mortality worldwide, particularly among the elderly and immuno-suppressed groups. Although adequate hand hygiene (HH) behaviour and compliance is widely accepted as being the most effective self-protective measure in preventing the spread of diseases like COVID-19, previous research suggests that normal hand hygiene compliance is poor, but generally improves during a disease pandemic. This research aimed to evaluate the hand hygiene behaviour and compliance of the general public in the initial weeks of the COVID-19 pandemic in Northern Ireland (NI). Methods: This cross-sectional study involved the use of infrared-imaging cameras to observe the hand hygiene behaviour and compliance of the general public when using one set of male and female public restrooms. Results: The findings of this study indicated that the level of hand hygiene compliance of the general public was poor in the initial weeks, with 82.93% overall not washing their hands adequately. Conclusions: Inadequate HH behaviour and compliance may have added significantly to the rapid rate of spread of COVID-19 in the initial weeks of the pandemic in NI. Current public health campaigns do not appear, based on this study, to have the desired impact and may need to be reviewed or re-enforced in order to achieve the levels of hand hygiene compliance required to slow the spread of COVID-19 and other communicable diseases in the future

To save the research literature, let's make literature reviews reproducible

Last week the Impact Blog featured a post from Richard P. Phelps, in which he proposed that journals get rid of their requirement for a literature review. Arnaud Vaganay agrees with much of what Phelps said, literature reviews are erratic and self-serving, but suggests doing away with them altogether is likely to make science less efficient and less credible. Instead, literature reviews and their component parts should be thought of as reproducible research assets, just like the protocol and dataset used in the empirical part of a study. Cumulative literature reviews should be based on a methodology that is more systematic, reproducible, and reusable than the traditional, narrative literature review

Evidence-based policy or policy-based evidence? The effect of policy commitment on government-sponsored evaluation in Britain (1997-2010)

In most mature welfare states, policy evaluations are sponsored by the very organisations that designed and implemented the intervention in the first place. Research in the area of clinical trials has consistently shown that this type of arrangement creates a moral hazard and may lead to overestimates of the effect of the treatment. Yet, no one so far has investigated whether social interventions were subject to such ‘confirmation bias’. The objective of this study was twofold. Firstly, it assessed the scientific credibility of a sample of government-sponsored pilot evaluations. Three common research prescriptions were considered: (a) the proportionality of timescales, (b) the representativeness of pilot sites; and (c) the completeness of outcome reporting. Secondly, it examined whether the known commitment of the government to a reform was associated with less credible evaluations. These questions were answered using a ‘meta-research’ methodology, which departs from the traditional interviews and surveys of agents that have dominated the literature so far. I developed the new PILOT dataset for that specific purpose. PILOT includes data systematically collected from over 230 pilot and experimental evaluations spanning 13 years of government-commissioned research in the UK (1997-2010) and four government departments (Department for Work and Pensions, Department for Education, Home Office and Ministry of Justice). PILOT was instrumental in (a) modeling pilot duration using event history analysis; (b) modeling pilot site selection using logistic regression; and (c) the systematic selection of six evaluation reports for qualitative content analysis. A total of 17 interviews with policy researchers were also conducted to inform the case study and the overall research design. The results show little overt evidence of crude bias or ‘bad’ design. On average, government-sponsored pilots (a) were based on timescales that were proportional to the scope of the research; (b) were not primarily designed with the aim of warranting representativeness; and (c) were rather comprehensively analysed in evaluation reports. In addition, the results indicate that the known commitment of the government to a reform had no significant effect on the selection of pilot sites and on the reporting of outcomes. However, it was associated with significantly shorter pilots. In conclusion, there is some evidence that the known commitment of a government to a reform is associated with less credible evaluations; however this effect is only tangible in the earlier stages of the research cycle. In this respect, sponsorship bias would appear to be more limited than in the context of industry-sponsored clinical trials. Policy recommendations are provided, as this project was severely hindered by important ‘black box’ issues and by the poor quality of evaluation reports