What’s methodology got to do with it? Public policy evaluations, observational analysis and RCTs

Are methodological choices critical to the success of an evaluation study? For policy evaluation research, the kind of research that governments and international organizations commission to find out whether policies or other interventions are working, we might expect methodology to play a more important role than for conventional academic research. If the questions evaluation research explores are relatively simple, empirical rather than theoretical issues – above all whether the program works or not, what is going wrong and how might it be fixed if not – and if governments make decisions committing huge public resources based on these evaluations, we might expect those who sponsor and conduct such research to be especially concerned with its scientific credibility as established through the empirical research techniques it uses (Box 31.1). This appears to be the reasoning behind those who advocate policy evaluation research adopting the ‘gold standard’ of randomized controlled trials (RCTs), which are especially popular among politicians and government officials since they are deemed to be ‘the best way of testing whether a policy is working’ (Cabinet Office 2012). However, the activity of evaluating policies is rarely simply a matter of developing and applying a convincing methodology to guide policy by showing government what works and what does not. This chapter looks at the role of methodology in evaluations from the perspective of whether there is any evidence that policy-makers are more likely to pay attention to, or act upon, studies that are deemed to be methodologically superior, whether by virtue of being more sophisticated, rigorous or appropriate. The concern of this chapter is not with establishing the merits and demerits of different methodologies in evaluation studies, but rather with assessing the role of methodology in explaining the impact or lack of impact of any evaluation studies. In practical terms it seeks to answer the question: if a researcher makes additional efforts to increase the integrity or sophistication of the research methodology used to perform an evaluation, will the effort pay off in terms of increased influence for that research

Undergraduate research: an apprenticeship approach to teaching political science methods

Undergraduate research, the practice of teaching students by engaging them in a research project, has a long record of achievement. Research-based evaluations show it is likely to have a range of positive educational and career outcomes for students. Many examinations of these benefits apply to STEM subjects. This paper sets out one approach to undergraduate research in political science, based on an apprenticeship model. Using a small survey of all those who have followed the GV314 course at the London School of Economics since 2004 the paper finds evidence that the benefits of undergraduate research appear to be quite striking outside STEM subjects too

New life at the top: special advisers in British government

While the position of special advisers (SpAds) in British government has become routinised, above all since 1997, we still know little about how they affect everyday life at the top of Whitehall departments. This paper, based on the first ever survey of SpAds, explores what they do and how they interact with policy-makers. SpAds fill a mix of ‘commissar’ and ‘fixer’ roles, and there is no evidence of a sharp division of labour between different types of SpAds. The commissar role they fill might be expected to bring SpAds into conflict with civil servants, yet there is surprisingly little evidence of conflict or rivalry. A variety of features of the special adviser's job appear to create incentives among both SpAds and civil servants to avoid conflict with each other. Since the influence of SpAds appears to depend on their relationship with the minister, claims that SpAds are a new, powerful group that shapes policy, are likely to be highly misleading

Evaluation under contract: government pressure and the production of policy research

Do governments lean on researchers who evaluate their policies to try to get them to produce politically useful results? Do researchers buckle under such pressure? This article, based on a survey of 205 academics who have recently completed commissioned research for government, looks at the degree to which British government departments seek to produce research that is designed to provide ‘political ammunition’, above all making them ‘look good’ or minimizing criticism of their policies. Looking at different stages in the research process – from deciding which policies to evaluate, shaping the nature and conduct of inquiry, and writing the results – the article finds evidence of government sponsors making significant efforts to produce politically congenial results. For the most part, researchers appear to resist these efforts, though the evidence base (researchers' own accounts of their work) suggests that this conclusion be treated with some caution

Galactic microwave emission at degree angular scales

We cross-correlate the Saskatoon Ka and Q-Band Cosmic Microwave Background (CMB) data with different maps to quantify possible foreground contamination. We detect a marginal correlation (2 sigma) with the Diffuse Infrared Background Experiment (DIRBE) 240, 140 and 100 microm maps, but we find no significant correlation with point sources, with the Haslam 408 MHz map or with the Reich and Reich 1420 MHz map. The rms amplitude of the component correlated with DIRBE is about 20% of the CMB signal. Interpreting this component as free-free emission, this normalization agrees with that of Kogut et al. (1996a; 1996b) and supports the hypothesis that the spatial correlation between dust and warm ionized gas observed on large angular scales persists to smaller angular scales. Subtracting this contribution from the CMB data reduces the normalization of the Saskatoon power spectrum by only a few percent.Comment: Minor revisions to match published version. 14 pages, with 2 figures included. Color figure and links at http://www.sns.ias.edu/~angelica/foreground.htm

Imidazole as a Small Molecule Analogue in Two-Component Signal Transduction

In two-component signal transduction systems (TCSs), responses to stimuli are mediated through phosphotransfer between protein components. Canonical TCSs use His → Asp phosphotransfer in which phosphoryl groups are transferred from a conserved His on a sensory histidine kinase (HK) to a conserved Asp on a response regulator (RR). RRs contain the catalytic core of His → Asp phosphotransfer, evidenced by the ability of RRs to autophosphorylate with small molecule analogs of phospho-His proteins. Phosphorelays are a more complex variation of TCSs that additionally utilize Asp → His phosphotransfer through the use of an additional component, the histidine-containing phosphotransfer domain (Hpt), which reacts with RRs both as phosphodonors and phosphoacceptors. Here we show that imidazole has features of a rudimentary Hpt. Imidazole acted as a nucleophile and attacked phosphorylated RRs (RR-P) to produce monophosphoimidazole (MPI) and unphosphorylated RR. Phosphotransfer from RR-P to imidazole required the intact RR active site, indicating that the RR provided the core catalytic machinery for Asp → His phosphotransfer. Imidazole functioned in an artificial phosphorelay to transfer phosphoryl groups between unrelated RRs. The X-ray crystal structure of an activated RR•imidazole complex showed imidazole oriented in the RR active site similarly to the His of an Hpt. Imidazole interacted with RR non-conserved active site residues, which influenced the relative reactivity of RR-P with imidazole versus water. Rate constants for reaction of imidazole or MPI with chimeric RRs suggested that the RR active site contributes to the kinetic preferences exhibited by the YPD1 Hpt

Phase II double-blind placebo-controlled randomized study of armodafinil for brain radiation-induced fatigue

BACKGROUND: Common acute-term side effects of brain radiotherapy (RT) include fatigue, drowsiness, decreased physical functioning, and decreased quality of life (QOL). We hypothesized that armodafinil (a wakefulness-promoting drug known to reduce fatigue and increase cognitive function in breast cancer patients receiving chemotherapy) would result in reduced fatigue and sleepiness for patients receiving brain RT. METHODS: A phase II, multi-institutional, placebo-controlled randomized trial assessed feasibility of armodafinil 150 mg/day in participants receiving brain RT, from whom we obtained estimates of variability for fatigue, sleepiness, QOL, cognitive function, and treatment effect. RESULTS: From September 20, 2010, to October 20, 2012, 54 participants enrolled with 80% retention and 94% self-reported compliance. There were no grade 4-5 toxicities, and the incidence of grade 2-3 toxicities was similar between treatment arms, the most common of which were anxiety and nausea (15%), headaches (19%), and insomnia (20%). There were no statistically significant differences in end-RT or 4 week post-RT outcomes between armodafinil and placebo in any outcomes (Functional Assessment of Chronic Illness Therapy [FACIT]-Fatigue, Brief Fatigue Inventory, Epworth Sleepiness Scale, FACT-Brain, and FACIT-cognitive function). However, in participants with more baseline fatigue, those treated with armodafinil did better than those who received the placebo on the end-RT assessments for several outcomes. CONCLUSION: Armodafinil 150 mg/day was well tolerated in primary brain tumor patients undergoing RT with good compliance. While there was no overall significant effect on fatigue, those with greater baseline fatigue experienced improved QOL and reduced fatigue when using armodafinil. These data suggest that a prospective, phase III randomized trial is warranted for patients with greater baseline fatigue