Following yesterday’s post criticising the LSE research that is underpinning the drive towards competition and choice in the NHS, Henry Overman provides a defence of the research findings and questions the extent to which public understanding of the evidence has been enhanced by this exchange

Overman, Henry G.

English

LSE Research Online

blogs.lse.ac.uk http://blogs.lse.ac.uk/politicsandpolicy/2012/03/06/bad-blogging-overman/NHS competition: bad science or bad blogging?Following yesterday’s post criticising the LSE research that is underpinning thedrive towards competition and choice in the NHS, Henry Overman provides adefence of the research findings and questions the extent to which publicunderstanding of the evidence has been enhanced by this exchange. Generally speaking I support the idea that academic researchers should engagewith public debate. If we have evidence that could help inform policy and widerdebates then it’s right that we should publicise that beyond our own narrowacademic communities. Blogging is one highly effective way of doing that. But every so often I readsomething on an academic blog that makes me pause and question whether more blogging will endup improving the quality of public debate in the long run.Yesterday’s post on NHS competition provides a good example of the kind of post that worriesme. This post criticises research about the impact of NHS competition on patient outcomes andaccuses my LSE colleagues of engaging in bad science, data dredging and faulty analysis. Iassume that this post will have been read by far more people than the original scientific paper.That’s to be expected – after all it is what the blog is trying to achieve. But as I read through the postI became increasingly puzzled by the fact that the criticism in the post appeared to bear little relationto the scientific papers I had read. When I went back to the original papers, this confirmed myoriginal concerns.As I know people may not have access to the academic paper, let me give some concreteexamples.The ‘major cause of reductions in AMI’?BLOG: “using [acute myocardial infarction] AMI mortality as a quality indicator, … mortality fell morequickly (i.e. quality improved) for patients living in more competitive markets after the introduction ofhospital competition (to the NHS) in January 2006″PAPER: “The major improvements in outcome after acute myocardial infarction can be attributed toimprovements in primary prevention in general practice and in hospital care”.These points are not contradictory. Couldn’t the major improvement be attributable to primaryprevention and hospital care while hospitals that face more competition saw mortality fall morequickly?Heart attack victims don’t choose where to be treatedBLOG: “the government’s cardiac Tzar, Sir Roger Boyle, was sufficiently angered by their claims torespond with withering criticism: AMI is a medical emergency: patients can’t choose where to havetheir heart attack or where to be treated!”PAPER: “we expect that AMI mortality will decrease more quickly in more competitive markets frommid-2006 onwards after hospitals were exposed to competition created from the new NHSreimbursement system and the expansion of patient choice. While providers are not explicitlycompeting for AMI patients because competition in the NHS is limited to the market for electivecare, we expect the market-based reforms to result in across-the-board improvements in hospitalperformance, which in turn will result in lower AMI death rates. To that end, Bloom et al. (2010)looked at NHS hospitals and found that better managed hospitals had significantly lower AMImortality and that greater hospital competition was associated with better hospital management.”In short, the paper is quite clear on the mechanism. Competition on elective care improvesmanagement which also happens to benefit AMI. Why not use elective care directly? Becausehospitals can ‘manipulate’ statistics around those in a way that it can’t with AMI precisely becausepatients have no choice! In other words, the authors clearly understand that patients have no choicefor AMI but this helps rather than hinders them in their research.Elective patients don’t choose hospitalsBLOG: “Less than the half patients surveyed in 2008 even remember being given a choice, andonly a tiny proportion made those choices based on data from the NHS choices website.”PAPER: There were three components to the health reform only one of which concerned patientchoice but all three of which sharpened incentives for hospitals. Also, even if patients don’tremember being given a choice: “since GPs are highly active in informing the destination of mostreferrals, GPs now play a substantial role dictating how money flows around the post-[reform] NHS.”There are several ways in which the reforms sharpened incentives for hospitals. Pointing to the factthat patients don’t remember being given a choice doesn’t seriously address whether or not theseincentives worked in practice.No biological mechanism for choice to affect outcomesBLOG: There is no biological mechanism to explain why having a choice of providers for electivehip and knee operations surgery [...] could affect the overall outcomes from AMI where patients donot exercise choice over where they are treated.”But the paper doesn’t ever claim that there is a biological mechanism. It claims there isan economic one via the incentives described above.They ignore the existing evidenceBLOG: “They sweep aside decades of careful economic theory and evidence which shows whymarkets do not work in health services”PAPER: Provides pointers to existing literature (and reviews) and specifically considers thereasons why evidence from the reforms of the mid 1990′s, the internal market might not be veryuseful “because the internal market never created significant financial incentives for hospitals tochange their behaviour”They engage in data dredging and their work should never have been publishedBLOG: “if you repeat an analysis often enough significant statistical associations will appear.” Thework was subsequently published in “the Economic Journal. That it got through that journal’s peer-review process is perhaps indicative of the poor understanding of healthcare and routine data fromreviewers of that journal.”I don’t see any basis for the first of these claims. Data ‘dredging’ is a serious problem – but not onethat appears to apply to this paper (which shows that the results are robust to many differentvariations in specification – the exact opposite of the data mining problem). The Economic Journalis one of the world’s leading peer-reviewed economics journals. I don’t believe that peer review iseverything, but simply insinuating that the referees and editors of that journal don’t know what theyare doing doesn’t cut much weight with me.Bad blogging versus bad scienceI could go on to discuss the errors around the second paper where, e.g., the blog claims that theydon’t control for the mix of operations when the paper actually considers within treatment changes inoutcomes (so mix is irrelevant). But I assume that the authors are perfectly capable of furtherdefending their own research.My point is simply that a blog that is supposed to help improve the public’s understanding of theevidence is carrying a post that is pretty misleading about what the papers actually do, what theyfind and what claims they make about their findings. To my mind, this raises far more concernsabout bad blogging than it does about bad science…This article first appeared on the LSE SERC blog.[Disclosure: Steve Gibbons is research director at the LSE's Spatial Economics Research Centre,for which I am the overall director.  I am also affiliated with the Centre for Economic Performance]Please read our comments policy before posting.You may also be interested in the following posts (automatically generated):1. Bad science concerning NHS competition is being used to support the controversial Healthand Social Care Bill2. Competition between NHS hospitals produces positive incentives. However, introducingprivate hospital competition can have adverse effects3. Hospital competition is good for patients, and for efficiency4. Economic studies showing positive competition effects on hospital performance fullycontrolled for the factors cited by recent critics

NHS competition: bad science or bad blogging?

Abstract

Similar works

Full text

Available Versions

LSE Research Online