The performance of individual clinicians is being monitored as never before. Su Mason and colleagues discuss the implications of this for clinical trials and recommend what should happen if during a trial the performance of one clinician or one centre is identified as being particularly poor. Tom Treasure, a surgeon, wants the monitoring to be done fairly and to take account of the complexities of clinical practice; and Heather Goodare, a patient, wants to be told when things go wrong.



The Department of Health in England has issued guidelines for research governance stating that healthcare organisations remain responsible for the quality of all aspects of patients' care whether or not some aspects of the care are part of a research study.1 We discuss how this obligation can be met in multicentre trials, given that data on the performance of clinicians are held by the trial management team, not by the host organisation

Lilford, R.

Mason, S.

Nicholl, J.

English

PubMed

Mason, S

Nicholl, J

Lilford, Richard

University of Birmingham Research Portal

What to do about poor clinical performance in clinical trials

S. Mason

Crossref

What to do about poor clinical performance in clinical trials Commentary: Of course patients should be told Commentary: The surgeon is only one factor

White Rose Research Online

This is a repository copy of What to do about poor clinical performance in clinical trials.White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/4910/Article:Mason, S., Nicholl, J. and Lilford, R. (2002) What to do about poor clinical performance in clinical trials. BMJ, 324 (7334). pp. 419-420. ISSN 0959-535X https://doi.org/10.1136/bmj.324.7334.419eprints@whiterose.ac.ukhttp  s  ://eprints.whiterose.ac.uk/Reuse See AttachedTakedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. Education and debateWhat to do about poor clinical performance inclinical trialsSu Mason, Jon Nicholl, Richard LilfordThe performance of individual clinicians is being monitored as never before. Su Mason and colleaguesdiscuss the implications of this for clinical trials and recommend what should happen if during a trialthe performance of one clinician or one centre is identified as being particularly poor. Tom Treasure, asurgeon, wants the monitoring to be done fairly and to take account of the complexities of clinicalpractice; and Heather Goodare, a patient, wants to be told when things go wrong.The Department of Health in England has issued guide-lines for research governance stating that healthcareorganisations remain responsible for the quality of allaspects of patients’ care whether or not some aspects ofthe care are part of a research study.1 We discuss how thisobligation can be met in multicentre trials, given thatdata on the performance of clinicians are held by thetrial management team, not by the host organisation.Should we monitor, and who shoulddo it?If neither the host organisation, nor the trial team,takes responsibility for monitoring performance thenpatients are left with no protection against substandardpractice.2 We are aware of the dangers of applying ahigher standard of scrutiny to clinical trials than toroutine practice, but clinical trials often involve arelatively new treatment (such as an innovative surgicaloperation) where outcomes vary by skill.3 4 With anynew treatment it is appropriate to scrutiniseoutcomes—whether or not the treatment is part of acomparative study. But in a clinical trial who should beresponsible for statistical monitoring of outcomes?Three possibilities arise:x Both the healthcare organisations and the trial teamcould collect and monitor performance data, but this islikely to lead to confusion of responsibility and a wasteof effort.x Healthcare organisations could be required tocollect and monitor outcomes independently of thetrial, but they would have, at best, limited access to thecomparative data. It would thus be hard to judgewhether, say, a 5% rate of blood transfusion was toohigh. This applies especially to new technologies wheregood quality data for comparison do not (yet) exist.x The trial statistician responsible for analysing data forinterim monitoring could scrutinise outcome byclinician or organisation at little extra effort. We suggestthat this is the best option and fits with the concept of“tracker trials.”5 This is the idea that a trial is not simply arandomised comparison of two generic treatmentmethods but also an observational comparison ofsubcategories of treatment (such as different devices anddifferent centres or clinicians). Thus, under our proposal,healthcare organisations would discharge part of theirresponsibility for patient welfare during a trial by explic-itly giving responsibility for monitoring to the trial team.We will not deal here with the tricky statistical issuesof identifying outliers.6–9 The aim of the trial statistician,however, would be to identify clinicians or centres thatlie outside the bounds of acceptable practice—thosewho are in a different division, not just bottom of theirleague. But when such a clinician is identified whatshould happen?Action on poor performanceThe responsibility to future patients, who might beharmed by a particular clinician, trumps all others. TheDepartment of Health guidance states: “The dignity,rights, well being and safety of participants must be theprimary consideration in any research study.”1 Thus,not only must the clinician’s participation in the trial besuspended, but the person responsible for clinical gov-ernance in that clinician’s healthcare organisation mustSummary pointsGuidelines on research governance from theDepartment of Health emphasise the importanceof patient safety in trialsWe suggest that healthcare organisations shouldmake the trial management team responsible formonitoring safety through statistical analysisTaking action on suboptimal results, however,remains the institution’s responsibilityThe rules for monitoring and responding tosuboptimal performance should be made clear toeveryone in advanceNorthern andYorkshire ClinicalTrials and ResearchUnit, University ofLeeds, LeedsLS2 9NGSu Masonjoint headMedical CareResearch Unit,University ofSheffield, SheffieldS1 4DAJon NicholldirectorDepartment ofPublic Health andEpidemiology,University ofBirmingham,BirminghamB15 2TTRichard Lilfordprofessor of clinicalepidemiologyCorrespondence to:Professor Lilford(r.j.lilford@bham.ac.uk)BMJ 2002;324:419–21419BMJ VOLUME 324 16 FEBRUARY 2002 bmj.combe informed. The organisation’s usual response mightbe to offer retraining to the clinician.However, three further issues arise.Firstly, we need to consider the welfare (particularlythe anonymity) of collaborating clinicians who take partin trials. It would not be in the public interest if cliniciansdeclined to take part in studies for fear that they might“incriminate” themselves.10 The anonymity of cliniciansshould therefore be respected even though their use ofparticular procedures is suspended. Since the purposeof the trial is to influence practice at large, trialprocedures should follow routine practice as closely aspossible. If we assume regular audit of results underclinical governance, then suboptimal performance indaily practice will be detected, and the consequencesshould be similar to those that occur in a trial—namely,suspension of activity pending retraining. (Detection ofpoor performance in routine post-trial practice would,of course, be facilitated by the availability of “bench-mark” results from preceding trials.)Secondly, a methodological issue arises if the datafrom one clinician is effectively censored. We advocatethat the data accrued by the clinician should beincluded up to the point when he or she wassuspended from the trial.The third issue to consider is whether there is aduty to inform patients who have already sufferedcomplications and who were under the care of a clini-cian subsequently found to have substandard results. Ifthere is such a duty then it conflicts with the obligationto protect clinicians’ anonymity. There are strong argu-ments against the routine feedback of individualperformance to past patients, which include thecreation of perverse incentives (for example, an incen-tive to treat only patients likely to have good outcomes)and the limited likelihood of net benefit from suchretrospective disclosure. Nevertheless, however societydecides to handle this issue, we think that trial practiceshould mirror routine practice.Next stepsFollowing a reasonable public debate, we suggest thatguidelines should be promulgated by organisationsresponsible for scientific governance. Such guidelinesshould state precisely who is responsible for doingwhat—for example, the trial statistician could beresponsible for identifying “outliers,” the data monitor-ing and ethics committee11 for ratifying conclusions,and the trial steering committee for informing the rel-evant health service organisation, which in turn wouldbe responsible for retraining. Secondly, clinicians andthose responsible for clinical governance should knowin advance that the outcomes of individual clinicianswill be analysed as a trial goes on, and they should beaware of their rights and responsibilities if problemsarise, so that they are not ambushed by the process.Patients should also be told what would happen if theircentre or clinician turned out to have suboptimalresults—for example, the circumstances under whichthis would or would not be divulged.We thank Professor Treasure and Dr Elizabeth Clough forhelpful comments.Contributors: The authors developed the article by a processof iteration, involving meetings and email and telephoneconversations.Funding: SM, JN, and RL are all supported by theDepartment of Health/NHS Executive, but the views expressedhere are entirely their own.Competing interests: None declared.1 www.doh.gov.uk/research/rd3/nhsrandd/researchgovernance.htm(accessed 18 Dec 2001).2 McArdle C. ABC of colorectal cancer. Primary treatment—does thesurgeon matter? BMJ 2000;321:1121-3.3 Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trialsgood for us (in the short term)? Evidence for a “trial effect”. J Clin Epide-miol 2000;54:217-24.4 English TA, Bailey AR, Dark JF, Williams WG. The UK cardiac surgicalregister, 1997-82. BMJ 1984;289:1205-8.5 Lilford R, Braunholtz D, Edwards S, Greenhalgh R. Randomised trialsand fast changing technologies: tracker trials. BMJ 2000;320:43-6.6 De Leval MR, Francois K, Bull C, Brawn W, Spieghalter D. Analysis of acluster of surgical failures. J Thorac Cardiovasc Surg 1994;107:914-24.7 Treasure T. Rational decision making about paediatric cardiac surgery.Lancet 2000;355:948.8 Stark J, Gallivan S, Lovegrove J, Hamilton JR, Monro JL, Pollock JC, Wat-terson KG. Mortality rates after surgery for congenital heart defects inchildren and surgeons’ performance. Lancet 2000;355:1004-7.9 Rothwell PM, Warlow CP. Interpretation of operative risks of individualsurgeons. European Carotid Surgery Trialists’ Collaborative Group.Lancet 1999;353:1325.10 Wu AW. Medical error: the second victim. BMJ 2000;320:726-7.11 MRC guidelines for good clinical practice in clinical trials. London: MRC,1998.Commentary: Of course patients should be toldHeather GoodareClinical trials have been proclaimed to be good for youon the grounds that patients in trials are “likely to betreated to the highest possible standards”1—with theimplication that those not in trials are not. If this is trueit is worrying. But is it true? Do such patients really dobetter?2 In some cases trials may actually damage yourhealth.My own experience of a clinical trial, the notoriousBristol study,3 was not a happy one,4 and I have sinceobserved suboptimal practice in seeking patients’ con-Monitor thisALEXANDERTSIARAS/SPLEducation and debate1 Heron Way,HorshamRH13 6DFHeather Goodarecancer counsellorhm.goodare@virgin.net420 BMJ VOLUME 324 16 FEBRUARY 2002 bmj.comsent for trials, where the benefits are exaggerated andthe risks downplayed. In one case the protocol for amulticentre chemotherapy trial had not even beensubmitted to the local research ethics committee, withunfortunate consequences. The local clinician hadstrayed from the protocol, omitting the assessment ofquality of life, and there were other serious problems.Careful audit by the trial team, as envisaged by Masonand colleagues, would eventually have picked this up.The authors seem to be more concerned withsurgical practice, where skill is of the essence, than withdrug treatments. Certainly, patients would want toknow if their surgeon was found to have a consistentlypoor record. But we need to think of other commonproblems in medical research that statistical monitor-ing may well uncover. Outliers should not be the onlyconcern. What about deliberate fraud? Statisticiansshould also be suspicious if one clinician’s results aretoo good to be true, or too similar. There are manyexamples, such as the returned packs of topical creamin which the tubes had all been squeezed in exactly thesame way (FO Wells, Joint Consensus Conference onMisconduct in Biomedical Research, London, 2000).So if dubious practice is uncovered, should patientsbe told? The authors distinguish between present andfuture patients, but some patients survive for a longtime after treatment. The Bristol women fought theirown battle (one that statisticians should certainly haveanticipated as the purported results were wildlyimprobable). Who is to speak up for truth, honesty, andintegrity when trialists veil their errors under a cloak ofconfidentiality? This will only lead to further mistrustof medical research. Patients and their relatives aremore robust than doctors might think. They are usedto hearing bad news: but the telling of it has to be donewith care and sensitivity.1 Tobias J. Cancer: what every patient needs to know. London: Bloomsbury,1995:63-72.2 Bydder S. It needs to be established whether patients really fare better intrials. BMJ 2000;321:1530-1.3 Bagenal FS, Easton DF, Harris E, Chilvers CED, McElwain TJ. Survival ofpatients with breast cancer attending Bristol Cancer Help Centre. Lancet1990;336:606-10.4 Goodare H,ed. Fighting spirit: the stories of women in the Bristol breast cancersurvey. London: Scarlet Press, 1996.Commentary: The surgeon is only one factorTom TreasurePatients deserve safety but it is disingenuous to link trialswith risk rather than benefit. Patients within trials havebetter outcomes1 and it is arguable that trials are saferthan routine care. Safety requires an amalgam of clinicaljudgment, attention to detail, commitment, care, techni-cal skill, and the close monitoring of performance. Theseattributes are as likely to be found among doctors doingresearch as among those who are not. If we are to recruitpatients to trials they must believe (with justification) thatwe will take good care of them. Monitoring of individualperformance will help to ensure that.There is precedent for trial monitors to be the firstto identify poor technical results. In the first of the cor-onary surgery multicentre trials there was a high mor-tality in 1970-2, which was noted and documented.2Simply archiving adverse data is not acceptable, but thestatistical significance of outliers remains problematicwhen individual numbers are small. An extreme butreal example is a trial of multimodality treatment ofsuperior sulcus lung cancer in which 111 patients wereoperated on by 76 different thoracic surgeons, amedian of one operation per surgeon.3 If we were totake the unreasonable step of attributing the outcometo the surgeon alone, most surgeons had 0% mortalitybut, of course, a few had 100%.A specific problem taxing us now is recruitingpatients into the MRC LU22 trial of induction chemo-therapy for operable lung cancer. Persuasive evidence(short of proof) suggests that outcomes would be betterif lung resection were preceded by chemotherapy, butthere is the fear that the cancer will progress during thedelay or that chemotherapy will increase the patient’svulnerability. Nothing but a randomised trial canresolve this issue, but surgeons, ever conscious of peri-operative safety, are wary. Personally I would like everypatient considered for multimodality cancer treatmentto be within a randomised trial because the issues arefar too complex and interwoven for any clinician torely on experience, clinical judgment, or “what worksbest in my hands.” Without trials we will continue totrade anecdotes and uncontrolled retrospective clinicalseries that pass for research. We are woefullyinadequate in the care of lung cancer4 and everyoperation outside a trial is data lost forever.When trials involve skilled interventions such as sur-gery it is appropriate to check the track record of the cli-nicians taking part against explicit criteria. This has beendone in trials of the technically exacting procedures onthe carotid arteries,5 but clusters of poor outcomes inmultimodality cancer trials are not necessarily attribut-able to the operating surgeon They are as likely to bedue to preoperative assessment, anaesthesia, drug errorsand side effects, and postoperative care. Surgical compe-tence is part of the whole treatment package and shouldbe documented along with everything else. But we alsoshould remember that for the big three operations (interms of volume)—cataract surgery, hip replacement,and coronary bypass operations—success relies on theirreproducibility in the hands of many competent andcommitted surgeons—on whom every complication ordeath weighs heavily. I do not mind whether it is trusts ortrialists who monitor standards as long as they do it fairlyand competently.1 Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trialsgood for us (in the short term)? Evidence for a “trial effect.” J Clin Epide-miol 2001;54:217-24.2 Hampton JR. Coronary artery bypass grafting for the reduction of mor-tality: an analysis of the trials. BMJ 1984;289:1166-70.3 Rusch VW, Giroux DJ, Kraut MJ, Crowley J, Hazuka M, Johnson D, et al.Induction chemoradiation and surgical resection for non-small cell lungcarcinomas of the superior sulcus: Initial results of Southwest OncologyGroup Trial 9416 (Intergroup Trial 0160). J Thorac Cardiovasc Surg2001;121:472-83.4 Does lung cancer need a lapel ribbon? Lancet 2000;356:1205.5 Spence D, Eliasziw M. Endarterectomy or angioplasty for treatment ofcarotid stenosis. Lancet 2001;357:1722-3.Education and debateGuy’s Hospital,London SE1 9RTTom Treasureprofessor ofcardiothoracic surgery421BMJ VOLUME 324 16 FEBRUARY 2002 bmj.com

http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1122341

What to do about poor clinical performance in clinical trials

Abstract

Similar works

Full text

Available Versions

University of Birmingham Research Portal

University of Birmingham Research Portal

Crossref

White Rose Research Online