Black, Ashly

Car, Josip

Majeed, Azeem

Sheikh, Aziz

Informatics in Primary Care (BCS, The Chartered Institute for IT)

EditorialStrategic considerations for improving thequality of eHealth research: we need toimprove the quality and capacity ofacademia to undertake informatics researchAshly Black BA MScDoctoral researcher in eHealth, eHealth Unit, Department of Primary Care and Social Medicine, ImperialCollege, London, UKJosip Car MD MSc SFM PhD DIC DSMDirector of eHealth Unit, Department of Primary Care and Social Medicine, Imperial College, London, UKAzeem Majeed MD FRCGP FFPHProfessor of Primary Care, Department of Primary Care and Social Medicine, Imperial College, London,UKAziz Sheikh BSc MSc MBBS MD FRCP FRCGP DCH DRCOGProfessor of Primary Care Research and Development, General Practice Section, Division of CommunityHealth Sciences, University of Edinburgh, Scotland, UKOur recent systematic overview of the eHealth liter-ature, in common with others, has found that a vastgap exists between the theoretical beneﬁts of eHealthtechnologies and what has been empirically demon-strated.1,2 Although the beneﬁts are promoted byenthusiasts and industry, the reality is that the strengthof the evidence base in support of the eﬀectiveness ofthese technological innovations remains weak. Thisreﬂects the paucity of experimental research in thisarea and, furthermore, the tendency for the researchthat has been conducted to be of low quality, both inrelation to methodological rigour3 and utility.4 Alsocontributing to this lack of evidence is our as yetimmature understanding of key socio-technical con-siderations relating to the design and deployment ofeHealth technologies in a safe and sustainablemanner.This lack of relevant high-quality evidence poses con-siderable diﬃculty for those responsible for makingdecisions on the commissioning of eHealth technologiesas they are under increasing pressure to base suchdecisions on a robust evidence base.England’s National Health Service (NHS) NationalProgramme for Information Technology (NPfIT) facesthe challenge of how to provide electronic patientrecords for a servicewhich ismaking substantial changesto the way that it delivers health care. Numerousopinion articles, editorials, news items and governmentand professional reports have been published aboutthe programme, placingNHSConnecting forHealth –the agency responsible for delivering NPfIT – in theunenviable position of trying to make the ‘right’ deci-sions in the absence of a secure evidence base. Given thecontinuing likely investments in technological solu-tions, it is important that this evidence vacuum israpidly ﬁlled and in order to do thiswe believe it is vitalthat higher order strategic research considerations areprioritised.Our detailed review of the literature has allowed usto identify a limited number of recent developmentsaiming to improve the quality, usefulness and trans-parency of reporting eHealth research.5–7 Notableamongst these early developments are Guidelines forBest Evaluation Practices inHealth Informatics (GEP-HI)8 – which aim to present general and practicalguidelines that provide evaluators with a set of struc-tured, comprehensive and understandable rules for bestevaluation practices – and the Statement onReportingof Evaluation Studies in Health Informatics (STARE-HI)9 – which aims to provide guidelines for writingand assessing evaluation reports in health informatics.These developments are important in that they arepromoting the establishment of consensus on whatconstitutes high-quality eHealth research and its report-ing, which is essential in strengthening the empiricalInformatics in Primary Care 2008;16:175–7 # 2008 PHCSG, British Computer SocietyA Black, J Car, A Majeed et al176eHealth evidence base which has heretofore beenrelatively weak. Developments such as GEP-HI andthe STARE-HI are therefore to be welcomed and will,we hope, prove useful to commissioners of researchand to those conducting and publishing eHealthresearch.There are a number of other methodological in-itiatives needed. One of the next logical developments,likely to be readily welcomedby peer reviewers, editorsand users of research alike, is an instrument to helpfacilitate the critical appraisal of eHealth research.Critical appraisal is an integral component of evidence-based health care, performed as part of evidencesynthesis exercises such as systematic reviews andhealth technology assessments. Experience suggest thatgeneric critical appraisal tools for primary research ineHealth, such as Standards for the Reporting of Diag-nostic Accuracy Studies (STARD)10 and Strengtheningthe Reporting of Observational Studies in Epidemi-ology (STROBE),11 are insuﬃciently sensitive to de-tect the methodological idiosyncrasies of conductingresearch in this ﬁeld.12,13 We have furthermore foundthat current critical appraisal tools for assessing thequality of secondary research, speciﬁcally systematicreviews, also have major limitations when used inrelation to the eHealth literature. As a result, we have hadto modify the Critical Appraisal Skills Programme14instrument for systematic reviews, for instance, tomore explicitly incorporate the assessment of socio-technical and contextual considerations and we hopethat this will also prove useful to colleagues undertak-ing related work in this area.1 Clearly, however, this isan area in need of further development.Once developed new tools will not improve thequality of eHealth research unless they arewidely used.Thus far, the use of these tools by eHealth researchershas been limited, even according to recently publishedreports.Why this is the case remains unknown. Someresearchers may not agree that eHealth is ‘diﬀerent’arguing that it therefore does not warrant its ownspeciﬁc tools. This does not, however, appear to be awidely held view, as demonstrated by the conclusionsof high proﬁle methodological contributions to thisﬁeld.15–19 This research has shown that eHealth re-search has often not been conducted rigorously and/orhas failed to assess some of the more contextualelements so important to understanding the nuancednature of socio-technical factors, which are particu-larly useful to allow assessment of generalisability andlocal relevance.Research toolkits for academics, commissioners ofresearch and journal editors are thus needed, in relationto both primary and secondary research techniques.Although some resources exist, such as the web-basedevaluation portal of the Working Group for Assess-ment of Health Information Systems of the EuropeanFederation for Medical Informatics,20 many researchersremain unaware of these tools. We must thereforework to ensure that both commissioners and pro-ducers of research are made aware of these resources.There is also the need to increase research capacityin this area: we recommend the creation of junior andsenior fellowship opportunities to foster interdisci-plinary expertise and the creation of multidisciplinaryresearch networks to promote methodological ad-vances.The competence and capacity to conduct clinicallyrelevant eHealth research are most likely to be realisedby the development and use of new tools, combinedwith programmes to ensure that researchers and con-sumers of researchmake use of them.Only then arewelikely to begin bridging the evidence gap betweeneHealth rhetoric and reality and start to fully realiseits potential beneﬁts for promoting the safety, qualityand eﬃciency of health care.REFERENCES1 Car J, Black A, Anandan C et al.The Impact of eHealth onthe Quality and Safety of Healthcare. 2008. www.pcpoh.bham.ac.uk/publichealth/cfhep/001.shtml2 ClampSandKeen J.TheValue of ElectronicHealthRecords.2006. www.ychi.leeds.ac.uk/ychi/res_current.aspx3 de Keizer NF and Ammenwerth E. The quality ofevidence in health informatics: how did the quality ofhealthcare IT evaluation publications develop from1982to 2005? International Journal of Medical Informatics2008;77:41–9.4 Kaplan B. Evaluating informatics applications – clinicaldecision support systems literature review. InternationalJournal of Medical Informatics 2001;64:15–37.5 Brender J. Overview of assessment methods. In: Hand-book of Evaluation Methods for Health Informatics. Aca-demic Press, 2006, pp. 61–72.NHS Connecting for Health EvaluationProgramme (CFHEP) work package: 0011The impact of eHealth on the quality and safetyof healthcareThe report provides a comprehensive overviewof fundamental informatics applications with atheoretical and empirical account of the beneﬁtsand risks associated with implementation. Add-itionally, implications for practice, policy andresearch are provided. Its multidisciplinary ap-proach is particularly well suited to health inform-aticians providing a well rounded discussion onboth the technical and psycho-social consider-ations of design, development anddeployment ofinterventions.Strategic considerations for improving the quality of eHealth research 1776 Rahimi B and Vimarlund V. Methods to evaluate healthinformation systems in healthcare settings: a literaturereview. Journal of Medical Systems 2007;3:397–432.7 Yusof MM, Papazafeiropoulou A, Paul RJ andStergioulas LK. Investigating evaluation frameworks forhealth information systems. International Journal ofMedical Informatics 2008;77:377–85.8 Ammenwerth E, Brender J, Nykanen P, Talmon J andWessel C.Guidelines for Best Evaluation Practices in HealthInformatics GEP-HI. iig.umit.at/efmi/9 Talmon J, Ammenwerth E, Brender J, de Keizer N,Nyka¨nen P and Rigby M. Statement on Reporting ofEvaluation Studies in Health Informatics STARE-HI. iig.umit.at/efmi10 STARD Statement: standards for the reporting of diagnos-tic accuracy studies. www.stard-statement.org/11 STROBE Statement: strengthening the reporting of obser-vational studies in epidemiology. www.strobe-statement.org/Checklist.html12 Ammenwerth E, Schnell-Inderst P, Machan C andSiebert U. The eﬀect of electronic prescribing on medi-cation errors and adverse drug events: a systematicreview. Journal of the American Medical InformaticsAssociation 2008;M2667.13 Liu JL, Wyatt JC, Deeks JJ et al. Systematic reviews ofclinical decision tools for acute abdominal pain. HealthTechnology Assessment 2006; 10:1–iv.14 Critical Appraisal Skills Programme. 10 Questions toHelp You Make Sense of Reviews. www.phru.nhs.uk/Pages/PHD/resources.htm15 Chuang JH, Hripcsak G and Jenders RA. Consideringclustering: a methodological review of clinical decisionsupport system studies. Proceedings of the AmericanMedical Informatics Association Annual Symposium 2000;146–50.16 Friedman CP and Abbas UL. Is medical informatics amature science? A review of measurement practice inoutcome studies of clinical systems. International Journal ofMedical Informatics 2003;69:261–72.17 Harris AD, McGregor JC, Perencevich EN et al. The useand interpretation of quasi-experimental studies inmedical informatics. Journal of the American MedicalInformatics Association 2006;13:16–23.18 Eisenstein EL, Ortiz M, Anstrom KJ, Crosslin DR andLobach DF. Economic evaluation in medical infor-mation technology: why the numbers don’t add up.Proceedings of the American Medical Informatics Associ-ation Annual Symposium 2006;914.19 Eisenstein EL, Ortiz M, Anstrom KJ, Crosslin DR andLobachDF. Assessing the quality ofmedical informationtechnology economic evaluations: room for improve-ment. Proceedings of the American Medical InformaticsAssociation Annual Symposium 2006;234–8.20 Website of theWorkingGroup for Assessment ofHealthInformation Systems of the European Federation forMedical Informatics (EFMI). iig.umit.at/efmi/ADDRESS FOR CORRESPONDENCEEmail: Aziz.Sheikh@ed.ac.ukAccepted August 2008

Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research

Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research

Abstract

Similar works

Full text

Available Versions

Informatics in Primary Care (BCS, The Chartered Institute for IT)

Directory of Open Access Journals