This EAHIL workshop focussed on three applications of text mining to assist with screening citations for systematic reviews, and encouraged participants to discuss issues affecting their adoption. This paper outlines these applications and summarises the factors raised by participants in relation to their uptake. Key aspects to uptake include having an accepted advantage over existing approaches, coupled with training and user support

O'Mara-Eves, A

Stansfield, CM

Thomas, J

UCL Discovery

9 Journal of EAHIL 2015; Vol. 11 (3): 8-10 2015 Workshop ReportIntroductionThis workshop focussed on using text mining toassist with citation screening, which is a necessarybut time-consuming step in conducting a systematicreview. The first half of the session described theprocesses and applications, which was followed bygroup discussions on challenges of adopting thistechnology for the different applications. We werepleased to have such an interested group and the 45participants fully engaged in the discussions. Theissues they raised will feed into guidance that theEPPI-Centre is developing on when and how to usethis technology.OverviewThe citation screening process in a systematic reviewinvolves checking a citation against specific criteriato assess whether it is suitable for answering thereview’s research questions. Typically this is carriedout on the titles and abstracts of citations that havebeen identified through systematic searches, beforethe full-text documents of relevant citations areretrieved. This can be a laborious and time-consuming process, with potentially tens ofthousands of citations to be screened. Text mining has the potential to automate at leastsome of this process with potential benefitsincluding a) reducing the time spent screening; b)ranking the citations so the most relevant items areidentified early on in the screening process; and c)providing a second-check to ensure relevant studiesare not missed by human reviewers. Moreover, ifscreening time is reduced it also offers the possibilityof conducting more sensitive searching as largernumbers of citations can be “screened”. Putting allthis together, this can change the approach tosystematic reviewing. Such technology does notreduce the need for skilled information professionalsin developing search strategies, as performancerelies on good training data of a suitable sample ofrelevant and irrelevant studies. Some of theprocesses, however, currently have some limitationsand need further evaluation.Current research and opportunitiesAt the 2015 EAHIL Workshop, James Thomaspresented an overview of the technologies andevaluations of their performance, and a livedemonstration was performed on a participants’Cochrane register. We also had time for a briefsnapshot on other applications of text mining:developing search strategies and mapping (obtainingan overview of the topics in a group of citations).  Anoverview of the technology for use in systematicreviews that were discussed is shown in Gjhvsf!2/In Gjhvsf!2, the technology for term recognition andReducing systematic review workload using textmining: opportunities and pitfallsClaire Stansfield, Alison O’Mara-Eves and James ThomasEvidence for Policy and Practice Information and Coordinating (EPPI-) Centre, Social Science Research Unit, UCL Institute of Education,  University College London, London,United KingdomBeesftt!gps!dpssftqpoefodf;!EPPI-Centre, Social Science Research Unit, 18 Woburn Square, UCL Institute of Education,London, UK WC1 0NR. Email: c.stansfield@ucl.ac.ukAbstractUijt!FBIJM!xpsltipq!gpdvttfe!po!uisff!bqqmjdbujpot!pg!ufyu!njojoh!up!bttjtu!xjui!tdsffojoh!djubujpot!gps!tztufnbujdsfwjfxt-!boe!fodpvsbhfe!qbsujdjqbout!up!ejtdvtt!jttvft!bggfdujoh!uifjs!bepqujpo/!Uijt!qbqfs!pvumjoft!uiftf!bqqmjdbujpotboe!tvnnbsjtft!uif!gbdupst!sbjtfe!cz!qbsujdjqbout!jo!sfmbujpo!up!uifjs!vqublf/!Lfz!btqfdut!up!vqublf!jodmvef!ibwjohbo!bddfqufe!bewboubhf!pwfs!fyjtujoh!bqqspbdift-!dpvqmfe!xjui!usbjojoh!boe!vtfs!tvqqpsu/Key words: systematic reviews; text mining; study selection; automation. :Journal of EAHIL 2015; Vol. 11 (3): 8-10 Reducing systematic review workloadautomatic clustering are based on the corpus of textunder analysis.  In contrast, automatic classificationanalyses both text and decisions from a humanscreener so it can assign levels of relevance to thecitations. Essentially it is trained by a human on arandomised sample of citations, though it needs asufficient number of relevant and irrelevant citationsfor optimal performance. When a human continuesscreening, the automatic classifier is retrained. Aregister of studies that have previously been classifiedby humans may also provide this training data.Focusing specifically on the screening stage insystematic reviews, there are currently three mainapplications:• Tdsffojoh! qsjpsjujtbujpo/ Using text mining toprioritise the order in which items are screened.After being “trained” by a human on a subset ofknown includes and excludes, the machineidentifies words (and combinations of words)that are associated with includes and those withexcludes, and lists all studies in order oflikelihood of inclusion for the human/s to screen.• Epvcmf! tdsffojoh. The use of text mining as a“second screener”. At least one human screensthe studies and their decisions are comparedwith the include/exclude recommendations ofthe machine. The researcher can specify how anyconflicts are dealt with.• (Semi-)automatic classification. The use of textmining to eliminate studies automatically orsemi-automatically. After being “trained” by ahuman on a subset of known includes andexcludes, the machine classifies all records aseither includes or excludes.Each of these three approaches has pros and cons,which were the focus of the group discussion(summarised later in this paper). Additionally, theresearch in this area is very new and fairly small, ashighlighted in a systematic review of the methodsused in text mining for screening (1).  This reviewconcluded that text mining to prioritise the order inwhich items are screened is suitable for use in reviews,and using text mining as a “second screener” may alsobe used cautiously. Using text mining toautomatically classify studies should be consideredpromising, but its utility is not fully proven. In highlytechnical/ clinical areas, it may be used with a highdegree of confidence; but more developmental andevaluative work is needed in other disciplines. Oneopportunity being investigated is part of theCochrane Collaborations’ Transform Project.Potentially, randomised controlled trials collectedfrom crowdsourcing initiatives could be automaticallydirected to the most relevant Cochrane review groupor systematic review (2). Issues for adoptionSome of these applications raise questions onprocesses that are inherent for traditional systematicreviews. For example, is it acceptable to not screenall of the studies identified through searches? Doesthe technology perform sufficiently well to use?Does it actually save time? This is also coupled with issues on adopting newinnovations in general.  Despite being available tosystematic reviewers since 2006, text mining has notbeen widely adopted (3). Rogers (4) proposed fivecharacteristics that affect the rate of adoption ofinnovations, which might be considered to explainthe low uptake (Box 1).  At the Edinburgh workshop,the groups discussed their relative importance inuptake of text mining for screening citations.Figure 1. Pwfswjfx;!ipx!ufyu!njojoh!dbo!cf!vtfe!jo!tzt.ufnbujd!sfwjfxtBox 1. Dibsbdufsjtujdt!uibu!bggfdu!uif!sbuf!pg!bepqujpo!pgjoopwbujpot!)bebqufe!gspn!Sphfst!3114*1. Perceived relative advantage (does it appearto have benefits to the user?)2. Compatibility (is it consistent with past ex-periences and the needs/values of the user?)3. Trialability (can the user try it out in theirown work?)4. Observability (are the results of the innova-tion visible to others?)5. Complexity (is it perceived as easy to under-stand and use?)21 Journal of EAHIL 2015; Vol. 11 (3): 8-10 2015 WORKSHOP REPORTOutcomes of group discussion on text miningfor screening citationsThe overall impression from the group discussionswere that people were positive about the benefits ofthe technology and were open to exploring issues onits application. Participants expressed that perceivedrelative advantage was seen as most critical touptake. Librarians and information scientists needto be able to demonstrate advantage to thereviewers. There were concerns that using newmethods for a systematic review would also needgeneral acceptance by publishers of reviews.  Trialability was considered important, whichincludes having an understanding of the technologyto try it out without needing computerprogrammers. This was combined with specificconcerns about access to software: is it open sourceor licensed; off-the-shelf or do they have to programthemselves; and what support is needed to use it. Training was seen as essential, and the developmentof guidance on its use was welcomed. There werealso concerns about transparency: how would oneknow if a mistake had been made given that it iscomplex to understand how one has obtained theresults. There is also perhaps a need to communicatedifferences between trained automatic classificationand the relevance-ranking function that exists incommercial bibliographic databases.Other issues included concern about literature indifferent languages, misspellings and symbols. Easeof importing datasets and the appropriateness forthe topic area were also raised. One participantobserved it might mean no need for removal ofduplicates, and another participant was relieved thatit would not be the end of manual screening alltogether. ConclusionWe enjoyed discussing text mining with so manyhealth librarians and information scientists. It wasparticularly useful to discover that people weregenerally open to the use of these technologies, withcaveats related to the issues on adoption and use.Acceptance that the technologies had a relativeadvantage over existing approaches, coupled withthorough training and user support, were seen ascritical to uptake. We aim to publish guidance onusing these technologies in 2016.  DeclarationJames Thomas is co-lead of the Cochrane“Transform” project, which is implementing some ofthe technologies discussed here. He also directsdevelopment and management of EPPI-Reviewer,the EPPI-Centre’s software for systematic reviews.The research project used for illustration was fundedby the Medical Research Council, UK. The viewsand opinions expressed by authors are those of theauthors and do not necessarily reflect those of theMRC.REFERENCES1. O’Mara-Eves A, Thomas J, McNaught J, MiwaM, Ananiadou S. Using text mining for studyidentification in systematic reviews: a systematicreview of current approaches. SystematicReviews. 2015;4:5. doi:10.1186/2046-4053-4-52. Project Transform [internet] 2015 [cited 2015 Jun30] http://community.cochrane.org/transform3. Thomas JM. Diffusion of innovation in systematicreview methodology: Why is study selection notyet assisted by automation? OA Evidence-BasedMedicine. 2013 Oct;1(2):12. 4. Rogers E. Diffusion of innovations. 5th ed. NewYork, NY: Free Press; 2003.

Reducing systematic review workload using text mining: opportunities and pitfalls

http://discovery.ucl.ac.uk/1476973/1/Stansfield_reducing_systematic_review_workload.pdf

Reducing systematic review workload using text mining: opportunities and pitfalls

Abstract

Similar works

Full text

Available Versions

UCL Discovery