Beller, Elaine

Cleo, Gina

Islam, Farhana

Julien, Blair

Scott, Anna Mae

English

Aim: New software packages help to improve the efficiency of conducting a systematic review through automation of key steps in the systematic review. The aim of this study was to gather qualitative data on the usability and acceptability of four systematic review automation software packages (Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst) for the citation screening step of a systematic review. Methods: We recruited three volunteer systematic reviewers and asked them to use allocated software packages during citation screening. They then completed a 12-item online questionnaire which was tailored to capture data for the software packages used. Findings: All four software packages were reported to be easy or very easy to learn and use. SRA-Helper for EndNote was most favoured by participants for screening citations and Covidence for resolving conflicts. Overall, participants reported that SRA-Helper for EndNote would be their software package of choice, primarily due to its efficiency. Conclusion: This study identified a number of considerations which systematic reviewers can use as a basis of their decision which software to use when performing the citation screening and dispute resolution steps of a systematic review.</p

Bond University Research Portal

Bond UniversityResearch RepositoryUsability and acceptability of four systematic review automation software packagesCleo, Gina; Scott, Anna Mae; Islam, Farhana; Julien, Blair; Beller, ElainePublished in:Systematic ReviewsDOI:10.1186/s13643-019-1069-6Published: 20/06/2019Document Version:Publisher's PDF, also known as Version of recordLink to publication in Bond University research repository.Recommended citation(APA):Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E. (2019). Usability and acceptability of four systematicreview automation software packages: A mixed method design. Systematic Reviews, 8(1), [145].https://doi.org/10.1186/s13643-019-1069-6General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.For more information, or if you believe that this document breaches copyright, please contact the Bond University research repositorycoordinator.Download date: 22 Aug 2019RESEARCH Open AccessUsability and acceptability of foursystematic review automation softwarepackages: a mixed method designGina Cleo1* , Anna Mae Scott1, Farhana Islam2, Blair Julien2 and Elaine Beller1AbstractAim: New software packages help to improve the efficiency of conducting a systematic review through automationof key steps in the systematic review. The aim of this study was to gather qualitative data on the usability andacceptability of four systematic review automation software packages (Covidence, SRA-Helper for EndNote, Rayyanand RobotAnalyst) for the citation screening step of a systematic review.Methods: We recruited three volunteer systematic reviewers and asked them to use allocated software packagesduring citation screening. They then completed a 12-item online questionnaire which was tailored to capture datafor the software packages used.Findings: All four software packages were reported to be easy or very easy to learn and use. SRA-Helper for EndNotewas most favoured by participants for screening citations and Covidence for resolving conflicts. Overall, participantsreported that SRA-Helper for EndNote would be their software package of choice, primarily due to its efficiency.Conclusion: This study identified a number of considerations which systematic reviewers can use as a basis of their decisionwhich software to use when performing the citation screening and dispute resolution steps of a systematic review.Keywords: Automation, Qualitative report, Acceptability, Usability, Software packages, Systematic Review AcceleratorBackgroundSystematic reviews are the foundation of evidence-basedpractice. Yet, despite advancements in automation ofsome of the steps of systematic reviews [1, 2], conductinga systematic review remains a largely manual process thatrequires considerable expertise, time and financial re-sources. Software packages have recently become availableto help to improve the efficiency of some of the steps ofthe systematic review process, including literature search-ing, de-duplicating of search results, screening citationsand resolving conflicts. Each software package has itsstrengths and limitations. This study aimed to, firstly, as-sess the usability (ease of use and learnability) and accept-ability (sufficient to serve the purpose for which it isintended) of the following, commonly used, systematicreview automation software packages: Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst, and sec-ondly, to identify key advantages and disadvantages ofeach, as perceived by the users.MethodsThree volunteer systematic reviewers, who were commen-cing a systematic review (882 total citation records toscreen), were recruited via email invitation from the studyauthors (GC and EB). The systematic reviewers wereasked to use allocated software packages for the title/ab-stract screening and dispute resolution steps of the sys-tematic review and to complete an online questionnairewhich was tailored to capture data for the software pack-age used. One participant was assigned to use and reviewCovidence and SRA-Helper for EndNote (BJ), and onewas assigned to Rayyan and RobotAnalyst (FI). The finalreviewer was assigned to use and review Covidence vsSRA-Helper for EndNote vs Rayyan vs RobotAnalyst(AMS). Each software was used to screen 220 or 221 refer-ences, for a total of 882 references screened.© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.* Correspondence: gcleo@bond.edu.au1Institute for Evidence-Based Healthcare, Bond University, Gold Coast,AustraliaFull list of author information is available at the end of the articleCleo et al. Systematic Reviews           (2019) 8:145 https://doi.org/10.1186/s13643-019-1069-6Covidence (www.covidence.org) is a web-based screen-ing and data extraction tool; it is one of the tools recom-mended by the Cochrane Collaboration [3]. Covidenceallows authors to import and screen citations and full-text articles, resolve conflicts, extract data using custom-isable forms and export results in various formats.SRA-Helper for EndNote (https://github.com/CREBP/EndNoteHelper) is a downloadable automation scriptwhich works as an add-on to EndNote; it is part of theSystematic Review Accelerator. SRA-Helper for EndNoteallows users to map keyboard keys (e.g. 1, 2, 3) to folders(e.g. include, exclude, background). When the user high-lights a reference (e.g. Smith 1998), and presses the key(e.g. 1), the reference automatically moves to the rele-vant folder (e.g. include).Rayyan (https://rayyan.qcri.org) is a web-based applica-tion which allows multiple authors to create and collaborateon systematic reviews. Throughout the citation screeningprocess, Rayyan offers suggestions for article inclusionbased on the authors’ prior selections [4].RobotAnalyst (www.nactem.ac.uk/robotanalyst) is aweb-based application, developed to support the citationscreening phase of systematic reviews. RobotAnalystprioritises references by relevancy predictions and up-dates the predictive model after the author makes eachscreening selection.To identify the advantages and disadvantages of eachsoftware package, we used a 12-item questionnaire whichincluded 3 Likert-scale questions and 9 free text questions(Additional file 1: Table S1). Questions 1–8 focused on theusability and acceptability of each software package; thesequestions were repeated for each software package beingassessed. Questions 9–11 were comparator questions,which assessed the user’s preference for one software toolTable 1 Sum of quantitative responses for each software packageCovidence SRA-Helper for EndNote Rayyan RobotAnalystHow easy was it to learn how to use…a 8 10 10 9How would you rate the general usability of…b 9 8 8 8How would you rate the response time of…c 6 10 7 5Total (out of a possible 30 points)d 23 28 25 22a2 = very difficult, 4 = difficult, 6 = not so difficult; 8 = easy, 10 = very easyb2 = not at all user friendly, 4 = not user friendly, 6 = slightly user friendly, 8 = fairly user friendly, 10 = very user friendlyc2 = very slow, 4 = slow, 6 =manageable, 8 = quick, 10 = very quickdThe greater the number, the more favourable the responseTable 2 Comparison between software packagesParticipant 1 Participant 2 Participant 3Covidence vs SRA-Helperfor EndNoteRayyan vs RobotAnalyst Covidence vs SRA-Helperfor EndNote vs Rayyan vsRobotAnalystWhich software package did youfind better for screening citations?Why did you make this selection?SRA-Helper for EndNote Rayyan SRA-Helper for EndNoteI am more proficient withcomputers than mostand so I prefer the moretechnical systemsBecause of the annotationsI am able to makeWeb-based tools are tooslow and do not allowmapping of keys ontodecisions (have to usethe mouse throughout)Which software package did you findbetter for resolving conflicts?Why did you make this selection?Covidence RobotAnalyst CovidenceThe automation ofCovidence reallystreamlines theprocess furtherBecause the software’s inputhelped me see things froma new perspectiveSRA-Helper for EndNoteis not particularly greatfor this—it’s a bit timeconsumingIf you were to conduct a screening ofcitations and resolving conflicts again,which of the two/four software packagesare you most likely to use? Why did youmake this selection?SRA-Helper forEndNoteRobotAnalyst SRA-Helper for EndNoteSRA-Helper forEndNote wasquickerThe continuous re-calibrationabout estimations in order tomake accurate predictions isa useful tool.Also, I enjoy knowing howmany citations I have leftto screenBecause of the ability tomap keyboard keys ontodecisions and automaticadvancing of new referenceto top of screen onceprevious reference decidedDo you have any further commentsor feedback?Nil Nil NilCleo et al. Systematic Reviews           (2019) 8:145 Page 2 of 5over another when screening citations and resolving con-flicts. We used Qualtrics (www.qualtrics.com) to dissemin-ate the questionnaires and collect the data.ResultsThe three systematic reviewers answered all of the ques-tions presented in the qualitative questionnaire.We summed the participants’ quantitative responses foreach software package as displayed in Table 1 (lowest pos-sible score = 2 points; highest possible score = 10 points).All four software packages were reported to be easy orvery easy to learn and use (Table 1). Covidence had thehighest rating for general usability, scoring 9 out of a pos-sible 10 points. This may be due to its ‘straightforwardprocess and simple layout’, and it being available online,without the need to download it, ‘making it versatile/ac-cessible when out of office’. SRA-Helper for EndNote wasrated fastest for response time (10/10), as ‘it’s notdependent on internet connection’ and RobotAnalyst theslowest (5/10). When combining scores for ease of learn-ing how to use the software, usability and response time,SRA-Helper for EndNote scored highest (28/30).SRA-Helper for EndNote was most favoured by partic-ipants for screening citations and Covidence for resolv-ing conflicts (Table 2). Overall, participants reportedthat in conducting future systematic reviews, SRA-Helper for EndNote would be their software package ofchoice, primarily due to its efficiency (Table 2).The key advantages considered relevant by systematicreviewers included visibility of already screened/yet to beTable 3 Summary of the reported advantages and disadvantages of each software packageAdvantages DisadvantagesCovidence- Easy to learn how to use- Straightforward process and simple layoutto follow, easy to use- Does not require downloading (available online),making it versatile/accessible when out of office- Countdown of screened citations- Option to highlight key words (regarding reasonfor including/excluding citation)- References yet to be screened, automaticallymove up to the top of the page- Fairly/very user friendly- Potential bugs/glitches in software- Slow to respond once user has made a judgementto include/exclude citation- No library of categorised articles (unable to viewhow many articles have been included or excluded)- User must refresh the library to update the number ofcitations yet to screen (not automatic)- User is unable to change their mind regarding includingor excluding a previous reference as the referencesdisappear once a decision is madeSRA-Helper for EndNote- Very easy to learn how to use- Offer keyboard shortcuts with the ability tomap buttons onto decisions- Very fast response time (only limited by thespeed of your computer)- No server delays as it is not dependent oninternet connection- Ability to see libraries of ‘include’, ‘exclude’ or‘maybe’, automatically populate- Intuitive layout- Efficient for screening titles/abstracts- Fairly user friendly- Requires downloading and installing- Bugs/crashing- May take time to learn how to use keyboard shortcutsand to create folders/map keys onto decisionsRayyan- Very easy to learn how to use- Intuitive interface- Ability to highlight included keywords ingreen and excluded in red- Fairly user friendly- Ability to tag notes/labels to articles- Manageable/quick response rate- Glitches in software causing delaysRobotAnalyst- Easy to learn how to use and easy to use- Intuitive layout- Presents an overall summary of progress(pie graph)- Live estimations tool aids in guidingresponses to subsequent citations- Entire citation highlights the relevant colouronce decision is made (i.e. red for exclude)- Fairly user-friendly- Slow/manageable response time- Lacks ability to input and highlight included/excludedterms, which would quicken screening- Predictions are not always reliable, especially whenfew inclusions- The decision buttons (include/exclude/unsure) aresmall and too close togetherCleo et al. Systematic Reviews           (2019) 8:145 Page 3 of 5screened citations (e.g. in a form of a countdown orsummary of progress), ability to highlight key terms forinclusion and exclusion, ability to use keyboard short-cuts, fast response time of the software (e.g. between de-cision to include and the reference moving off thescreen and into the ‘included’ folder), ability to add notesor labels to the references, guidance of decision for sub-sequent references on the basis of prior decisions andease of learning how to use the tool and intuitiveness ofthe layout.The key disadvantages considered relevant by the sys-tematic reviewers included glitches in the software (e.g.crashes), slow response time of the software, inability tosee progress (references screened vs those remaining tobe screened), inability to change mind once decision toinclude/exclude was made, requirement to download orinstall software, inability to highlight included/excludedterms, unreliable predictions and poor layout (e.g. deci-sion buttons too close together). The advantages anddisadvantages of using each automation software pack-age to screen citations and resolve conflicts are sum-marised in Table 3.DiscussionOverall, the systematic reviewers found all four of the soft-ware tools easy to learn and use. SRA-Endnote helper wasstrongly preferred (28/30 points), with RobotAnalyst,Covidence and Rayyan scoring lower but similarly (22, 23and 24/30, respectively). The strong preference for SRA-Endnote Helper may be explained by its ease of learningto use and very quick response time due to it being a desk-top (rather than web-based) tool.Among the key characteristics considered relevantwere display of screening progress, ability to revise deci-sions, ability to highlight inclusion/exclusion terms, abil-ity to use keyboard shortcuts rather than the mouse,software response time and reliability (i.e. no bugs orcrashes). Users, particularly those newer to systematicreviews, also cited the intuitiveness of the layout andease of learning how to use the tool as important.It is worth emphasising here that what is consideredan advantage or disadvantage will vary by systematic re-viewer: one may prefer a web-based tool that is a bitslower in response time but available anywhere withouta download or install, whilst another may prefer a down-loadable tool that is faster in response time but requiresa download or install. However, whilst the individualpreferences will vary, our aim here was to identify whatthose key considerations are—to help systematic re-viewers (particularly those new to these tools) to maketheir own decisions which to use.Although the sample of systematic reviewers includedin the present assessment is small (n = 3), it deliberatelyincluded both novice systematic reviewers and anexperienced systematic reviewer. We are thereforeconfident that the considerations they raised are likely tobe the issues considered relevant by the larger systematicreview population. This can only be formally measuredwith a larger sample in future research.Ongoing and future developments in the automationof screening—including automated screening and textmining—would help to increase the efficiency and re-duce human effort required [1, 2].ConclusionThe results of this qualitative report highlight multiple ad-vantages and disadvantages of automation software pack-ages for screening in systematic reviews. The outcomesshow that the SRA-Endnote helper was the preferred soft-ware due to the fast response time, user-friendly set up,intuitive layout and the fact that it does not rely on inter-net connection. This study identified a number of relevantconsiderations for systematic review software packages,and individual systematic reviewers can take this into con-sideration whilst performing the citation screening anddispute resolution steps of a systematic review.Additional fileAdditional file 1: Table S1. Qualitative questionnaire schedule.(DOCX 16 kb)AbbreviationSRA: Systematic Review AcceleratorAcknowledgementsNot applicableAuthors’ contributionsGC developed the interview schedule and analysed the data. AMS, FI and BJ testedthe software and completed the questionnaires. EB conceived and was overseeingthe study. All authors read, contributed to and approved the final manuscript.FundingGC, FI and BJ received no funding with respect for this study. AMS and EB aresupported by a NHMRC grant APP1044904 (Centre for Research Excellence inMinimising Antibiotic Resistance for Acute Respiratory Infection, CREMARA).Availability of data and materialsAll data generated or analysed during this study are included in this publishedarticle [and its supplementary information files].Ethics approval and consent to participateAs this research involved staff participating by virtue of their professional role,ethics approval was not sought.Consent for publicationNot applicableCompeting interestsEndNote Helper was developed by our research centre (Centre for Researchin Evidence-Based Practice). The authors declare that they have no othercompeting interests.Cleo et al. Systematic Reviews           (2019) 8:145 Page 4 of 5Author details1Institute for Evidence-Based Healthcare, Bond University, Gold Coast,Australia. 2Faculty of Health Sciences and Medicine, Bond University, GoldCoast, Australia.Received: 7 August 2018 Accepted: 10 June 2019References1. Tsafnat G, Glasziou P, Karystianis G, Coiera E. Automated screening ofresearch studies for systematic reviews using study characteristics. Syst Rev.2018;25:64.2. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using textmining for study identification in systematic reviews: a systematic review ofcurrent approaches. Syst Rev. 2015;4:5.3. Covidence. Cochrane Community. https://community.cochrane.org/help/tools-and-software/covidence. Accessed 3 Jul 2018.4. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web andmobile app for systematic reviews. Syst Rev. 2016;5:210.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Cleo et al. Systematic Reviews           (2019) 8:145 Page 5 of 5

Usability and acceptability of four systematic review automation software packages: A mixed method design

https://pure.bond.edu.au/ws/files/32373754/Usability_and_acceptability_of_four_systemativ_review.pdf

Usability and acceptability of four systematic review automation software packages: A mixed method design

Abstract

Similar works

Full text

Available Versions

Bond University Research Portal