Di Nunzio, Giorgio Maria

English

Padua Research Archive

Finding all the needles in a haystack: A System to estimate the costs of e-discovery and systematic reviews

Archivio istituzionale della ricerca - Università di Padova

Finding all the Needles in a HaystackA System to Estimate the Costs of e-Discovery and Systematic ReviewsGiorgio Maria Di NunzioDept. of Information Engineering, University of PaduaPadova, Italygiorgiomaria.dinunzio@unipd.itABSTRACTSystematic review and e-Discovery have a common task in whichthe objective is to findmost (if not all) of the relevant documents in acollection by means of a (semi-)manual screening of the potentiallyinteresting documents [3]. However, the high cost of e-Discoverysoftware and the management of the advanced e-Discovery mech-anism are expected to affect the growth of this market (which isexpected to reach 17.32 billion dollars by 2023). 1 Moreover, thelarge and growing number of published studies makes the taskof identifying relevant studies in systematic reviews in an unbi-ased way both complex and time consuming [4]. In this paper, wepresent an active learning system which combines different sam-pling approaches in order to estimate a 95% confidence interval ofthe number of relevant documents while taking into account themonetary costs of running the system itself.KEYWORDSProbabilistic Retrieval Models, Continuous Active LearningSYSTEM OVERVIEWIn [3], the authors proposed a jointly research agenda betweene-Discovery and systematic review by focusing on the question of“how much is enough”. For e-Discovery, this question is partiallyanswered by the effort required of additional document reviewcompared to the expected impact on legal proceedings [5]. Forsystematic reviews, there are professional guidelines that describethe application of the scientific method to uncover and minimizebias and error in the selection and treatment of studies [4]. In orderto save hours of tedious work, the Cochrane Transform Projectis now applying a machine learning process in order to analyzethousands of reports and automatically select those to include insystematic reviews. This process addresses the difficulty in findingreports of studies for inclusion in a review in a reliable way.2In this paper, we discuss some of the points of the researchagenda [3] by means of an interactive system for systematic reviewsbased on an active learning framework [2]. This system monitorsthe monetary costs of a variable thresholding approach and a mixedstrategy for sampling documents. This approach achieved a recallgreater than 95% with 25,000 documents less than the best per-forming systems during an evaluation task for technology assisted1https://www.researchandmarkets.com/research/v5kb4w2http://community.cochrane.org/help/tools-and-software/evidence-pipeline/DESIRES 2018, August 2018, Bertinoro, Italy© 2018 Copyright held by the author(s).Model UserSelect firstPick randomFeedbackTrainCollectionRankUpdatecostsFigure 1: High level overview of the system. The system iscontinuously updated with the explicit relevance feedbackof the user. Randomly picked documents are used to esti-mate the number of relevant documents in the collection.reviews in empirical medicine [1].3 In Figure 1, we show a highlevel overview: 1) the model ranks the document in the collectionand shows the top ranked document to the user; 2) the user givesthe feedback on the document (either relevant or not) and thisfeedback is used to re-train the model and re-rank the documentof the collection. The system allows to adjust the proportion ofdocuments that are sampled against those that are selected by theautomatic system in order to balance the amount of money wewant to spend to estimate the confidence intervals more accuratelyor get the most relevant information as quick as possible.REFERENCES[1] G. M. Di Nunzio. 2018. A Study of an Automatic Stopping Strategy for Tech-nologically Assisted Medical Reviews. In Proc. of ECIR 2018. Springer, 672–677.https://doi.org/10.1007/978-3-319-76941-7_61[2] G. M. Di Nunzio, M. Maistro, and F. Vezzani. 2018. A Gamified Approach to NaïveBayes Classification: A Case Study for Newswires and Systematic Medical Reviews.In Companion of the The Web Conference 2018 WWW 2018, Lyon , France, April23-27, 2018. 1139–1146. https://doi.org/10.1145/3184558.3191547[3] M. Lease, G. V. Cormack, A. T. Nguyen, T. A. Trikalinos, and B. C. Wallace. 2016.Systematic review is e-discovery in doctor’s clothing. In Proceedings of the 2ndSIGIR workshop on Medical Information Retrieval (MedIR).[4] M. Miwa, J. Thomas, A. O’Mara-Eves, and S. Ananiadou. 2014. Reducing system-atic review workload through certainty-based screening. Journal of BiomedicalInformatics 51 (2014), 242 – 253.[5] D. W. Oard, J. R. Baron, B. Hedin, D. D. Lewis, and S. Tomlinson. 2010. Evaluationof information retrieval for E-discovery. Artif. Intell. Law 18, 4 (2010), 347–386.https://doi.org/10.1007/s10506-010-9093-93https://github.com/CLEFeHealth

https://www.research.unipd.it/bitstream/11577/3302622/1/short9.pdf

Finding all the needles in a haystack: A System to estimate the costs of e-discovery and systematic reviews

Abstract

Similar works

Full text

Available Versions

Padua Research Archive

Archivio istituzionale della ricerca - Università di Padova