Capture-mark-recapture to estimate the number of missed articles for systematic reviews in surgery

Abstract BACKGROUND: Systematic reviews are an important knowledge synthesis tool, but with new literature available each day, reviewers must balance identifying all relevant literature against timely synthesis. METHODS: This study tested capture-mark-recapture (CMR), an ecology-based technique, to estimate the total number of articles in the literature identified in a systematic review of adult trauma care quality indicators. RESULTS: The systematic review included 40 articles identified from online searches and citation references. The CMR model suggested that 3 (95% confidence interval [CI]: 0 to 6) articles were missed and the database search provided 93% (one-sided 95% CI: R83%) of known articles for inclusion in the systematic review. The search order used for identifying the articles was optimal among the 24 that could have been used. CONCLUSIONS: The CMR technique can be used in systematic reviews in surgery to estimate the closeness to capturing the total body of literature for a specific topic. Ó 2013 Elsevier Inc. All rights reserved. Systematic reviews are increasingly used in surgery to synthesize knowledge so that evidence can inform clinical practice (eg, guideline development). Exhaustive searching in multiple large bibliographic databases is time consuming and resource intensive, but it needs to be efficient so the results are presented before the information becomes outdated. However, this is difficult because the amount of literature that exists on any given topic is unknown. Capture-mark-recapture (CMR) is a technique originating in ecology that has been applied to systematic reviews of randomized controlled trials of interventions in osteoporosis, gastroenterology, and hematology 1-3 to estimate The project was supported by a Synthesis Grant (KRS-91770) from the Canadian Institutes of Health Research. Dr. Stelfox is supported by a New Investigator Award from the Canadian Institutes of Health Research and a Population Health Investigator Award from Alberta Innovates. Funding sources had no role in the design, conduct, or reporting of this study and we are unaware of any conflicts of interest. None of the authors have financial or professional conflicts of interest that would influence the conduct or reporting of this study. Drs. Stelfox, Foster, and Goldsmith had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Presented at the Cochrane Canada Symposium May 9, 2012, Winnipeg, Canada. * Corresponding author. The American Journal of Surgery (2013) 206, 439-440 the number of articles in the literature. The technique has not been evaluated in observational studies or surgical studies. We therefore tested CMR to estimate the total number of articles in the literature identified in a systematic review of adult trauma care quality indicators. We performed a scoping review to identify quality indicators for evaluating trauma care 4 and subsequently systematically reviewed the evidence. 5 Horizon estimates (estimated total population of articles) were calculated for full text review and final inclusion using the 4 step CMR technique: (1) capture an initial sample from a population of interest (eg, catch fish in lake); (2) mark the elements in the sample (eg, tag the fish); (3) release the sample back into the population (eg, release tagged fish back into lake); and (4) resample the population (catch fish from same lake to see how many are tagged). Articles found in our 1st database (MEDLINE) were marked as being retrieved from that search and compared with articles retrieved through subsequent searches (eg, Embase was the 2nd database). Articles identified through bibliography reviews were attributed to the originating electronic database. Poisson regression models were used to calculate fitted estimates of the cell counts (number of articles missed) and estimate the total horizon of articles (SAS v 9.2; SAS, Cary, NC). The results of the horizon estimate are shown in We repeated the process for articles selected for inclusion in the systematic review. The final horizon estimate was 43 (40 to 46) articles, representing a difference of 3 (0 to 6) articles between the projected total literature and the number of articles captured (40 articles). Thus, selection of articles for inclusion in the systematic review represents the capture of 93% (87% to 100%) of the estimated pool of available articles. The search order used for identifying the articles was optimal among the 24 that could have been used. CMR is a technique that can be applied to estimate the total number of relevant articles for a given topic. This study demonstrates that CMR can be successfully used for systematic reviews of observational studies in surgery. Future systematic reviews may consider including horizon estimates as possible stopping rules (eg, search until 80% of articles captured) to identify when a sufficient literature search has been completed. CMR may help improve the completeness and efficiency of systematic reviews