Producing Cochrane systematic reviews—a qualitative study of current approaches and opportunities for innovation and improvement

Background: Producing high-quality, relevant systematic reviews and keeping them up to date is challenging. Cochrane is a leading provider of systematic reviews in health. For Cochrane to continue to contribute to improvements in heath, Cochrane Reviews must be rigorous, reliable and up to date. We aimed to explore existing models of Cochrane Review production and emerging opportunities to improve the efficiency and sustainability of these processes. Methods: To inform discussions about how to best achieve this, we conducted 26 interviews and an online survey with 106 respondents. Results: Respondents highlighted the importance and challenge of creating reliable, timely systematic reviews. They described the challenges and opportunities presented by current production models, and they shared what they are doing to improve review production. They particularly highlighted significant challenges with increasing complexity of review methods; difficulty keeping authors on board and on track; and the length of time required to complete the process. Strong themes emerged about the roles of authors and Review Groups, the central actors in the review production process. The results suggest that improvements to Cochrane's systematic review production models could come from improving clarity of roles and expectations, ensuring continuity and consistency of input, enabling active management of the review process, centralising some review production steps; breaking reviews into smaller "chunks", and improving approaches to building capacity of and sharing information between authors and Review Groups. Respondents noted the important role new technologies have to play in enabling these improvements. Conclusions: The findings of this study will inform the development of new Cochrane Review production models and may provide valuable data for other systematic review producers as they consider how best to produce rigorous, reliable, up-to-date reviews

Living systematic reviews: 2. Combining human and machine effort

Published online 11 September 2017New approaches to evidence synthesis, which use human effort and machine automation in mutually reinforcing ways, can enhance the feasibility and sustainability of living systematic reviews. Human effort is a scarce and valuable resource, required when automation is impossible or undesirable, and includes contributions from online communities ("crowds") as well as more conventional contributions from review authors and information specialists. Automation can assist with some systematic review tasks, including searching, eligibility assessment, identification and retrieval of full-text reports, extraction of data, and risk of bias assessment. Workflows can be developed in which human effort and machine automation can each enable the other to operate in more effective and efficient ways, offering substantial enhancement to the productivity of systematic reviews. This paper describes and discusses the potential-and limitations-of new ways of undertaking specific tasks in living systematic reviews, identifying areas where these human/machine "technologies" are already in use, and where further research and development is needed. While the context is living systematic reviews, many of these enabling technologies apply equally to standard approaches to systematic reviewing.James Thomas, Anna Noel-Storr, Iain Marshall, Byron Wallace, Steven McDonald, Chris Mavergames, Paul Glasziou, Ian Shemilt, Anneliese Synnot, Tari Turner, Julian Elliott, on behalf of the Living Systematic Review Network (Zachary Munn

Living systematic review: 1. Introduction - the why, what, when, and how

Systematic reviews are difficult to keep up to date, but failure to do so leads to a decay in review currency, accuracy, and utility. We are developing a novel approach to systematic review updating termed "Living systematic review" (LSR): systematic reviews that are continually updated, incorporating relevant new evidence as it becomes available. LSRs may be particularly important in fields where research evidence is emerging rapidly, current evidence is uncertain, and new research may change policy or practice decisions. We hypothesize that a continual approach to updating will achieve greater currency and validity, and increase the benefits to end users, with feasible resource requirements over time.Julian H. Elliott, Anneliese Synnot, Tari Turner, Mark Simmonds, Elie A. Akl, Steve McDonald, Georgia Salanti, Joerg Meerpohl, Harriet MacLehose, John Hilton, David Tovey, Ian Shemilt, James Thomas … Zachary Munn … [et al.] On behalf of the Living Systematic Review Networ

Living systematic reviews: 4. Living guideline recommendations

While it is important for the evidence supporting practice guidelines to be current, that is often not the case. The advent of living systematic reviews has made the concept of "living guidelines" realistic, with the promise to provide timely, up-to-date and high-quality guidance to target users. We define living guidelines as an optimization of the guideline development process to allow updating individual recommendations as soon as new relevant evidence becomes available. A major implication of that definition is that the unit of update is the individual recommendation and not the whole guideline. We then discuss when living guidelines are appropriate, the workflows required to support them, the collaboration between living systematic reviews and living guideline teams, the thresholds for changing recommendations, and potential approaches to publication and dissemination. The success and sustainability of the concept of living guideline will depend on those of its major pillar, the living systematic review. We conclude that guideline developers should both experiment with and research the process of living guidelines.Elie A. Akl, Joerg J. Meerpohl, Julian Elliott, Lara A. Kahale, Holger J. Schünemann … Zachary Munn … [et al.] On behalf of the Living Systematic Review Networ

Living systematic reviews: 3. Statistical methods for updating meta-analyses

A living systematic review (LSR) should keep the review current as new research evidence emerges. Any meta-analyses included in the review will also need updating as new material is identified. If the aim of the review is solely to present the best current evidence standard meta-analysis may be sufficient, provided reviewers are aware that results may change at later updates. If the review is used in a decision-making context, more caution may be needed. When using standard meta-analysis methods, the chance of incorrectly concluding that any updated meta-analysis is statistically significant when there is no effect (the type I error) increases rapidly as more updates are performed. Inaccurate estimation of any heterogeneity across studies may also lead to inappropriate conclusions. This paper considers four methods to avoid some of these statistical problems when updating meta-analyses: two methods, that is, law of the iterated logarithm and the Shuster method control primarily for inflation of type I error and two other methods, that is, trial sequential analysis and sequential meta-analysis control for type I and II errors (failing to detect a genuine effect) and take account of heterogeneity. This paper compares the methods and considers how they could be applied to LSRs.Mark Simmonds, Georgia Salanti, Joanne McKenzie, Julian Elliott … Zachary Munn … [et al.] On behalf of the Living Systematic Review Networ

Immunogenicity and reactogenicity of a 13-valent-pneumococcal conjugate vaccine administered at 2, 4, and 12 months of age: a double-blind randomized active-controlled trial

Background: A 2-, 4-, and 12-month schedule of a novel 13-valent-pneumococcal conjugate vaccine (PCV13), containing serotype 1, 3, 4, 5, 6A, 6B 7F, 9V, 14, 18C, 19A, 19F, and 23F polysaccharides individually conjugated to CRM197 was evaluated in a randomized, double-blind, controlled infant study.Methods: Two hundred eighty-six healthy infants received PCV13 or the 7-valent-pneumococcal conjugate vaccine (PCV7) at 2, 4, and 12 months of age, alongside a serogroup C meningococcal (MenC) vaccine (2 and 4 months of age), DTaP-IPV-Hib (2, 3, and 4 months), and a Hib-MenC vaccine (12 months). Specific antibody responses were assessed at age 5, 12, and 13 months.Results: At 13 months of age, &gt;97% of PCV13 recipients had pneumococcal serotype-specific serum IgG concentrations ?0.35 ?g/mL for each vaccine serotype except serotype 3 (88.2%), and at least 93% of PCV13 recipients had OPA titers ?1:8 for each serotype. At 5 months, 110/114 (96.5%) of PCV13 recipients and 100/102 (98.0%) of PCV7 recipients had serum anti-PRP (Hib) IgG concentration ?0.15 ?g/mL (difference, 1.5%; CI, ?7.1%–3.7%), while 119/120 (99.2%) and 117/118 (99.2%), respectively, had MenC serum bactericidal assay titers of ?1:8. All PCV13 recipients and 110/113 (97.3%) of PCV7 recipients had IgG concentrations against fimbrial agglutinogens of ?2.2 EU/mL; IgG concentrations for the remaining pertussis antigens were ?5 EU/mL for all participants. Local reactions and systemic events were similar in the PCV13 and PCV7 groups.Conclusions: A 2-, 4-, and 12-month course of PCV13 was immunogenic for all 13 vaccine serotypes and was well tolerated.<br/