Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier

BACKGROUND: This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies. METHODS: A ML classifier for retrieving COVID-19 research studies (the 'Cochrane COVID-19 Study Classifier') was developed using a data set of title-abstract records 'included' in, or 'excluded' from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records 'included' in, or 'excluded' from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records 'included' in, or 'excluded' from, the CCSR between the 4th and 19th of January 2021. RESULTS: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were 'included' in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were 'included' in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded). CONCLUSIONS: The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register

Facilitating Web-Based Collaboration in Evidence Synthesis (TaskExchange): Development and Analysis

Background: The conduct and publication of scientific research are increasingly open and collaborative. There is growing interest in Web-based platforms that can effectively enable global, multidisciplinary scientific teams and foster networks of scientists in areas of shared research interest. Designed to facilitate Web-based collaboration in research evidence synthesis, TaskExchange highlights the potential of these kinds of platforms.// Objective: This paper describes the development, growth, and future of TaskExchange, a Web-based platform facilitating collaboration in research evidence synthesis.// Methods: The original purpose of TaskExchange was to create a platform that connected people who needed help with their Cochrane systematic reviews (rigorous syntheses of health research) with people who had the time and expertise to help. The scope of TaskExchange has now been expanded to include other evidence synthesis tasks, including guideline development. The development of TaskExchange was initially undertaken in 5 agile development phases with substantial user engagement. In each phase, software was iteratively deployed as it was developed and tested, enabling close cycles of development and refinement.// Results: TaskExchange enables users to browse and search tasks and members by keyword or nested filters, post and respond to tasks, sign up to notification emails, and acknowledge the work of TaskExchange members. The pilot platform has been open access since August 2016, has over 2300 members, and has hosted more than 630 tasks, covering a wide range of research synthesis-related tasks. Response rates are consistently over 75%, and user feedback has been positive.// Conclusions: TaskExchange demonstrates the potential for new technologies to support Web-based collaboration in health research. Development of a relatively simple platform for peer-to-peer exchange has provided opportunities for systematic reviewers to get their reviews completed more quickly and provides an effective pathway for people to join the global health evidence community

Empirical Study of Data Sharing by Authors Publishing in PLoS Journals

Many journals now require authors share their data with other investigators, either by depositing the data in a public repository or making it freely available upon request. These policies are explicit, but remain largely untested. We sought to determine how well authors comply with such policies by requesting data from authors who had published in one of two journals with clear data sharing policies.We requested data from ten investigators who had published in either PLoS Medicine or PLoS Clinical Trials. All responses were carefully documented. In the event that we were refused data, we reminded authors of the journal's data sharing guidelines. If we did not receive a response to our initial request, a second request was made. Following the ten requests for raw data, three investigators did not respond, four authors responded and refused to share their data, two email addresses were no longer valid, and one author requested further details. A reminder of PLoS's explicit requirement that authors share data did not change the reply from the four authors who initially refused. Only one author sent an original data set.We received only one of ten raw data sets requested. This suggests that journal policies requiring data sharing do not lead to authors making their data sets available to independent investigators

Estimating the Incidence of Symptomatic Rotavirus Infections: A Systematic Review and Meta-Analysis

BACKGROUND:We conducted for the first time a systematic review, including a meta-analysis, of the incidence of symptomatic rotavirus (RV) infections, because (1) it was shown to be an influential factor in estimating the cost-effectiveness of RV vaccination, (2) multiple community-based studies assessed it prospectively, (3) previous studies indicated, inconclusively, it might be similar around the world. METHODOLOGY:Pubmed (which includes Medline) was searched for surveys assessing prospectively symptomatic (diarrheal) episodes in a general population and situation, which also reported on the number of the episodes being tested RV+ and on the persons and the time period observed. A bias assessment tool was developed and used according to Cochrane guidelines by 4 researchers with different backgrounds. Heterogeneity was explored graphically and by comparing fits of study-homogenous 'fixed effects' and -heterogeneous 'random effects' models. Data were synthesized using these models. Sensitivity analysis for uncertainty regarding data abstraction, bias assessment and included studies was performed. PRINCIPAL FINDINGS:Variability between the incidences obtained from 20 studies is unlikely to be due to study groups living in different environments (tropical versus temperate climate, slums versus middle-class suburban populations), nor due to the year the study was conducted (from 1967 to 2003). A random effects model was used to incorporate unexplained heterogeneity and resulted in a global incidence estimate of 0.31 [0.19; 0.50] symptomatic RV infections per personyear of observation for children below 2 years of age, and of 0.24 [0.17; 0.34] when excluding the extreme high value of 0.84 reported for Mayan Indians in Guatemala. Apart from the inclusion/exclusion of the latter study, results were robust. CONCLUSIONS/SIGNIFICANCE:Rather than assumptions based on an ad-hoc selection of one or two studies, these pooled estimates (together with the measure for variability between populations) should be used as an input in future cost-effectiveness analyses of RV vaccination

When and how to update systematic reviews: consensus and checklist.

Updating of systematic reviews is generally more efficient than starting all over again when new evidence emerges, but to date there has been no clear guidance on how to do this. This guidance helps authors of systematic reviews, commissioners, and editors decide when to update a systematic review, and then how to go about updating the review.This is the final version of the article. It first appeared from the BMJ Publishing Group via http://dx.doi.org/10.1136/bmj.i350

Living systematic reviews : an emerging opportunity to narrow the evidence-practice gap

The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action. Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed. We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence. Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication. Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem