Using median survival in meta-analysis of experimental time-to-event data

Abstract Background Time-to-event data is frequently reported in both clinical and preclinical research spheres. Systematic review and meta-analysis is a tool that can help to identify pitfalls in preclinical research conduct and reporting that can help to improve translational efficacy. However, pooling of studies using hazard ratios (HRs) is cumbersome especially in preclinical meta-analyses including large numbers of small studies. Median survival is a much simpler metric although because of some limitations, which may not apply to preclinical data, it is generally not used in survival meta-analysis. We aimed to appraise its performance when compared with hazard ratio-based meta-analysis when pooling large numbers of small, imprecise studies. Methods We simulated a survival dataset with features representative of a typical preclinical survival meta-analysis, including with influence of a treatment and a number of covariates. We calculated individual patient data-based hazard ratios and median survival ratios (MSRs), comparing the summary statistics directly and their performance at random-effects meta-analysis. Finally, we compared their sensitivity to detect associations between treatment and influential covariates at meta-regression. Results There was an imperfect correlation between MSR and HR, although the opposing direction of treatment effects between summary statistics appeared not to be a major issue. Precision was more conservative for HR than MSR, meaning that estimates of heterogeneity were lower. There was a slight sensitivity advantage for MSR at meta-analysis and meta-regression, although power was low in all circumstances. Conclusions We believe we have validated MSR as a summary statistic for use in a meta-analysis of small, imprecise experimental survival studies—helping to increase confidence and efficiency in future reviews in this area. While assessment of study precision and therefore weighting is less reliable, MSR appears to perform favourably during meta-analysis. Sensitivity of meta-regression was low for this set of parameters, so pooling of treatments to increase sample size may be required to ensure confidence in preclinical survival meta-regressions

A Systematic Online Living Evidence Summary of experimental Alzheimer's disease research

BACKGROUND: Despite extensive investment, the development of effective treatments for Alzheimer's disease (AD) has been largely unsuccessful. To improve translation, it is crucial to ensure the quality and reproducibility of foundational evidence generated from laboratory models. Systematic reviews play a key role in providing an unbiased overview of the evidence, assessing rigour and reporting, and identifying factors that influence reproducibility. However, the sheer pace of evidence generation is prohibitive to evidence synthesis and assessment.NEW METHOD: To address these challenges, we have developed AD-SOLES, an integrated workflow of automated tools that collect, curate, and visualise the totality of evidence from in vivo experiments.RESULTS: AD-SOLES is a publicly accessible interactive dashboard aiming to surface and expose data from in vivo experiments. It summarises the latest evidence, tracks reporting quality and transparency, and allows research users to easily locate evidence relevant to their specific research question.COMPARISON WITH EXISTING METHODS: Using automated screening methodologies within AD-SOLES, systematic reviews can begin at an accelerated starting point compared to traditional approaches. Furthermore, through text-mining approaches within the full-text of publications, users can identify research of interest using specific models, outcomes, or interventions without relying on details in the title and/or abstract.CONCLUSIONS: By automating the collection, curation, and visualisation of evidence from in vivo experiments, AD-SOLES addresses the challenges posed by the rapid pace of evidence generation. AD-SOLES aims to offer guidance for research improvement, reduce research waste, highlight knowledge gaps, and support informed decision making for researchers, funders, patients, and the public.</p

Attitudes towards animal study registries and their characteristics: An online survey of three cohorts of animal researchers

OBJECTIVES: Prospective registration of animal studies has been suggested as a new measure to increase value and reduce waste in biomedical research. We sought to further explore and quantify animal researchers' attitudes and preferences regarding animal study registries (ASRs). DESIGN: Cross-sectional online survey. SETTING AND PARTICIPANTS: We conducted a survey with three different samples representing animal researchers: i) corresponding authors from journals with high Eigenfactor, ii) a random Pubmed sample and iii) members of the CAMARADES network. MAIN OUTCOME MEASURES: Perceived level of importance of different aspects of publication bias, the effect of ASRs on different aspects of research as well as the importance of different research types for being registered. RESULTS: The survey yielded responses from 413 animal researchers (response rate 7%). The respondents indicated, that some aspects of ASRs can increase administrative burden but could be outweighed by other aspects decreasing this burden. Animal researchers found it more important to register studies that involved animal species with higher levels of cognitive capabilities. The time frame for making registry entries publicly available revealed a strong heterogeneity among respondents, with the largest proportion voting for "access only after consent by the principal investigator" and the second largest proportion voting for "access immediately after registration". CONCLUSIONS: The fact that the more senior and experienced animal researchers participating in this survey clearly indicated the practical importance of publication bias and the importance of ASRs underscores the problem awareness across animal researchers and the willingness to actively engage in study registration if effective safeguards for the potential weaknesses of ASRs are put into place. To overcome the first-mover dilemma international consensus statements on how to deal with prospective registration of animal studies might be necessary for all relevant stakeholder groups including animal researchers, academic institutions, private companies, funders, regulatory agencies, and journals

A protocol for the systematic review and meta-analysis of thigmotactic behaviour in the open field test in rodent models associated with persistent pain

OBJECTIVE: Thigmotaxis is an innate predator avoidance behaviour of rodents and is enhanced when animals are under stress. It is characterised by the preference of a rodent to seek shelter, rather than expose itself to the aversive open area. The behaviour has been proposed to be a measurable construct that can address the impact of pain on rodent behaviour. This systematic review will assess whether thigmotaxis can be influenced by experimental persistent pain and attenuated by pharmacological interventions in rodents. SEARCH STRATEGY: We will conduct search on three electronic databases to identify studies in which thigmotaxis was used as an outcome measure contextualised to a rodent model associated with persistent pain. All studies published until the date of the search will be considered. SCREENING AND ANNOTATION: Two independent reviewers will screen studies based on the order of (1) titles and abstracts, and (2) full texts. DATA MANAGEMENT AND REPORTING: For meta-analysis, we will extract thigmotactic behavioural data and calculate effect sizes. Effect sizes will be combined using a random-effects model. We will assess heterogeneity and identify sources of heterogeneity. A risk-of-bias assessment will be conducted to evaluate study quality. Publication bias will be assessed using funnel plots, Egger’s regression and trim-and-fill analysis. We will also extract stimulus-evoked limb withdrawal data to assess its correlation with thigmotaxis in the same animals. The evidence obtained will provide a comprehensive understanding of the strengths and limitations of using thigmotactic outcome measure in animal pain research so that future experimental designs can be optimised. We will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines and disseminate the review findings through publication and conference presentation

Selective outcome reporting in randomised controlled trials including participants with stroke or transient ischaemic attack:A systematic review

IntroductionThe prevalence of outcome reporting bias (ORB, i.e. selective reporting according to the results observed) across primary outcomes in randomised controlled trials (RCTs) including participants with stroke or transient ischaemic attack (TIA) is unknown.Materials and methodsWe searched the Cochrane Database of Systematic Reviews on 3 February 2021 for reviews published 2008-2020 with at least one RCT of a therapeutic intervention, for participants with stroke or TIA, and a safety or efficacy outcome. We took a random sample of these RCTs and included those with a trial registry record or protocol published before reporting results. Two reviewers assessed discrepancies in outcome reporting across the trial registry record, protocol, statistical analysis plan, and publication for each RCT, using the classification system designed by the Outcome Reporting Bias in Trials group.ResultsOf 600 RCTs, we identified a trial registry record in 120 (20%), a protocol in 28 (5%), and a statistical analysis plan in 5 (1%) with 123 (21%) distinct RCTs being eligible for assessment: 110 (89%, 95% CI 83-94) were at no risk, 7 (6%, 95% CI 3-11) RCTs were at low risk, and 6 (5%, 95% CI 2-10) were at high risk of ORB.DiscussionThe prevalence of ORB in primary outcomes was low in stroke/TIA RCTs that were included in Cochrane reviews and had an identifiable trial registry record or protocol. Concerningly, we were unable to identify a trial registry record or protocol in most of our sample.ConclusionWork is needed to further reduce ORB in stroke/TIA RCTs and explore the generalisability of these findings to RCTs outside of Cochrane reviews or without a registry record or protocol, as well as to secondary outcomes

Effects of MVA85A vaccine on tuberculosis challenge in animals: systematic review

Abstract Background: The existing Bacillus Calmette-Gué rin (BCG) vaccination provides partial protection against tuberculosis (TB). The modified vaccinia ankara virus-expressing antigen 85A (MVA85A) aims to boost BCG immunity. We evaluated the animal evidence supporting the testing of MVA85A in humans. Methods: Our protocol included in vivo preclinical studies of the MVA85A booster with BCG compared with BCG alone, followed by a TB challenge. We used standard methods for systematic review of animal studies, and summarized mortality, measures of pathology and lung bacterial load. The comprehensive literature search was to September 2014. Two independent investigators assessed eligibility and performed data extraction. We assessed study quality and pooled bacteria load using random effect meta-analysis. Findings: We included eight studies in 192 animals. Three experiments were in mice, two in guinea pigs, two in macaques and one in calves. Overall, study quality was low with no randomization, baseline comparability not described and blinding not reported. For animal death (including euthanasia due to severe morbidity), studies were underpowered, and overall no benefit demonstrated. No difference was shown for lung pathology measured on an ordinal scale or bacterial load. The largest mortality trial carried out in macaques had more deaths in the MVA85A vaccine group, and was published after a trial in South Africa had started recruiting children. Conclusions: This independent assessment of the animal data does not provide evidence to support efficacy of MVA85A as a BCG booster. More rigorous conduct and reportin

A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus)

Background: The ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines are widely endorsed but compliance is limited. We sought to determine whether journal-requested completion of an ARRIVE checklist improves full compliance with the guidelines. Methods: In a randomised controlled trial, manuscripts reporting in vivo animal research submitted to PLOS ONE (March-June 2015) were randomly allocated to either requested completion of an ARRIVE checklist or current standard practice. Authors, academic editors, and peer reviewers were blinded to group allocation. Trained reviewers performed outcome adjudication in duplicate by assessing manuscripts against an operationalised version of the ARRIVE guidelines that consists 108 items. Our primary outcome was the between-group differences in the proportion of manuscripts meeting all ARRIVE guideline checklist subitems. Results: We randomised 1689 manuscripts (control: n = 844, intervention: n = 845), of which 1269 were sent for peer review and 762 (control: n = 340; intervention: n = 332) accepted for publication. No manuscript in either group achieved full compliance with the ARRIVE checklist. Details of animal husbandry (ARRIVE subitem 9b) was the only subitem to show improvements in reporting, with the proportion of compliant manuscripts rising from 52.1 to 74.1% (X 2 = 34.0, df = 1, p = 2.1 × 10-7) in the control and intervention groups, respectively. Conclusions: These results suggest that altering the editorial process to include requests for a completed ARRIVE checklist is not enough to improve compliance with the ARRIVE guidelines. Other approaches, such as more stringent editorial policies or a targeted approach on key quality items, may promote improvements in reporting

Development and uptake of an online systematic review platform: the early years of the CAMARADES Systematic Review Facility (SyRF)

Preclinical research is a vital step in the drug discovery pipeline and more generally in helping to better understand human disease aetiology and its management. Systematic reviews (SRs) can be powerful in summarising and appraising this evidence concerning a specific research question, to highlight areas of improvements, areas for further research and areas where evidence may be sufficient to take forward to other research domains, for instance clinical trial. Guidance and tools for preclinical research synthesis remain limited despite their clear utility. We aimed to create an online end-to-end platform primarily for conducting SRs of preclinical studies, that was flexible enough to support a wide variety of experimental designs, was adaptable to different research questions, would allow users to adopt emerging automated tools and support them during their review process using best practice. In this article, we introduce the Systematic Review Facility (https://syrf.org.uk), which was launched in 2016 and designed to support primarily preclinical SRs from small independent projects to large, crowdsourced projects. We discuss the architecture of the app and its features, including the opportunity to collaborate easily, to efficiently manage projects, to screen and annotate studies for important features (metadata), to extract outcome data into a secure database, and tailor these steps to each project. We introduce how we are working to leverage the use of automation tools and allow the integration of these services to accelerate and automate steps in the systematic review workflow

Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classifier

Objective: Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records. Methods: We used a systematic review of oxygen-glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies. Results: Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1.000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764). For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975). At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700. Conclusion: In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used. Our algorithm might be used as a first selection phase in in vitro systematic reviews to limit the extent of full text screening required.</p