Monitoring metrics over time: Why clinical trialists need to systematically collect site performance metrics

Background: Over the last decade, there has been an increasing interest in risk-based monitoring (RBM) in clinical trials, resulting in a number of guidelines from regulators and its inclusion in ICH GCP. However, there is a lack of detail on how to approach RBM from a practical perspective, and insufficient understanding of best practice. Purpose: We present a method for clinical trials units to track their metrics within clinical trials using descriptive statistics and visualisations. Research Design: We suggest descriptive statistics and visualisations within a SWAT methodology. Study Sample: We illustrate this method using the metrics from TEMPER, a monitoring study carried out in three trials at the MRC Clinical Trials Unit at UCL. Data Collection: The data collection for TEMPER is described in DOI: 10.1177/1740774518793379. Results: We show the results and discuss a protocol for a Study-Within-A-Trial (SWAT 167) for those wishing to use the method. Conclusions: The potential benefits metric tracking brings to clinical trials include enhanced assessment of sites for potential corrective action, improved evaluation and contextualisation of the influence of metrics and their thresholds, and the establishment of best practice in RBM. The standardisation of the collection of such monitoring data would benefit both individual trials and the clinical trials community

Lack of transparent reporting of trial monitoring approaches in randomised controlled trials: A systematic review of contemporary protocol papers

Background: Monitoring is essential to ensure patient safety and data integrity in clinical trials as per Good Clinical Practice. The Standard Protocol Items: Recommendations for Interventional Trials Statement and its checklist guides authors to include monitoring in their protocols. We investigated how well monitoring was reported in published ‘protocol papers’ for contemporary randomised controlled trials. Methods: A systematic search was conducted in PubMed to identify eligible protocol papers published in selected journals between 1 January 2020 and 31 May 2020. Protocol papers were classified by whether they reported monitoring and, if so, by the details of monitoring. Data were summarised descriptively. Results: Of 811 protocol papers for randomised controlled trials, 386 (48%; 95% CI: 44%–51%) explicitly reported some monitoring information. Of these, 20% (77/386) reported monitoring information consistent with an on-site monitoring approach, and 39% (152/386) with central monitoring, 26% (101/386) with a mixed approach, while 14% (54/386) did not provide sufficient information to specify an approach. Only 8% (30/386) of randomised controlled trials reported complete details about all of scope, frequency and organisation of monitoring; frequency of monitoring was the least reported. However, 6% (25/386) of papers used the term ‘audit’ to describe ‘monitoring’. Discussion: Monitoring information was reported in only approximately half of the protocol papers. Suboptimal reporting of monitoring hinders the clinical community from having the full information on which to judge the validity of a trial and jeopardises the value of protocol papers and the credibility of the trial itself. Greater efforts are needed to promote the transparent reporting of monitoring to journal editors and authors

Use of NHS Digital datasets as trial data in the UK: a position paper

Background: Clinical trial teams increasingly want to make use of data from healthcare systems (“healthcare data”), particularly to enhance recruitment and follow-up of participants, to reduce time and cost, and to stop the duplication of effort. However, there is continued uncertainty of how regulators regard healthcare data used for trial purposes, in terms of provenance, quality and reliability. Objectives: There were two key objectives: First, to demonstrate the data integrity of two datasets held by NHS Digital (NHSD) that are most requested by trial teams; and second, to set out an approach by which any other healthcare systems datasets can be similarly evaluated. Method: The data lifecycles of the datasets were carefully documented, mapping the flow of data from the originating healthcare provider’s databases to NHSD warehouses and onwards to clinical trials teams. These were assessed for evidence of whether the datasets are accurate, reliable, complete, contemporaneous, and well-governed. Result: The assessment method was applied to (a) the Hospital Episode Statistics Admitted Patient Care (HES APC) dataset and (b) the Civil Registration of Deaths (CRD) dataset. This paper clearly demonstrates that their collection and management through NHSD systems ensure their integrity and reliability. The datasets are accurate representations of the data held by the originating providers (acute NHS trusts and local registrars). Conclusion: Based on these findings, the HES APC and CRD datasets satisfy the assessment criteria that demonstrate they are reliable transcribed copies of the original source data. Implications: First, these datasets can be used directly for clinical trial data, with trial teams focusing on the accuracy of algorithms and processes to identify particular outcomes rather than on the integrity of the data flow. Second, this assessment approach should be used to assess whether other healthcare systems datasets are ready to be used as transcribed copies of source data, and for data providers to take appropriate steps to redress this matter if they are not

Healthcare systems data in the context of clinical trials - A comparison of cardiovascular data from a clinical trial dataset with routinely collected data

BACKGROUND: Routinely-collected healthcare systems data (HSD) are proposed to improve the efficiency of clinical trials. A comparison was undertaken between cardiovascular (CVS) data from a clinical trial database with two HSD resources. METHODS: Protocol-defined and clinically reviewed CVS events (heart failure (HF), acute coronary syndrome (ACS), thromboembolic stroke, venous and arterial thromboembolism) were identified within the trial data. Data (using pre-specified codes) was obtained from NHS Hospital Episode Statistics (HES) and National Institute for Cardiovascular Outcomes Research (NICOR) HF and myocardial ischaemia audits for trial participants recruited in England between 2010 and 2018 who had provided consent. The primary comparison was trial data versus HES inpatient (APC) main diagnosis (Box-1). Correlations are presented with descriptive statistics and Venn diagrams. Reasons for non-correlation were explored. RESULTS: From 1200 eligible participants, 71 protocol-defined clinically reviewed CVS events were recorded in the trial database. 45 resulted in a hospital admission and therefore could have been recorded by either HES APC/ NICOR. Of these, 27/45 (60%) were recorded by HES inpatient (Box-1) with an additional 30 potential events also identified. HF and ACS were potentially recorded in all 3 datasets; trial data recorded 18, HES APC 29 and NICOR 24 events respectively. 12/18 (67%) of the HF/ACS events in the trial dataset were recorded by NICOR. CONCLUSION: Concordance between datasets was lower than anticipated and the HSD used could not straightforwardly replace current trial practices, nor directly identify protocol-defined CVS events. Further work is required to improve the quality of HSD and consider event definitions when designing clinical trials incorporating HSD

Effects of go/no-go training on food-related action tendencies, liking and choice

This is the final version. Available on open access from the Royal Society via the DOI in this recordData accessibility: All study data and analysis scripts are freely available on the Open Science Framework (https://osf.io/hz2nb/). The study protocol was preregistered prior to data collection at https://osf.io/wav8p. The data are provided in electronic supplementary material [86].Inhibitory control training effects on behaviour (e.g. 'healthier' food choices) can be driven by changes in affective evaluations of trained stimuli, and theoretical models indicate that changes in action tendencies may be a complementary mechanism. In this preregistered study, we investigated the effects of food-specific go/no-go training on action tendencies, liking and impulsive choices in healthy participants. In the training task, energy-dense foods were assigned to one of three conditions: 100% inhibition (no-go), 0% inhibition (go) or 50% inhibition (control). Automatic action tendencies and liking were measured pre- and post-training for each condition. We found that training did not lead to changes in approach bias towards trained foods (go and no-go relative to control), but we warrant caution in interpreting this finding as there are important limitations to consider for the employed approach-avoidance task. There was only anecdotal evidence for an effect on food liking, but there was evidence for contingency learning during training, and participants were on average less likely to choose a no-go food compared to a control food after training. We discuss these findings from both a methodological and theoretical standpoint and propose that the mechanisms of action behind training effects be investigated further.Economic and Social Research Council (ESRC)Biotechnology and Biological Sciences Research Council (BBSRC)European Research Council (ERC

Accessing routinely collected health data to improve clinical trials: recent experience of access.

BACKGROUND: Routinely collected electronic health records (EHRs) have the potential to enhance randomised controlled trials (RCTs) by facilitating recruitment and follow-up. Despite this, current EHR use is minimal in UK RCTs, in part due to ongoing concerns about the utility (reliability, completeness, accuracy) and accessibility of the data. The aim of this manuscript is to document the process, timelines and challenges of the application process to help improve the service both for the applicants and data holders. METHODS: This is a qualitative paper providing a descriptive narrative from one UK clinical trials unit (MRC CTU at UCL) on the experience of two trial teams' application process to access data from three large English national datasets: National Cancer Registration and Analysis Service (NCRAS), National Institute for Cardiovascular Outcomes Research (NICOR) and NHS Digital to establish themes for discussion. The underpinning reason for applying for the data was to compare EHRs with data collected through case report forms in two RCTs, Add-Aspirin (ISRCTN 74358648) and PATCH (ISRCTN 70406718). RESULTS: The Add-Aspirin trial, which had a pre-planned embedded sub-study to assess EHR, received data from NCRAS 13 months after the first application. In the PATCH trial, the decision to request data was made whilst the trial was recruiting. The study received data after 8 months from NICOR and 15 months for NHS Digital following final application submission. This concluded in May 2020. Prior to application submission, significant time and effort was needed particularly in relation to the PATCH trial where negotiations over consent and data linkage took many years. CONCLUSIONS: Our experience demonstrates that data access can be a prolonged and complex process. This is compounded if multiple data sources are required for the same project. This needs to be factored in when planning to use EHR within RCTs and is best considered prior to conception of the trial. Data holders and researchers are endeavouring to simplify and streamline the application process so that the potential of EHR can be realised for clinical trials

Human papillomavirus infection : protocol for a randomised controlled trial of imiquimod cream (5%) versus podophyllotoxin cream (0.15%), in combination with quadrivalent human papillomavirus or control vaccination in treatment and prevention of recurrence of anogenital warts (HIPvac trial)

BACKGROUND: Anogenital warts are the second most common sexually transmitted infection diagnosed in sexual health services in England. About 90% of genital warts are caused by human papillomavirus (HPV) types 6 or 11, and half of episodes diagnosed are recurrences. The best and most cost-effective treatment for patients with anogenital warts is unknown. The commonly used treatments are self-administered topical agents, podophyllotoxin (0.15% cream) or imiquimod (5% cream), or cryotherapy with liquid nitrogen. Quadrivalent HPV (qHPV) vaccination is effective in preventing infection, and disease, but whether it has any therapeutic effect is not known. METHODS AND DESIGN: To investigate the efficacy of clearance and prevention of recurrence of external anogenital warts by topical treatments, podophyllotoxin 0.15% cream or imiquimod 5% cream, in combination with a three-dose regimen of qHPV or control vaccination. 500 adult patients presenting with external anogenital warts with either a first or subsequent episode of anogenital warts will be entered into this randomised, controlled partially blinded 2 × 2 factorial trial. DISCUSSION: The trial is expected to provide the first high-quality evidence of the comparative efficacy and cost-effectiveness of the two topical treatments in current use, as well as investigate the potential benefit of HPV vaccination, in the management of anogenital warts. TRIAL REGISTRATION: The trial was registered prior to starting recruitment under the following reference numbers: International Standard Randomized Controlled Trial Number (ISRCTN) Registry - ISRCTN32729817 (registered 25 July 2014); European Union Clinical Trials Register (EudraCT) - 2013-002951-14 (registered 26 June 2013)

Imiquimod versus podophyllotoxin, with and without human papillomavirus vaccine, for anogenital warts: the HIPvac factorial RCT.

BACKGROUND: The comparative efficacy, and cost-effectiveness, of imiquimod or podophyllotoxin cream, either alone or in combination with the quadrivalent HPV vaccine (Gardasil®, Merck Sharp & Dohme Corp., Merck & Co., Inc., Whitehouse Station, NJ, USA) in the treatment and prevention of recurrence of anogenital warts is not known. OBJECTIVE: The objective was to compare the efficacy of imiquimod and podophyllotoxin creams to treat anogenital warts and to assess whether or not the addition of quadrivalent human papillomavirus vaccine increases wart clearance or prevention of recurrence. DESIGN: A randomised, controlled, multicentre, partially blinded factorial trial. Participants were randomised equally to four groups, combining either topical treatment with quadrivalent human papillomavirus vaccine or placebo. Randomisation was stratified by gender, a history of previous warts and human immunodeficiency virus status. There was an accompanying economic evaluation, conducted from the provider perspective over the trial duration. SETTING: The setting was 22 sexual health clinics in England and Wales. PARTICIPANTS: Participants were patients with a first or repeat episode of anogenital warts who had not been treated in the previous 3 months and had not previously received quadrivalent human papillomavirus vaccine. INTERVENTIONS: Participants were randomised to 5% imiquimod cream (Aldara®; Meda Pharmaceuticals, Takeley, UK) for up to 16 weeks or 0.15% podophyllotoxin cream (Warticon®; GlaxoSmithKlein plc, Brentford, UK) for 4 weeks, which was extended to up to 16 weeks if warts persisted. Participants were simultaneously randomised to quadrivalent human papillomavirus vaccine (Gardasil) or saline control at 0, 8 and 24 weeks. Cryotherapy was permitted after week 4 at the discretion of the investigator. MAIN OUTCOME MEASURES: The main outcome measures were a combined primary outcome of wart clearance at week 16 and remaining wart free at week 48. Efficacy analysis was by logistic regression with multiple imputation for missing follow-up values; economic evaluation considered the costs per quality-adjusted life-year. RESULTS: A total of 503 participants were enrolled and attended at least one follow-up visit. The mean age was 31 years, 66% of participants were male (24% of males were men who have sex with men), 50% had a previous history of warts and 2% were living with human immunodeficiency virus. For the primary outcome, the adjusted odds ratio for imiquimod cream versus podophyllotoxin cream was 0.81 (95% confidence interval 0.54 to 1.23), and for quadrivalent human papillomavirus vaccine versus placebo, the adjusted odds ratio was 1.46 (95% confidence interval 0.97 to 2.20). For the components of the primary outcome, the adjusted odds ratio for wart free at week 16 for imiquimod versus podophyllotoxin was 0.77 (95% confidence interval 0.52 to 1.14) and for quadrivalent human papillomavirus vaccine versus placebo was 1.30 (95% confidence interval 0.89 to 1.91). The adjusted odds ratio for remaining wart free at 48 weeks (in those who were wart free at week 16) for imiquimod versus podophyllotoxin was 0.98 (95% confidence interval 0.54 to 1.78) and for quadrivalent human papillomavirus vaccine versus placebo was 1.39 (95% confidence interval 0.73 to 2.63). Podophyllotoxin plus quadrivalent human papillomavirus vaccine had inconclusive cost-effectiveness compared with podophyllotoxin alone. LIMITATIONS: Hepatitis A vaccine as control was replaced by a saline placebo in a non-identical syringe, administered by someone outside the research team, for logistical reasons. Sample size was reduced from 1000 to 500 because of slow recruitment and other delays. CONCLUSIONS: A benefit of the vaccine was not demonstrated in this trial. The odds of clearance at week 16 and remaining clear at week 48 were 46% higher with vaccine, and consistent effects were seen for both wart clearance and recurrence separately, but these differences were not statistically significant. Imiquimod and podophyllotoxin creams had similar efficacy for wart clearance, but with a wide confidence interval. The trial results do not support earlier evidence of a lower recurrence with use of imiquimod than with use of podophyllotoxin. Podophyllotoxin without quadrivalent human papillomavirus vaccine is the most cost-effective strategy at the current vaccine list price. A further larger trial is needed to definitively investigate the effect of the vaccine; studies of the immune response in vaccine recipients are needed to investigate the mechanism of action. TRIAL REGISTRATION: Current Controlled Trials. Current Controlled Trials ISRCTN32729817 and EudraCT 2013-002951-14. FUNDING: This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 24, No. 47. See the NIHR Journals Library website for further project information

Getting our ducks in a row:The need for data utility comparisons of healthcare systems data for clinical trials

BACKGROUND: Better use of healthcare systems data, collected as part of interactions between patients and the healthcare system, could transform planning and conduct of randomised controlled trials. Multiple challenges to widespread use include whether healthcare systems data captures sufficiently well the data traditionally captured on case report forms. "Data Utility Comparison Studies" (DUCkS) assess the utility of healthcare systems data for RCTs by comparison to data collected by the trial. Despite their importance, there are few published UK examples of DUCkS.METHODS-AND-RESULTS: Building from ongoing and selected recent examples of UK-led DUCkS in the literature, we set out experience-based considerations for the conduct of future DUCkS. Developed through informal iterative discussions in many forums, considerations are offered for planning, protocol development, data, analysis and reporting, with comparisons at "patient-level" or "trial-level", depending on the item of interest and trial status.DISCUSSION: DUCkS could be a valuable tool in assessing where healthcare systems data can be used for trials and in which trial teams can play a leading role. There is a pressing need for trials to be more efficient in their delivery and research waste must be reduced. Trials have been making inconsistent use of healthcare systems data, not least because of an absence of evidence of utility. DUCkS can also help to identify challenges in using healthcare systems data, such as linkage (access and timing) and data quality. We encourage trial teams to incorporate and report DUCkS in trials and funders and data providers to support them.</p