Using Real-World Data in Health Technology Assessment (HTA) Practice:A Comparative Study of Five HTA Agencies

BACKGROUND: Reimbursement decisions are conventionally based on evidence from randomised controlled trials (RCTs), which often have high internal validity but low external validity. Real-world data (RWD) may provide complimentary evidence for relative effectiveness assessments (REAs) and cost-effectiveness assessments (CEAs). This study examines whether RWD is incorporated in health technology assessment (HTA) of melanoma drugs by European HTA agencies, as well as differences in RWD use between agencies and across time. METHODS: HTA reports published between 1 January 2011 and 31 December 2016 were retrieved from websites of agencies representing five jurisdictions: England [National Institute for Health and Care Excellence (NICE)], Scotland [Scottish Medicines Consortium (SMC)], France [Haute Autorité de santé (HAS)], Germany [Institute for Quality and Efficacy in Healthcare (IQWiG)] and The Netherlands [Zorginstituut Nederland (ZIN)]. A standardized data extraction form was used to extract information on RWD inclusion for both REAs and CEAs. RESULTS: Overall, 52 reports were retrieved, all of which contained REAs; CEAs were present in 25 of the reports. RWD was included in 28 of the 52 REAs (54%), mainly to estimate melanoma prevalence, and in 22 of the 25 (88%) CEAs, mainly to extrapolate long-term effectiveness and/or identify drug-related costs. Differences emerged between agencies regarding RWD use in REAs; the ZIN and IQWiG cited RWD for evidence on prevalence, whereas the NICE, SMC and HAS additionally cited RWD use for drug effectiveness. No visible trend for RWD use in REAs and CEAs over time was observed. CONCLUSION: In general, RWD inclusion was higher in CEAs than REAs, and was mostly used to estimate melanoma prevalence in REAs or to predict long-term effectiveness in CEAs. Differences emerged between agencies' use of RWD; however, no visible trends for RWD use over time were observed

Addressing the dichotomy between individual and societal approaches to personalised medicine in oncology

Academic, industry, regulatory leaders and patient advocates in cancer clinical research met in November 2018 at the Innovation and Biomarkers in Cancer Drug Development meeting in Brussels to address the existing dichotomy between increasing calls for personalised oncology approaches based on individual molecular profiles and the need to make resource and regulatory decisions at the societal level in differing health-care delivery systems around the globe. Novel clinical trial designs, the utility and limitations of real-world evidence (RWE) and emerging technologies for profiling patient tumours and tumour-derived DNA in plasma were discussed. While randomised clinical trials remain the gold standard approach to defining clinical utility of local and systemic therapeutic interventions, the broader adoption of comprehensive tumour profiling and novel trial designs coupled with RWE may allow patient and physician autonomy to be appropriately balanced with broader assessments of safety and overall societal benefit. (C) 2019 Published by Elsevier Ltd

Treatment effects may remain the same even when trial participants differed from the target population

Objective RCTs have been criticised for lacking external validity. We assessed whether a trial in people with type I diabetes mellitus (T1DM) mirrored the wider population, and applied sample-weighting methods to assess the impact of differences on our trial's findings. Study design and setting The REPOSE trial was nested within a large UK cohort capturing demographic, clinical and quality of life (QoL) data for people with T1DM undergoing structured diabetes-specific education. We firstly assessed whether our RCT participants were comparable to this cohort using propensity score modelling. Following this we re-weighted the trial population to better match the wider cohort and re-estimated the treatment effect. Results Trial participants differed from the cohort in regards to sex, weight, HbA1c and also QoL and satisfaction with current treatment. Nevertheless, the treatment effects derived from alternative model weightings were similar to that of the original RCT. Conclusions Our RCT participants differed in composition to the wider population but the original findings were unaffected by sampling adjustments. We encourage investigators take steps to address criticisms of generalisability, but doing so is problematic: external data, even if available, may contain limited information and analyses can be susceptible to model misspecification

Implementation of conditional reimbursement schemes in HTA practice: Experiences from the Netherlands

Objectives: In 2007, the National Healthcare Institute (ZIN) initiated conditional financing (CF) of expensive hospital drugs as an example of conditional reimbursement schemes (CRS). CF is a 4-year procedure encompassing initial HTA assessment (T= 0) followed by additional data collection via outcomes research (separately assessing appropriate use & cost effectiveness in routine practice) and re-assessment (T= 4). This study aims to review performance and experiences with CF in the Netherlands to date. Methods: All dossiers for drugs that underwent the full CF procedure were reviewed. Using a standardised data abstraction form, 2 researchers independently extracted information on procedural and methodological aspects (i.e. related to implemented outcomes research & evidence assessment). A scoring algorithm was used to assess both aspects. Results: Forty-seven candidates were nominated for CF; 44 underwent T= 0 assessments and 10 T= 4 assessments. The CF procedure extended beyond 4 years for 8 of the 10 candidates. For the 10 candidates, applicants clearly defined study designs and data collection methods for outcomes research proposals addressing 14 of 20 research questions posed in T= 0 reports. ZIN provided discussion points and recommendations regarding research proposals for 16 of 20 research questions. Applicants implemented recommendations fully in 2 cases and partially in 14. Sufficient data was available at T= 4 to answer 13 of 20 research questions posed at T= 0. However, discussion points remained regarding implemented outcomes research for all 10 candidates at T= 4. ZIN advised to continue reimbursement for 6 candidates and to stop reimbursement for 2. In 2 of the 6 candidates, reimbursement was continued on the condition of additional evidence generation beyond T= 4. Two candidates did not receive a final recommendation. Conclusions: Theoretically, CF provides a valuable option for enabling quick but conditional access to medicines in the Netherlands. However, procedural and methodological aspects related to scheme design and implementation may affect its value in decision-making practice

Implementation of conditional reimbursement schemes in HTA practice: Experiences from the Netherlands

Objectives: In 2007, the National Healthcare Institute (ZIN) initiated conditional financing (CF) of expensive hospital drugs as an example of conditional reimbursement schemes (CRS). CF is a 4-year procedure encompassing initial HTA assessment (T= 0) followed by additional data collection via outcomes research (separately assessing appropriate use & cost effectiveness in routine practice) and re-assessment (T= 4). This study aims to review performance and experiences with CF in the Netherlands to date. Methods: All dossiers for drugs that underwent the full CF procedure were reviewed. Using a standardised data abstraction form, 2 researchers independently extracted information on procedural and methodological aspects (i.e. related to implemented outcomes research & evidence assessment). A scoring algorithm was used to assess both aspects. Results: Forty-seven candidates were nominated for CF; 44 underwent T= 0 assessments and 10 T= 4 assessments. The CF procedure extended beyond 4 years for 8 of the 10 candidates. For the 10 candidates, applicants clearly defined study designs and data collection methods for outcomes research proposals addressing 14 of 20 research questions posed in T= 0 reports. ZIN provided discussion points and recommendations regarding research proposals for 16 of 20 research questions. Applicants implemented recommendations fully in 2 cases and partially in 14. Sufficient data was available at T= 4 to answer 13 of 20 research questions posed at T= 0. However, discussion points remained regarding implemented outcomes research for all 10 candidates at T= 4. ZIN advised to continue reimbursement for 6 candidates and to stop reimbursement for 2. In 2 of the 6 candidates, reimbursement was continued on the condition of additional evidence generation beyond T= 4. Two candidates did not receive a final recommendation. Conclusions: Theoretically, CF provides a valuable option for enabling quick but conditional access to medicines in the Netherlands. However, procedural and methodological aspects related to scheme design and implementation may affect its value in decision-making practice

Implementing managed entry agreements in practice: The Dutch reality check

Background: Conditional financing (CF) of expensive hospital drugs was applied in the Netherlands between 2006 and 2012; a 4-year coverage with evidence development (CED) framework for expensive hospital drugs. This study aims to evaluate the CF framework, focusing on Health Technology Assessment (HTA) procedures. Methods: Using a standardised data extraction form, researchers independently extracted information on procedural, methodological and decision-making aspects from HTA reports of drugs selected for CF. Results: Forty-nine drugs were chosen for CF, of which 12 underwent the full procedure. The procedure extended beyond the envisioned 4 years period for 11/12 drugs. Outcomes research studies conducted as part of CF provided insufficient scientific data to reach conclusions on appropriate use and cost-effectiveness of 5/12 drugs. After re-assessment, continuation of reimbursement was advised for 10/12 drugs, with 6 necessitating yet additional conditions for evidence generation. Notably, advice to discontinue reimbursement for 2/12 drugs has not yet been implemented in Dutch healthcare practice. Conclusions: Theoretically, CF provided an option for quick but conditional access to drugs. However, numerous aspects related to the design and implementation of CF negatively affected its value in practice. Future CED schemes should aim to incorporate learnings from the CF example to increase their impact in healthcare practice. (C) 2018 Elsevier B.V. All rights reserved

Enhancing Bug Localization Using Phase-Based Approach

Software bug localization is an important step in the software maintenance process. Automatic bug localization can reduce the time consumed in the process of localization. Some techniques are applied in the bug localization process, but those techniques suffer from limitations in time and accuracy. This paper proposes a phase-based bug localization approach to overcome these limitations. The approach is composed of three main phases which are raw data preparation, package classification, and source code recommendation. The main input to our approach is a bug report and the source code of the past versions for the target system of interest. From the bug report, various information is utilized: the summary, the description, the stack traces, and the fixed source code files. The raw data preparation phase is used to restructure those inputs. The package classification phase aims to locate the package that would include the source code to be modified as a first step, hence reducing the time needed to locate the source code file due to the lexical mismatch between those files and the bug report data. Bidirectional Encoder Representations from Transformers (BERT), which is a sentence embedding technique, is utilized in the package classification and source code recommendation phases. The experimental results show that our approach outperforms existing approaches according to TOP-N rank and Mean Reciprocal Rank (MRR) evaluation metrics