An Attempt to Replicate Randomized Trials of Diabetes Treatments Using a Japanese Administrative Claims and Health Checkup Database: A Feasibility Study

Abstract Background Use of real-world evidence (RWE) has been limited for evaluating effectiveness because of the lack of confidence in its reliability. Examining whether a rigorously designed observational study using real-world data (RWD) can reproduce the results of a randomized controlled trial (RCT) will provide insights into the implementation of high-quality RWE studies that can produce valid conclusions. Objective We aimed to replicate published RCTs using a Japanese claims and health checkup database and examine whether the emulated RWE studies’ results agree with those of the original RCTs. Methods We selected three RCTs on diabetes medications for replication in patients with type 2 diabetes. The study outcome was either the change or percentage change in HbA1c levels from baseline. We designed three observational studies using the RWD to mimic the critical study elements of the respective RCTs as closely as possible. We performed 1:1 propensity score nearest-neighbor matching to balance the groups for potential confounders. The differences in outcomes between the groups and their 95% confidence intervals (CIs) were calculated in each RWE study, and the results were compared with those of the RCT. Results Patient characteristics, such as age, sex, and duration of diabetes, differed between the RWE studies and RCTs. In Trial 1 emulation, the percentage changes in HbA1c levels were larger in the treatment group than in the comparator group (difference −6.21, 95% confidence interval (CI) −11.01 to −1.40). In Trial 2, the change in HbA1c level was larger in the treatment group (difference −0.01; 95% CI −0.25 to 0.23), and in Trial 3, it was smaller in the treatment group (difference 0.46; 95% CI −0.01 to 0.94). These results did not show regulatory or estimate agreement with the RCTs. Conclusions None of the three emulated RWE studies using this claims and health checkup database reproduced the same conclusions as the RCTs. These discrepancies could largely be attributed to design differences between RWE studies and RCTs, primarily due to the lack of necessary data in the database. This particular RWD source may not be the best fit for evaluating treatment effects using laboratory data as the study outcome

Validation of Algorithms to Identify Bone Metastases Using Administrative Claims Data in a Japanese Hospital

Abstract Background Validated coding algorithms are essential to generate high-quality, real-world evidence from claims data studies. Objective We aimed to evaluate the validity of the algorithms to identify patients with bone metastases using claims data from a Japanese hospital. Patients and Methods This study used administrative claims data and electronic medical records at Juntendo University Hospital from April 2017 to March 2019. We developed two candidate claims-based algorithms to detect bone metastases, one based on diagnosis codes alone (Algorithm 1) and the other based on the combination of diagnosis and imaging test codes (Algorithm 2). Of the patients identified by Algorithm 1, 100 patients were randomly sampled. Among these 100 patients, 88 patients met the conditions of Algorithm 2; further, 12 additional patients were randomly sampled from those identified by Algorithm 2, thus obtaining a total of 100 patients for Algorithm 2. They were evaluated for their true diagnosis using the patient chart review as the gold standard. The positive predictive value (PPV) was calculated to assess the accuracy of each algorithm. Results For Algorithm 1, 82 patients were analyzed after excluding 18 patients without diagnostic imaging reports. Of these, 69 patients were true positive by chart review, resulting in a PPV of 84.1% (95% confidence interval (CI) 74.5–90.6). For Algorithm 2, 92 patients were analyzed after excluding eight patients whose diagnoses were not judged by chart review. Of these, 76 patients were confirmed positive by chart review, yielding a PPV of 82.6% (95% CI 73.4–89.1). Conclusion Both claims-based algorithms yielded high PPVs of approximately 85%, with no improvement in PPV by adding imaging test conditions. The diagnosis code-based algorithm is sufficient and valid for identifying bone metastases in this Japanese hospital