Assessing Binary Measurement Systems

Binary measurement systems (BMS) are widely used in both manufacturing industry and medicine. In industry, a BMS is often used to measure various characteristics of parts and then classify them as pass or fail, according to some quality standards. Good measurement systems are essential both for problem solving (i.e., reducing the rate of defectives) and to protect customers from receiving defective products. As a result, it is desirable to assess the performance of the BMS as well as to separate the effects of the measurement system and the production process on the observed classifications. In medicine, BMSs are known as diagnostic or screening tests, and are used to detect a target condition in subjects, thus classifying them as positive or negative. Assessing the performance of a medical test is essential in quantifying the costs due to misclassification of patients, and in the future prevention of these errors. In both industry and medicine, the most commonly used characteristics to quantify the performance a BMS are the two misclassification rates, defined as the chance of passing a nonconforming (non-diseased) unit, called the consumer's risk (false positive), and the chance of failing a conforming (diseased) unit, called the producer's risk (false negative). In most assessment studies, it is also of interest to estimate the conforming (prevalence) rate, i.e. probability that a randomly selected unit is conforming (diseased). There are two main approaches for assessing the performance of a BMS. Both approaches involve measuring a number of units one or more times with the BMS. The first one, called the "gold standard" approach, requires the use of a gold-standard measurement system that can determine the state of units with no classification errors. When a gold standard does not exist, is too expensive or time-consuming, another option is to repeatedly measure units with the BMS, and then use a latent class approach to estimate the parameters of interest. In industry, for both approaches, the standard sampling plan involves randomly selecting parts from the population of manufactured parts. In this thesis, we focus on a specific context commonly found in the manufacturing industry. First, the BMS under study is nondestructive. Second, the BMS is used for 100% inspection or any kind of systematic inspection of the production yield. In this context, we are likely to have available a large number of previously passed and failed parts. Furthermore, the inspection system typically tracks the number of parts passed and failed; that is, we often have baseline data about the current pass rate, separate from the assessment study. Finally, we assume that during the time of the evaluation, the process is under statistical control and the BMS is stable. Our main goal is to investigate the effect of using sampling plans that involve random selection of parts from the available populations of previously passed and failed parts, i.e. conditional selection, on the estimation procedure and the main characteristics of the estimators. Also, we demonstrate the value of combining the additional information provided by the baseline data with those collected in the assessment study, in improving the overall estimation procedure. We also examine how the availability of baseline data and using a conditional selection sampling plan affect recommendations on the design of the assessment study. In Chapter 2, we give a summary of the existing estimation methods and sampling plans for a BMS assessment study in both industrial and medical settings, that are relevant in our context. In Chapters 3 and 4, we investigate the assessment of a BMS in the case where we assume that the misclassification rates are common for all conforming/nonconforming parts and that repeated measurements on the same part are independent, conditional on the true state of the part, i.e. conditional independence. We call models using these assumptions fixed-effects models. In Chapter 3, we look at the case where a gold standard is available, whereas in Chapter 4, we investigate the "no gold standard" case. In both cases, we show that using a conditional selection plan, along with the baseline information, substantially improves the accuracy and precision of the estimators, compared to the standard sampling plan. In Chapters 5 and 6, we investigate the case where we allow for possible variation in the misclassification rates within conforming and nonconforming parts, by proposing some new random-effects models. These models relax the fixed-effects model assumptions regarding constant misclassification rates and conditional independence. As in the previous chapters, we focus on investigating the effect of using conditional selection and baseline information on the properties of the estimators, and give study design recommendations based on our findings. In Chapter 7, we discuss other potential applications of the conditional selection plan, where the study data are augmented with the baseline information on the pass rate, especially in the context where there are multiple BMSs under investigation

Building reliable evidence from real-world data: methods, cautiousness and recommendations

Routinely stored information on healthcare utilisation in everyday clinical practice has proliferated over the past several decades. There is, however, some reluctance on the part of many health professionals to use observational data to support healthcare decisions, especially when data are derived from large databases. Challenges in conducting observational studies based on electronic databases include concern about the adequacy of study design and methods to minimise the effect of both misclassifications (in the absence of direct assessments of exposure and outcome validity) and confounding (in the absence of randomisation). This paper points out issues that may compromise the validity of such studies, and approaches to managing analytic challenges. First, strategies of sampling within a large cohort, as an alternative to analysing the full cohort, will be presented. Second, methods for controlling outcome and exposure misclassifications will be described. Third, several techniques that take into account both measured and unmeasured confounders will also be presented. Fourth, some considerations regarding random uncertainty in the framework of observational studies using healthcare utilisation data will be discussed. Finally, some recommendations for good research practice are listed in this paper. The aim is to provide researchers with a methodological framework, while commenting on the value of new techniques for more advanced users

Assessing Binary Measurement Systems Using Targeted Verification with a Gold Standard

Binary Measurement Systems (BMS) are used to classify objects into two categories. Sometimes the categories represent some intrinsically dichotomous characteristic of the object, but sometimes continuous or even multidimensional characteristics are simplified into a dichotomy. In medicine, pregnancy is the typical example of a truly dichotomous characteristic; whereas Alzheimer’s disease may be a continuous or multidimensional characteristic that one may none-the-less wish to simplify into a dichotomy in diagnosis. In both cases BMS are used to classify the patient into two categories, pregnant or not pregnant, diseased or non-diseased. Most BMS are not inerrant, they misclassify patients and these misclassifications can have very damaging consequences for the patients’ health. Therefore in the search to understand and improve the BMS being used or developed, there needs to be a formalized way of studying and judging the merits of a BMS. While BMS are used throughout society, the two main areas where they are formalized in this way are medicine and manufacturing. Medical BMS are designed to determine the presence of a disease or other medical condition. Manufacturing BMS are designed to determine whether a manufactured item meets a specified quality standard. This abstract will use language and examples typical in the medical application because this is easier to understand and relate to for most people. However most of the thesis was written with an eye to publication in journals for quality improvement and thus typically is written for that audience. There are two primary attributes of BMS that are used to judge their quality: when measuring a subject once with the BMS what is the probability of a false positive diagnosis, and what is the probability of a false negative diagnosis. In the standard statistical framework (PPDAC – Problem, Plan, Data, Analysis, and Conclusion), the problem this thesis tries to address is determining these two quantities for a BMS. It develops new plans and estimation techniques for this purpose. These plans assume that a perfect “gold standard” measurement system is available. It also assumes that it is possible to repeatedly measure a subject, and one measurement does not affect other measurements. The plans in this thesis consider reducing the number of gold standard measurements needed for a given level of precision as a primary goal. The context usually implies that there is some difficulty in using the gold standard measurement system in practice; were this not the case the gold standard could be used instead of the BMS being assessed. For example some gold standard measurement systems can only be performed on a dead patient while, the BMS being assessed is intended for a living patient. Alternately the gold standard could be very expensive because no errors are permitted. The thesis considers two scenarios; one assessing a new BMS where no information is available prior to the study and where only sampling directly from the population of subjects is possible. The second, assessing a BMS that is currently in use where some information is available prior to the study and where subjects previously classified by the BMS are available to sample from. Chapters 2 and 3 consider the first scenario, while Chapters 4 and 5 consider the second scenario. Chapter 1 gives an introduction to the assessment of BMS and a review of the academic literature relevant to this thesis. Chapter 2 considers a sequential statistical plan for assessing a BMS that introduces a new innovative design concept called Targeted Verification. Targeted Verification refers to targeting specific parts to “verify” with the gold standard based on the outcome of previous phases in the sequential plan. This plan can dramatically reduce the number of patients that need to be verified while attaining performance similar to that of plans that verify all patients and avoiding the pitfalls of plans that verify no patients. Chapter 3 develops a set of closed form estimates that avoid making subjective assumptions and thus have relevant theoretical properties but retain competitive empirical performance. Chapter 4 takes the Targeted Verification concept and adapts it to the second scenario where a BMS is currently in use. It incorporates the information that is previously available about the BMS and takes advantage of the availability of patients previously categorized by the BMS in sampling. It shows that the Targeted Verification concept is much more efficient than similar plans that would verify all subjects, and much more reliable than plans than do not use a gold standard. Chapter 5 develops a set of estimates with a design philosophy the same as that of Chapter 3. To incorporate the design elements of Chapter 4, the new estimates are no longer closed form, but still avoid making subjective assumptions. The estimates have relevant theoretical properties and competitive empirical performance. Chapter 6 summarizes and discusses the findings of the thesis. It also provides directions for future work that make use of the Targeted Verification concept

Covariate Misclassification under Covariate-Adaptive Randomization: Understanding the Impact and Method for Bias Correction

Covariate-adaptive randomization has been frequently used in randomized controlled trials (RCTs) because it can well balance prognostic factors between treatment groups. However when a subject is assigned a wrong covariate value or misplaced in a wrong cohort during the randomization procedure, it may not only impact the balancing of the covariate, but also influence the treatment assignment based on the assigned cohort. Furthermore, it is preferred that covariates that are adjusted during the randomization procedure should also be adjusted for in the primary analysis. It is not clear whether a corrected covariate value, if it could be ascertained, or a misclassified covariate value should be used for the analysis, since the covariate value is tied both to the randomization procedure and analytic model. In this research, the impact of such misclassification on the type I error rate, power for hypothesis testing for the treatment effect and estimation bias of the treatment effect is explored under covariate-adaptive randomization in Aim 1. In Aim 2, a latent class model, the Continuous-time Hidden Markov Model (CTHMM) is used to estimate the misclassification issue with respect to both the estimation of misclassification probabilities and model diagnosis. An AIC based approach, which is calculated from a modified full data likelihood, is developed to test the assumption of misclassification. In Aim 3, a two-stage analysis strategy is proposed, which combines the CTHMM and the Misclassification Simulation-Extrapolation method (MCSIMEX), to correct the estimation bias of the perfectly measured variable caused by covariate misclassification. We apply the proposed analysis strategy to data from the Interventional Management of Stroke III trial to demonstrate the two-stage model

The Impact of Measurement Error in Regression Models Using Police Recorded Crime Rates

Objectives Assess the extent to which measurement error in police recorded crime rates impact the estimates of regression models exploring the causes and consequences of crime. Methods We focus on linear models where crime rates are included either as the response or as an explanatory variable, in their original scale or log-transformed. Two measurement error mechanisms are considered, systematic errors in the form of under-recorded crime, and random errors in the form of recording inconsistencies across areas. The extent to which such measurement error mechanisms impact model parameters is demonstrated algebraically using formal notation, and graphically using simulations. Results The impact of measurement error is highly variable across different settings. Depending on the crime type, the spatial resolution, but also where and how police recorded crime rates are introduced in the model, the measurement error induced biases could range from negligible to severe, affecting even estimates from explanatory variables free of measurement error. We also demonstrate how in models where crime rates are introduced as the response variable, the impact of measurement error could be eliminated using log-transformations. Conclusions The validity of a large share of the evidence base exploring the effects and consequences of crime is put into question. In interpreting findings from the literature relying on regression models and police recorded crime rates, we urge researchers to consider the biasing effects shown here. Future studies should also anticipate the impact in their findings and employ sensitivity analysis if the expected measurement error induced bias is non-negligible

Missing data and chance variation in public reporting of cancer stage at diagnosis: Cross-sectional analysis of population-based data in England

This is the final published version. Available from Elsevier via the DOI in this record.Background: The percentage of cancer patients diagnosed at an early stage is reported publicly for geographically-defined populations corresponding to healthcare commissioning organisations in England, and linked to pay-for-performance targets. Given that stage is incompletely recorded, we investigated the extent to which this indicator reflects underlying organisational differences rather than differences in stage completeness and chance variation. Methods We used population-based data on patients diagnosed with one of ten cancer sites in 2013 (bladder, breast, colorectal, endometrial, lung, ovarian, prostate, renal, NHL, and melanoma). We assessed the degree of bias in CCG (Clinical Commissioning Group) indicators introduced by missing-is-late and complete-case specifications compared with an imputed ‘gold standard’. We estimated the Spearman-Brown (organisation-level) reliability of the complete-case specification. We assessed probable misclassification rates against current pay-for-performance targets. Results Under the missing-is-late approach, bias in estimated CCG percentage of tumours diagnosed at an early stage ranged from −2 to −30 percentage points, while bias under the complete-case approach ranged from −2 to +7 percentage points. Using an annual reporting period, indicators based on the least biased complete-case approach would have poor reliability, misclassifying 27/209 (13%) CCGs against a pay-for-performance target in current use; only half (53%) of CCGs apparently exceeding the target would be correctly classified in terms of their underlying performance. Conclusions Current public reporting schemes for cancer stage at diagnosis in England should use a complete-case specification (i.e. the number of staged cases forming the denominator) and be based on three-year reporting periods. Early stage indicators for the studied geographies should not be used in pay-for-performance schemes.Cancer Research U