Use of Area Under the Curve (AUC) from Propensity Model to Estimate Accuracy of the Estimated Effect of Exposure

Objective: To investigate the relationship between the area under the Receiver Operating Characteristic curve (AUC) of the propensity model for exposure and the accuracy of the estimated effect of the exposure on the outcome of interest.Methods: A Monte Carlo simulation study was performed where multiple realizations of three binary variables: outcome, exposure of interest and a covariate were repeatedly generated from the distribution determined by the parameters of the "propensity" and "main" models and the prevalence of the exposure. "Propensity" model was a logistic regression with the exposure of interest as a dependent variable and a single covariate as an "independent" variable. "Main" model was a logistic regression with outcome as a dependent variable, exposure of interest and covariate as "independent" variables. A total of 500 simulations were performed for each considered combination of the model parameters and the prevalence of the exposure. AUC was estimated from the probabilities predicted by the propensity score model. The accuracy of the estimated effect of exposure was primarily assessed with the square root of Mean Square Error (RMSE); the fifth and ninety-fifth percentile of the empirical distribution of the estimator were used to illustrate a range of not unlikely deviations from the true value.Results: The square root of Mean Square Error of the estimated effect of exposure increases as AUC increases from 0.6 to 0.9. Varying values for parameters of the propensity score model or the main effect model does not change the direction of this trend. As the proportion of exposed subjects changes away from 0.5 the RMSE increases, but the effect of AUC on RMSE remains approximately the same. Similarly, as sample size changes from 50 to 100 or 200, the RMSE of effect estimate decreases on average, but the effect of AUC on RMSE remains approximately the same. Also, the rate of change in RMSE increases with increasing AUC; the rate is the lowest when AUC changes from 0.6 to 0.7 and is highest when AUC changes from 0.8 to 0.9.Conclusions: The AUC of the propensity score model for exposure provides a single, relatively easy to compute, and suitable for various kind of data statistic, which can be used as an important indicator of the accuracy of the estimated effect of exposure on the outcome of interest. The public health importance is that it can be considered as an alternative to the previously suggested (Rubin, 2001) simultaneous consideration of the conditions of closeness of means and variances of the propensity scores in the different exposure groups. Our simulations indicate that the estimated effect of exposure is highly unreliable if AUC of the propensity model is larger than 0.8; at the same time AUCs of less than 0.7 are not associated with any substantial increase of inaccuracy of the estimated effect of exposure

Effect of training-sample size and classification difficulty on the accuracy of genomic predictors

Introduction: As part of the MicroArray Quality Control (MAQC)-II project, this analysis examines how the choice of univariate feature-selection methods and classification algorithms may influence the performance of genomic predictors under varying degrees of prediction difficulty represented by three clinically relevant endpoints. Methods: We used gene-expression data from 230 breast cancers (grouped into training and independent validation sets), and we examined 40 predictors (five univariate feature-selection methods combined with eight different classifiers) for each of the three endpoints. Their classification performance was estimated on the training set by using two different resampling methods and compared with the accuracy observed in the independent validation set. Results: A ranking of the three classification problems was obtained, and the performance of 120 models was estimated and assessed on an independent validation set. The bootstrapping estimates were closer to the validation performance than were the cross-validation estimates. The required sample size for each endpoint was estimated, and both gene-level and pathway-level analyses were performed on the obtained models. Conclusions: We showed that genomic predictor accuracy is determined largely by an interplay between sample size and classification difficulty. Variations on univariate feature-selection methods and choice of classification algorithm have only a modest impact on predictor performance, and several statistically equally good predictors can be developed for any given classification problem

A Review of Statistical Learning Machines from ATR to DNA Microarrays: design, assessment, and advice for practitioners

Statistical Learning is the process of estimating an unknown probabilistic input-output relationship of a system using a limited number of observations; and a statistical learning machine (SLM) is the machine that learned such a process. While their roots grow deeply in Probability Theory, SLMs are ubiquitous in the modern world. Automatic Target Recognition (ATR) in military applications, Computer Aided Diagnosis (CAD) in medical imaging, DNA microarrays in Genomics, Optical Character Recognition (OCR), Speech Recognition (SR), spam email filtering, stock market prediction, etc., are few examples and applications for SLM; diverse fields but one theory. The field of Statistical Learning can be decomposed to two basic subfields, Design and Assessment. Three main groups of specializations-namely statisticians, engineers, and computer scientists (ordered ascendingly by programming capabilities and descendingly by mathematical rigor)-exist on the venue of this field and each takes its elephant bite. Exaggerated rigorous analysis of statisticians sometimes deprives them from considering new ML techniques and methods that, yet, have no "complete" mathematical theory. On the other hand, immoderate add-hoc simulations of computer scientists sometimes derive them towards unjustified and immature results. A prudent approach is needed that has the enough flexibility to utilize simulations and trials and errors without sacrificing any rigor. If this prudent attitude is necessary for this field it is necessary, as well, in other fields of Engineering.Comment: This manuscript was composed in 2006 as part of a the author's Ph.D. dissertatio

On The Smoothness of Cross-Validation-Based Estimators Of Classifier Performance

Many versions of cross-validation (CV) exist in the literature; and each version though has different variants. All are used interchangeably by many practitioners; yet, without explanation to the connection or difference among them. This article has three contributions. First, it starts by mathematical formalization of these different versions and variants that estimate the error rate and the Area Under the ROC Curve (AUC) of a classification rule, to show the connection and difference among them. Second, we prove some of their properties and prove that many variants are either redundant or "not smooth". Hence, we suggest to abandon all redundant versions and variants and only keep the leave-one-out, the $K$-fold, and the repeated $K$-fold. We show that the latter is the only among the three versions that is "smooth" and hence looks mathematically like estimating the mean performance of the classification rules. However, empirically, for the known phenomenon of "weak correlation", which we explain mathematically and experimentally, it estimates both conditional and mean performance almost with the same accuracy. Third, we conclude the article with suggesting two research points that may answer the remaining question of whether we can come up with a finalist among the three estimators: (1) a comparative study, that is much more comprehensive than those available in literature and conclude no overall winner, is needed to consider a wide range of distributions, datasets, and classifiers including complex ones obtained via the recent deep learning approach. (2) we sketch the path of deriving a rigorous method for estimating the variance of the only "smooth" version, repeated $K$-fold CV, rather than those ad-hoc methods available in the literature that ignore the covariance structure among the folds of CV.Comment: The paper is currently under review in Pattern Recognition Letters (PRL

AUC: Nonparametric Estimators and Their Smoothness

Nonparametric estimation of a statistic, in general, and of the error rate of a classification rule, in particular, from just one available dataset through resampling is well mathematically founded in the literature using several versions of bootstrap and influence function. This article first provides a concise review of this literature to establish the theoretical framework that we use to construct, in a single coherent framework, nonparametric estimators of the AUC (a two-sample statistic) other than the error rate (a one-sample statistic). In addition, the smoothness of some of these estimators is well investigated and explained. Our experiments show that the behavior of the designed AUC estimators confirms the findings of the literature for the behavior of error rate estimators in many aspects including: the weak correlation between the bootstrap-based estimators and the true conditional AUC; and the comparable accuracy of the different versions of the bootstrap estimators in terms of the RMS with little superiority of the .632+ bootstrap estimator

Image thresholding techniques for localization of sub-resolution fluorescent biomarkers

In this article, we explore adaptive global and local segmentation techniques for a lab-on-chip nutrition monitoring system (NutriChip). The experimental setup consists of Caco-2 intestinal cells that can be artificially stimulated to trigger an immune response. The eventual response is optically monitored using immunofluoresence techniques targeting toll-like receptor 2 (TLR2). Two problems of interest need to be addressed by means of image processing. First, a new cell sample must be properly classified as stimulated or not. Second, the location of the stained TLR2 must be recovered in case the sample has been stimulated. The algorithmic approach to solving these problems is based on the ability of a segmentation technique to properly segment fluorescent spots. The sample classification is based on the amount and intensity of the segmented pixels, while the various segmenting blobs provide an approximate localization of TLR2. A novel local thresholding algorithm and three well-known spot segmentation techniques are compared in this study. Quantitative assessment of these techniques based on real and synthesized data demonstrates the improved segmentation capabilities of the proposed algorithm

Uncertainty Estimation for Target Detection System Discrimination and Confidence Performance Metrics

This research uses a Bayesian framework to develop probability densities for target detection system performance metrics. The metrics include the receiver operating characteristic (ROC) curve and the confidence error generation (CEG) curve. The ROC curve is a discrimination metric that quantifies how well a detection system separates targets and non-targets, and the CEG curve indicates how well the detection system estimates its own confidence. The degree of uncertainty in these metrics is a concern that previous research has not adequately addressed. This research formulates probability densities of the metrics and characterizes their uncertainty using confidence bands. Additional statistics are obtained that verify the accuracy of the confidence bands. Methods for the generation and characterization of the probability densities of the metrics are specified and demonstrated, where the initial analysis employs beta densities to model target and non-target samples of detection system output. For given target and non-target data, given functional forms of the data densities (such as beta density forms), and given prior densities of the form parameters, the methods developed here provide exact performance metric probability densities. Computational results compare favorably with existing approaches in cases where they can be applied; in other cases the methods developed here produce results that existing approaches cannot address

Image processing on reconfigurable hardware for continuous monitoring of fluorescent biomarkers in cell cultures

Fluorescence microscopy is a widespread tool in biological research. It is the primary modality for bioimaging and empowers the study and analysis of multitudes of biological processes. It can be applied to fixed biosamples, that is samples with frozen biological features by mean of chemical linkers, or live biosamples providing useful insights on the spatio-temporal behavior of fluorescently stained biomarkers. Current fluorescent microscopy techniques use digital image sensors which are used to leverage quantitative studies instead qualitative outcomes. However, state-of-the-art techniques are not suitable for integration in small, contained and (semi-)autonomous systems. They remain costly, bulky and rather quantitatively inefficient methods for monitoring fluorescent biomarkers, which is not on par with the design constraints found in modern Lab-on-a-Chip or Point-of-Use systems requiring the use of miniaturized and integrated fluroscence microscopy. In this thesis, I summarize my research and engineering efforts in bringing an embedded image processing system capable of monitoring fluorescent biomarkers in cell cultures in a continuous and real-time manner. Three main areas related to the problem at hand were explored in the course of this work: simulation, segmentation algorithms and embedded image processing. n the area of simulation, a novel approach for generating synthetic fluorescent 2D images of cell cultures is presented. This approach is dichotomized in a first part focusing on the modeling and generation of synthetic populations of cells (i.e. cell cultures) at the level of single fluorescent biomarkers and in a second part simulating the imaging process occurring in a traditional digital fluorescent microscope to produce realistic images of the synthetic cell cultures. The objective of the proposed approach aims at providing synthetic data at will in order to test and validate image processing systems and algorithms. Various image segmentation algorithms are considered and compared for the purpose of segmenting fluorescent spots in microscopic images. The study presented in this thesis includes a novel image thresholding technique for spot extraction along with three well-known spot segmentation techniques. The comparison is undertaken on two aspects. The segmentation masks provided by the methods are used to extract further metrics related to the fluorescent signals in order to (i) evaluate how well the segmentation masks can provide data for classifying real fluorescent biological samples from negative control samples and (ii) quantitatively compare the segmentations masks based on simulated data from the previously stated simulation tool. Finally, the design of an embedded image processing system based on FPGA technologies is showcased. A semi-autonomous smart camera is conceived for the continuous monitoring of fluorescent biomarkers based on one of the segmentation methods incorporated in the previously stated comparison. Keeping the focus on the need for integration in fluorescence microscopy, the image processing core at the heart of the smart camera results from the use of a novel image processing suite; a suite of IP cores developed under the constraints dictated by the bioimaging needs of fluorescence microscopy for use in FPGA and SoC technologies. As a proof of concept, the smart camera is applied to the monitoring of the kinetics of the uptake of fluorescent silica nano-particles in cell cultures