A comparison of methods to assess diagnostic performance when using imperfect reference standards

Ph. D. Thesis.Background: Estimating the diagnostic accuracy (sensitivity and specificity) of a new medical test in the absence of a gold standard or perfect reference standard is a common problem in diagnostic accuracy studies. Failing to correct for this imperfection risks under- or overestimating the accuracy measures of the index test. Aim: To identify and compare methods employed to evaluate the diagnostic accuracy of medical tests in the absence of a gold standard. Methodology: A systematic review was conducted to identify methods employed to evaluate the diagnostic accuracy in the absence of a gold standard. Promising correction methods and latent class models were explored and compared using simulation studies and clinical datasets. Results: The methods identified from the systematic review were classified into four main groups: methods employed when there is a missing gold standard; when there are multiple imperfect reference standards; correction methods; and other methods such as the test positivity rate. Following the simulation studies undertaken to compare the correction methods, the Staquet et al method was found to outperform the Brenner method. Investigation of the latent class models alongside the analysis of a clinical dataset indicates that the assumptions made on the tests being evaluated affect the estimates obtained and clinical decisions. Given three conditionally dependent tests, the fixed effect model and random effect model via logit link tended to be preferred to the finite mixture model and random effect model via probit link because they are less impacted by the choice of priors. Conclusion: Many methods have been developed to estimate the diagnostic accuracy of a medical test in the absence of a gold standard. The choice of method employed depends on the varying assumptions or characteristics of the tests under investigation as this can affect the estimates obtained and the decisions made in practice.Newcastle University Research Excellence Award, the School of Mathematics, Statistics and Physics, the Health Economics Group in the Population Health Sciences Institute (previously known as the Institute of Health and Society), and the National Institute for Health Research, Newcastle In-Vitro Diagnostics Co-operativ

Flexible regression models for ROC and risk analysis, with or without a gold standard

A novel semiparametric regression model is developed for evaluating the covariate-specific accuracy of a continuous medical test or biomarker. Ideally, studies designed to estimate or compare medical test accuracy will use a separate, flawless gold-standard procedure to determine the true disease status of sampled individuals. We treat this as a special case of the more complicated and increasingly common scenario in which disease status is unknown because a gold-standard procedure does not exist or is too costly or invasive for widespread use. To compensate for missing data on disease status, covariate information is used to discriminate between diseased and healthy units. We thus model the probability of disease as a function of 'disease covariates'. In addition, we model test/biomarker outcome data to depend on 'test covariates', which provides researchers the opportunity to quantify the impact of covariates on the accuracy of a medical test. We further model the distributions of test outcomes using flexible semiparametric classes. An important new theoretical result demonstrating model identifiability under mild conditions is presented. The modeling framework can be used to obtain inferences about covariate-specific test accuracy and the probability of disease based on subject-specific disease and test covariate information. The value of the model is illustrated using multiple simulation studies and data on the age-adjusted ability of soluble epidermal growth factor receptor - a ubiquitous serum protein - to serve as a biomarker of lung cancer in men. SAS code for fitting the model is provided. Copyright © 2015 John Wiley & Sons, Ltd

Untangling hotel industry’s inefficiency: An SFA approach applied to a renowned Portuguese hotel chain

The present paper explores the technical efficiency of four hotels from Teixeira Duarte Group - a renowned Portuguese hotel chain. An efficiency ranking is established from these four hotel units located in Portugal using Stochastic Frontier Analysis. This methodology allows to discriminate between measurement error and systematic inefficiencies in the estimation process enabling to investigate the main inefficiency causes. Several suggestions concerning efficiency improvement are undertaken for each hotel studied.info:eu-repo/semantics/publishedVersio

Spatio temporal modeling of species distribution

The aim of this thesis is study spatial distribution of different groups from different perspectives and to analyse the different approaches to this problem. We move away from the classical approach, commonly used by ecologists, to more complex solutions, already applied in several disciplines. We are focused in applying advanced modelling techniques in order to understand species distribution and species behaviour and the relationships between them and environmental factors and have used first the most common models applied in ecology to move then to more advanced and complex perspectives. From a general perspective and comparing the different models applied during the process, from MaxEnt to spatio-temporal models with INLA, we can affirm that the models that we have developed show better results that the already built. Also, it is difficult to compare between the different approaches, but the Bayesian approach shows more flexibility and also the inclusion of spatial field or the latent spatio-temporal process allows to include residuals as a proxy for unmeasured variables. Compared with additive models with thin plate splines, probably considered one of the greatest methods to analyse species distribution models working with presence-absence data, comparable to MaxEnt, CART and MARS, our results show a better fit and more flexibility in the design. As a natural process we have realised that the Bayesian approach could be a better solution or at least a different approach for consideration. The main advantage of the Bayesian model formulation is the computational ease in model fit and prediction compared to classical geostatistical methods. To do so, instead of MCMC we have used the novel integrated nested Laplace approximation approach through the Stochastic Partial Differential Equation (SPDE) approach. The SPDE approach can be easily implemented providing results in reasonable computing time (comparing with MCMC). We showed how SPDE is a useful tool in the analysis of species distribution. This modelling could be expanded to the spatio-temporal domain by incorporating an extra term for the temporal effect, using parametric or semiparametric constructions to reflect linear, nonlinear, autoregressive or more complex behaviours. We can conclude that spatial and spatio-temporal Bayesian models are a really interesting approach for the understanding of environmental dynamics, not only because of the possibility to develop and solve more complex problems but also for the easy understanding of the implementation processes.The aim of this thesis is study spatial distribution of different groups from different perspectives and to analyse the different approaches to this problem. We move away from the classical approach, commonly used by ecologists, to more complex solutions, already applied in several disciplines. We are focused in applying advanced modelling techniques in order to understand species distribution and species behaviour and the relationships between them and environmental factors and have used first the most common models applied in ecology to move then to more advanced and complex perspectives. From a general perspective and comparing the different models applied during the process, from MaxEnt to spatio-temporal models with INLA, we can affirm that the models that we have developed show better results that the already built. Also, it is difficult to compare between the different approaches, but the Bayesian approach shows more flexibility and also the inclusion of spatial field or the latent spatio-temporal process allows to include residuals as a proxy for unmeasured variables. Compared with additive models with thin plate splines, probably considered one of the greatest methods to analyse species distribution models working with presence-absence data, comparable to MaxEnt, CART and MARS, our results show a better fit and more flexibility in the design. As a natural process we have realised that the Bayesian approach could be a better solution or at least a different approach for consideration. The main advantage of the Bayesian model formulation is the computational ease in model fit and prediction compared to classical geostatistical methods. To do so, instead of MCMC we have used the novel integrated nested Laplace approximation approach through the Stochastic Partial Differential Equation (SPDE) approach. The SPDE approach can be easily implemented providing results in reasonable computing time (comparing with MCMC). We showed how SPDE is a useful tool in the analysis of species distribution. This modelling could be expanded to the spatio-temporal domain by incorporating an extra term for the temporal effect, using parametric or semiparametric constructions to reflect linear, nonlinear, autoregressive or more complex behaviours. We can conclude that spatial and spatio-temporal Bayesian models are a really interesting approach for the understanding of environmental dynamics, not only because of the possibility to develop and solve more complex problems but also for the easy understanding of the implementation processes

Assessing Predictive Performance: From Precipitation Forecasts over the Tropics to Receiver Operating Characteristic Curves and Back

Educated decision making involves two major ingredients: probabilistic forecasts for future events or quantities and an assessment of predictive performance. This thesis focuses on the latter topic and illustrates its importance and implications from both theoretical and applied perspectives. Receiver operating characteristic (ROC) curves are key tools for the assessment of predictions for binary events. Despite their popularity and ubiquitous use, the mathematical understanding of ROC curves is still incomplete. We establish the equivalence between ROC curves and cumulative distribution functions (CDFs) on the unit interval and elucidate the crucial role of concavity in interpreting and modeling ROC curves. Under this essential requirement, the classical binormal ROC model is strongly inhibited in its flexibility and we propose the novel beta ROC model as an alternative. For a class of models that includes the binormal and the beta model, we derive the large sample distribution of the minimum distance estimator. This allows for uncertainty quantification and statistical tests of goodness-of-fit or equal predictive ability. Turning to empirical examples, we analyze the suitability of both models and find empirical evidence for the increased flexibility of the beta model. A freely available software package called betaROC is currently prepared for release for the statistical programming language R. Throughout the tropics, probabilistic forecasts for accumulated precipitation are of economic importance. However, it is largely unknown how skillful current numerical weather prediction (NWP) models are at timescales of one to a few days. For the first time, we systematically assess the quality of nine global operational NWP ensembles for three regions in northern tropical Africa, and verify against station and satellite-based observations and for the monsoon seasons 2007-2014. All examined NWP models are uncalibrated and unreliable, in particular for high probabilities of precipitation, and underperform in the prediction of amount and occurrence of precipitation when compared to a climatological reference forecast. Statistical postprocessing corrects systematic deficiencies and realizes the full potential of ensemble forecasts. Postprocessed forecasts are calibrated and reliable and outperform raw ensemble forecasts in all regions and monsoon seasons. Disappointingly however, they have predictive performance only equal to the climatological reference. This assessment is robust and holds for all examined NWP models, all monsoon seasons, accumulation periods of 1 to 5 days, and station and spatially aggregated satellite-based observations. Arguably, it implies that current NWP ensembles cannot translate information about the atmospheric state into useful information regarding occurrence or amount of precipitation. We suspect convective parameterization as likely cause of the poor performance of NWP ensemble forecasts as it has been shown to be a first-order error source for the realistic representation of organized convection in NWP models. One may ask if the poor performance of NWP ensembles is exclusively confined to northern tropical Africa or if it applies to the tropics in general. In a comprehensive study, we assess the quality of two major NWP ensemble prediction systems (EPSs) for 1 to 5-day accumulated precipitation for ten climatic regions in the tropics and the period 2009-2017. In particular, we investigate their skill regarding the occurrence and amount of precipitation as well as the occurrence of extreme events. Both ensembles exhibit clear calibration problems and are unreliable and overconfident. Nevertheless, they are (slightly) skillful for most climates when compared to the climatological reference, except tropical and northern arid Africa and alpine climates. Statistical postprocessing corrects for the lack of calibration and reliability, and improves forecast quality. Postprocessed ensemble forecasts are skillful for most regions except the above mentioned ones. The lack of NWP forecast skill in tropical and northern arid Africa and alpine climates calls for alternative approaches for the prediction of precipitation. In a pilot study for northern tropical Africa, we investigate whether it is possible to construct skillful statistical models that rely on information about recent rainfall events. We focus on the prediction of the probability of precipitation and find clear evidence for its modulation by recent precipitation events. The spatio-temporal correlation of rainfall coincides with meteorological assumptions, is reasonably pronounced and stable, and allows to construct meaningful statistical forecasts. We construct logistic regression based forecasts that are reliable, have a higher resolution than the climatological reference forecast, and yield an average improvement of 20% for northern tropical Africa and the period 1998-2014

Test-dependent sampling design and semi-parametric inference for the ROC curve

The receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) are used to describe the ability of a screening test to discriminate between diseased and non-diseased subjects. As evaluating the true disease status can be costly, researchers can increase study efficiency by allowing selection probabilities to depend on the screening test. We consider a test dependent sampling (TDS) design where TDS inclusion depends on a continuous screening test measure. Disease status is validated only for those in the SRS and TDS components. To improve efficiency, this sampling design incorporates three components: the simple random sample (SRS) component, TDS component, and the un-sampled subjects. We propose semi-parametric empirical likelihood estimators for the AUC, partial AUC, and the covariate-specific ROC curve. First, the AUC estimator allows us to summarize the ability of the screening test to distinguish between diseased and non-diseased subjects. Empirical likelihood methods are used to avoid making distributional assumptions for the screening test variable. Second, the AUC estimator is adapted to estimate partial AUC when a subset of false positive rates is more clinically relevant. Third, the covariate-specific ROC curve is estimated using a binormal model for the screening test variable. Although parametric assumptions are made for the screening test, distributional assumptions are avoided for the covariates by using empirical likelihood methods. This ROC curve estimator allows us to assess the influence covariates have on the accuracy of the diagnostic test. This cost-effective sampling design allows for a more powerful study on the same budget. Efficiency is gained in all three estimators by incorporating information from both the sampled and un-sampled portions of the population.Doctor of Philosoph

Uncertainty Estimation for Target Detection System Discrimination and Confidence Performance Metrics

This research uses a Bayesian framework to develop probability densities for target detection system performance metrics. The metrics include the receiver operating characteristic (ROC) curve and the confidence error generation (CEG) curve. The ROC curve is a discrimination metric that quantifies how well a detection system separates targets and non-targets, and the CEG curve indicates how well the detection system estimates its own confidence. The degree of uncertainty in these metrics is a concern that previous research has not adequately addressed. This research formulates probability densities of the metrics and characterizes their uncertainty using confidence bands. Additional statistics are obtained that verify the accuracy of the confidence bands. Methods for the generation and characterization of the probability densities of the metrics are specified and demonstrated, where the initial analysis employs beta densities to model target and non-target samples of detection system output. For given target and non-target data, given functional forms of the data densities (such as beta density forms), and given prior densities of the form parameters, the methods developed here provide exact performance metric probability densities. Computational results compare favorably with existing approaches in cases where they can be applied; in other cases the methods developed here produce results that existing approaches cannot address