Uncertainty Estimation for Target Detection System Discrimination and Confidence Performance Metrics

This research uses a Bayesian framework to develop probability densities for target detection system performance metrics. The metrics include the receiver operating characteristic (ROC) curve and the confidence error generation (CEG) curve. The ROC curve is a discrimination metric that quantifies how well a detection system separates targets and non-targets, and the CEG curve indicates how well the detection system estimates its own confidence. The degree of uncertainty in these metrics is a concern that previous research has not adequately addressed. This research formulates probability densities of the metrics and characterizes their uncertainty using confidence bands. Additional statistics are obtained that verify the accuracy of the confidence bands. Methods for the generation and characterization of the probability densities of the metrics are specified and demonstrated, where the initial analysis employs beta densities to model target and non-target samples of detection system output. For given target and non-target data, given functional forms of the data densities (such as beta density forms), and given prior densities of the form parameters, the methods developed here provide exact performance metric probability densities. Computational results compare favorably with existing approaches in cases where they can be applied; in other cases the methods developed here produce results that existing approaches cannot address

Visual and Contextual Modeling for the Detection of Repeated Mild Traumatic Brain Injury.

Currently, there is a lack of computational methods for the evaluation of mild traumatic brain injury (mTBI) from magnetic resonance imaging (MRI). Further, the development of automated analyses has been hindered by the subtle nature of mTBI abnormalities, which appear as low contrast MR regions. This paper proposes an approach that is able to detect mTBI lesions by combining both the high-level context and low-level visual information. The contextual model estimates the progression of the disease using subject information, such as the time since injury and the knowledge about the location of mTBI. The visual model utilizes texture features in MRI along with a probabilistic support vector machine to maximize the discrimination in unimodal MR images. These two models are fused to obtain a final estimate of the locations of the mTBI lesion. The models are tested using a novel rodent model of repeated mTBI dataset. The experimental results demonstrate that the fusion of both contextual and visual textural features outperforms other state-of-the-art approaches. Clinically, our approach has the potential to benefit both clinicians by speeding diagnosis and patients by improving clinical care

Aircraft state estimation using cameras and passive radar

Multiple target tracking (MTT) is a fundamental task in many application domains. It is a difficult problem to solve in general, so applications make use of domain specific and problem-specific knowledge to approach the problem by solving subtasks separately. This work puts forward a MTT framework (MTTF) which is based on the Bayesian recursive estimator (BRE). The MTTF extends a particle filter (PF) to handle the multiple targets and adds a probabilistic graphical model (PGM) data association stage to compute the mapping from detections to trackers. The MTTF was applied to the problem of passively monitoring airspace. Two applications were built: a passive radar MTT module and a comprehensive visual object tracking (VOT) system. Both applications require a solution to the MTT problem, for which the MTTF was utilized. The VOT system performed well on real data recorded at the University of Cape Town (UCT) as part of this investigation. The system was able to detect and track aircraft flying within the region of interest (ROI). The VOT system consisted of a single camera, an image processing module, the MTTF module and an evaluation module. The world coordinate frame target localization was within ±3.2 km and these results are presented on Google Earth. The image plane target localization has an average reprojection error of ±17.3 pixels. The VOT system achieved an average area under the curve value of 0.77 for all receiver operating characteristic curves. These performance figures are typical over the ±1 hr of video recordings taken from the UCT site. The passive radar application was tested on simulated data. The MTTF module was designed to connect to an existing passive radar system developed by Peralex Electronics Pty Ltd. The MTTF module estimated the number of targets in the scene and localized them within a 2D local world Cartesian coordinate system. The investigations encompass numerous areas of research as well as practical aspects of software engineering and systems design

Weighing votes in human-machine collaboration for hazard recognition: Inferring hazard perceptual threshold and decision confidence from electroencephalogram wavelets

Purpose: Human-machine collaboration is a promising strategy to improve hazard inspection. However, research on the effective integration of opinions from humans with machines for optimal group decision making is lacking. Hence, considering the benefits of a brain-computer interface (BCI) to enable intuitive commutation, this study proposes a novel method to predict human hazard response choices and decision confidence from brain activities for a superior confidence-weighted voting strategy. Methodology: First, we developed a Bayesian inference-based algorithm to ascertain the decision threshold above which a hazard is reported from human brain signals. This method was tested empirically with electroencephalogram (EEG) data collected in a laboratory setting and cross-validated using behavioral indices of the signal detection theory. Subsequently, based on numerical simulations, the decision criteria for low-, medium-, and high-confidence level differentiations characterized by parietal alpha-band EEG power were determined. Findings : The investigated hazard recognition task was described as a process of probabilistic inference involving a decision uncertainty evaluation. The results demonstrated the feasibility of EEG measurements in observing human internal representations of hazard discrimination. Moreover, the optimal criteria to differentiate between low-, medium-, and high-confidence levels were obtained by benchmarking against an optimal Bayesian observer. Originality: This research demonstrates the potential of a BCI as an effective channel for telecommunication, laying the foundation for the design of future hazard detection techniques in the collaborative human-machine systems research field

Flock-level seroprevalence against avian pneumovirus amongst uruguayan broiler chickens

The objective of this study was to estimate the true prevalence of seropositive broiler chickens against avian pneumovirus at flock-level in Uruguay, using the Rogan-Gladen estimator in conjunction with Bayesian inference. A total of 181 pooled samples (consisting of 10 individual-chicken sera each) from the study area were examined with the enzyme-linked immunosorbent assay. All individual-chicken samples in the pools were also examined with the same assay. Forty-four pools were classified as test positive, because they included at least one individual-chicken classified as positive. The estimates for the deterministic (Rogan-Gladen approach) and stochastic (Bayesian approach) true prevalence were 30.9% [95% confidence interval (CI): 26.8-35.0%] and 31.4% (95% CI: 15.4-49.5%), respectively.Facultad de Ciencias Veterinaria

Comparison of Two Gas Selection Methodologies: An Application of Bayesian Model Averaging

One goal of hyperspectral imagery analysis is the detection and characterization of plumes. Characterization includes identifying the gases in the plumes, which is a model selection problem. Two gas selection methods compared in this report are Bayesian model averaging (BMA) and minimum Akaike information criterion (AIC) stepwise regression (SR). Simulated spectral data from a three-layer radiance transfer model were used to compare the two methods. Test gases were chosen to span the types of spectra observed, which exhibit peaks ranging from broad to sharp. The size and complexity of the search libraries were varied. Background materials were chosen to either replicate a remote area of eastern Washington or feature many common background materials. For many cases, BMA and SR performed the detection task comparably in terms of the receiver operating characteristic curves. For some gases, BMA performed better than SR when the size and complexity of the search library increased. This is encouraging because we expect improved BMA performance upon incorporation of prior information on background materials and gases

Measurement uncertainty in screening immunoassays in blood establishments

No contexto dos serviços de sangue, a segurança pós‐transfusão do recetor de sangue é considerada um tópico maior. Considerando a segurança dos recetores de sangue, estes serviços deverão testar um conjunto de agentes transmissíveis com prevalência significante na população. O papel dos resultados dos testes de rastreio na validação das dádivas de sangue é crítico para a garantia da segurança pós‐transfusão. Nos países da União Europeia, o rastreio é requerido pela diretiva 2001/83/EC e nos EUA pelas “normas para bancos de sangue e serviços de transfusão”. Dada a incerteza associada aos testes de rastreio, um resultado reportado poderá ser falso. Se um resultado falso positivo não tem consequência para a segurança pós‐transfusão, já um resultado falso negativo tem um grande impacto já que existe uma probabilidade elevada de o recetor do componente sanguíneo ser infetado. Consequentemente, os laboratórios de rastreio devem considerar a avaliação da incerteza de medição nos esquemas de controlo de qualidade. Contudo, os requisitos da União Europeia ou dos EUA relacionados com incerteza são genéricos. Esta tese reúne um conjunto de artigos com discussões acerca de métodos para a determinação da incerteza da medição em imunoensaios de rastreio, assim como de abordagens complementares e modelos de exatidão diagnóstica. A utilização do Guia para a Expressão da Incerteza na Medição (GUM) não é sistemática nos laboratórios de rastreio dos serviços de sangue, tal como noutros laboratórios clínicos. Contudo, têm sido usados nos serviços de sangue outras estimativas de incerteza usando métodos estatísticos. Assim, a discussão reúne, não só modelos de acordo com os princípios do GUM focados na incerteza de resultados próximos ou iguais ao “cutoff” (ponto de decisão clínica), mas também modelos complementares: erro total analítico, período de janela seronegativo e modelo delta para uma estimativa inicial do impacto de resultados incertos no orçamento de um serviço de sangue. Na perspetiva da incerteza diagnóstica reúnem‐se os modelos de exatidão diagnóstica para a determinação da sensibilidade clínica, especificidade clínica e área sob a curva da característica de operação do recetor. Complementariamente, são discutidos modelos para a estimação da concordância de resultados entre testes. Para ilustrar os vários métodos de cálculo foram usados resultados de um imunoensaio quimiluminescente comum para antivírus da hepatite C. Os cálculos mostram incerteza expandida em torno do “cutoff” de 21 a 36 %, tendo sido usado para a estimativa da zona de rejeição [0.70, +[, na qual se registaram somente 0.19 % dos resultados de 9805 amostras. O erro total analítico foi de 23 %. O modelo de período de janela estimou um período seronegativo de 97 dias, considerando a zona de rejeição do imunoensaio. O valor delta indicou que a amostragem de doentes testada no “teste 1” tem uma menor probabilidade para gerar resultados indeterminados, não tendo sido evidenciada uma diferença estatisticamente significativa do “teste 1 e “teste 2” gerarem resultados indeterminados na amostra de não doentes. Os intervalos de confiança a 95 % para a sensibilidade e especificidade clínicas foram, respetivamente, de ICS,95% = [88.3, 100 %] e ICE,95% = [98.5, 100 %], tendo‐se obtido uma área sob a curva da característica de operação do recetor entre 0.99 e 1.00. A concordância total entre os resultados de dois imunoensaios para antivírus da hepatite C foi de 98 a 99 %, tendo sido a concordância entre resultados positivos de 87.9 a 100 % e a concordância entre resultados negativos de 97.8 a 99.9 %. A aplicação do intervalo de confiança a 95 % às estimativas de probabilidades e concordâncias é similar ao conceito de incerteza expandida da incerteza da medição. Todos os métodos apresentados têm aplicação na avaliação da incerteza de medição e incerteza diagnóstica de um imunoensaio, embora tenham condições de utilização diferentes. Recomenda‐se um esquema para a seleção dos modelos para a estimativa da incerteza em laboratórios de rastreio de serviços de sangue, atendendo ao papel de cada modelo na segurança pós‐transfusão.Post‐transfusion safety of blood receptors is considered a major issue in the context of blood establishments. Considering the safety of blood receptors, these establishments must test a set of transmissible agents with a significant prevalence in the population. The role of screening tests’ results in the validation of the blood donations is critical for the assurance of posttransfusion safety. In the European Union countries, this screening is required by the Directive 2001/83/EC and in US by the “standards for blood banks and transfusion services”. Due to the uncertainty associated to screening tests’ results, a reported result can be false. A false positive result has no consequence in terms of the blood receptor safety. However, a false negative result has a major impact since there is a high probability of resulting in the infection of the blood component receptor. Consequently, screening laboratories should consider the evaluation of measurement uncertainty in the quality control schemes. However, the European Union or the US requests are generic when dealing with uncertainty. This thesis collects a set of articles discussing methods for the evaluation of measurement uncertainty in screening immunoassays such as complementary approaches and models of diagnostic accuracy. The Guide to the Expression of Uncertainty in Measurement (GUM) is not systematically used in blood establishments’ screening laboratories such as in other medical laboratories. However, other uncertainty estimations, namely based on statistical methods, have been used in blood establishments. Accordingly, the discussion includes not only models fulfilling GUM principles focused on uncertainty of ratio results close or equal to the “cutoff” (clinical decision value), but also complementary models: total analytical error, seronegative window period and the delta for a first estimation of the impact of uncertain results in the blood establishment budget. From the perspective of diagnostic uncertainty are reunited diagnostic accuracy models for determination of diagnostic sensitivity, diagnostic specificity and area under the receiver operating characteristic curve. Complementary are discussed models to agreement between tests’ results. To illustrate the evaluation of measurement uncertainty through the different methods, the results from a same anti‐hepatitis C virus chemiluminescence immunoassay are used. The computations reveal expanded uncertainties around “cutoff” from 21 to 36 % which were used to estimate the “rejection zone” [0.70, + [. Only 0.19 % of 9805 samples were in this region. The total analytical error was 23 %. Considering the immunoassay rejection zone, the window period model estimated a seronegative period of 97 days. The delta‐value indicated that the infected individuals’ sample tested in “test 1” is less likely to produce indeterminate results and has not been shown a statistically significant difference between “test 1” and “test 2” to generate indeterminate results in sample of healthy individuals. The diagnostic sensitivity and diagnostic specificity 95 % confidence intervals were [88.3, 100 %] and [98.5, 100 %], respectively, and the values of the area under the receiver operating characteristic curve obtained were between 0.99 and 1.00. The overall agreement of results between two anti‐hepatitis C virus immunoassays was from 98 to 99 %, the positive results agreement was from 87.9 to 100 % and the negative results agreement from 97.8 to 99.9 %. The 95 % confidence interval applied to the probabilities and agreements estimates is equivalent to the expanded uncertainty concept of measurement uncertainty. The presented methods have all application in the evaluation of measurement uncertainty and diagnostic uncertainty in an immunoassay, although under different conditions of use. A scheme for the selection of models for the estimation of uncertainty in screening laboratories in blood establishments is recommended, taking into account the role of each model in post‐transfusion safety

Dynamic Bayesian Combination of Multiple Imperfect Classifiers

Classifier combination methods need to make best use of the outputs of multiple, imperfect classifiers to enable higher accuracy classifications. In many situations, such as when human decisions need to be combined, the base decisions can vary enormously in reliability. A Bayesian approach to such uncertain combination allows us to infer the differences in performance between individuals and to incorporate any available prior knowledge about their abilities when training data is sparse. In this paper we explore Bayesian classifier combination, using the computationally efficient framework of variational Bayesian inference. We apply the approach to real data from a large citizen science project, Galaxy Zoo Supernovae, and show that our method far outperforms other established approaches to imperfect decision combination. We go on to analyse the putative community structure of the decision makers, based on their inferred decision making strategies, and show that natural groupings are formed. Finally we present a dynamic Bayesian classifier combination approach and investigate the changes in base classifier performance over time.Comment: 35 pages, 12 figure