Ground truth data requirements for altimeter performance verification

The amount and type of ground truth required for an altimeter experiment is a function of the uncertainty in the satellite orbit, the altimeter error budget and the type of operation being performed. Ground truth requirements will be discussed with reference to three areas of operation: the global mode, the high intensity mode and calibration

Truth and trust in communication - Experiments on the effect of a competitive context

The paper employs laboratory experimentation to study the effect of competition on truth telling and trust in communication. A sequence of either competitive or cooperative interactions preceded an experimental communication game. In the game, informed advisors sent a recommendation to decision-makers who faced uncertainty about the consequences of their choice. While many advisors told the truth against their monetary self-interest, the propensity to tell the truth was unaffected by the contextual priming. In contrast, decision-makers trusted significantly less in a competitive context. The effect was strongest when they faced full uncertainty. The paper relates this result to psychological and neuro-economic findings on automatic information processing. The data of this study were largely in line with Subjective Equilibrium Analysis (Kalai and Lehrer, 1995).

Kochen-Specker Theorem for Finite Precision Spin One Measurements

Unsharp spin 1 observables arise from the fact that a residual uncertainty about the actual orientation of the measurement device remains. If the uncertainty is below a certain level, and if the distribution of measurement errors is covariant under rotations, a Kochen-Specker theorem for the unsharp spin observables follows: There are finite sets of directions such that not all the unsharp spin observables in these directions can consistently be assigned approximate truth-values in a non-contextual way.Comment: 4 page

A fuzzified BRAIN algorithm for learning DNF from incomplete data

Aim of this paper is to address the problem of learning Boolean functions from training data with missing values. We present an extension of the BRAIN algorithm, called U-BRAIN (Uncertainty-managing Batch Relevance-based Artificial INtelligence), conceived for learning DNF Boolean formulas from partial truth tables, possibly with uncertain values or missing bits. Such an algorithm is obtained from BRAIN by introducing fuzzy sets in order to manage uncertainty. In the case where no missing bits are present, the algorithm reduces to the original BRAIN

On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation

Uncertainty estimation methods are expected to improve the understanding and quality of computer-assisted methods used in medical applications (e.g., neurosurgical interventions, radiotherapy planning), where automated medical image segmentation is crucial. In supervised machine learning, a common practice to generate ground truth label data is to merge observer annotations. However, as many medical image tasks show a high inter-observer variability resulting from factors such as image quality, different levels of user expertise and domain knowledge, little is known as to how inter-observer variability and commonly used fusion methods affect the estimation of uncertainty of automated image segmentation. In this paper we analyze the effect of common image label fusion techniques on uncertainty estimation, and propose to learn the uncertainty among observers. The results highlight the negative effect of fusion methods applied in deep learning, to obtain reliable estimates of segmentation uncertainty. Additionally, we show that the learned observers' uncertainty can be combined with current standard Monte Carlo dropout Bayesian neural networks to characterize uncertainty of model's parameters.Comment: Appears in Medical Image Computing and Computer Assisted Interventions (MICCAI), 201

Automated analysis of radar imagery of Venus: handling lack of ground truth

Lack of verifiable ground truth is a common problem in remote sensing image analysis. For example, consider the synthetic aperture radar (SAR) image data of Venus obtained by the Magellan spacecraft. Planetary scientists are interested in automatically cataloging the locations of all the small volcanoes in this data set; however, the problem is very difficult and cannot be performed with perfect reliability even by human experts. Thus, training and evaluating the performance of an automatic algorithm on this data set must be handled carefully. We discuss the use of weighted free-response receiver-operating characteristics (wFROCs) for evaluating detection performance when the “ground truth” is subjective. In particular, we evaluate the relative detection performance of humans and automatic algorithms. Our experimental results indicate that proper assessment of the uncertainty in “ground truth” is essential in applications of this nature

Estimating continuous affect with label uncertainty

Continuous affect estimation is a problem where there is an inherent uncertainty and subjectivity in the labels that accompany data samples -- typically, datasets use the average of multiple annotations or self-reporting to obtain ground truth labels. In this work, we propose a method for uncertainty-aware continuous affect estimation, that models explicitly the uncertainty of the ground truth label as a uni-variate Gaussian with mean equal to the ground truth label, and unknown variance. For each sample, the proposed neural network estimates not only the value of the target label (valence and arousal in our case), but also the variance. The network is trained with a loss that is defined as the KL-divergence between the estimation (valence/arousal) and the Gaussian around the ground truth. We show that, in two affect recognition problems with real data, the estimated variances are correlated with measures of uncertainty/error in the labels that are extracted by considering multiple annotations of the data