Bayesian learning of models for estimating uncertainty in alert systems: application to air traffic conflict avoidance

Alert systems detect critical events which can happen in the short term. Uncertainties in data and in the models used for detection cause alert errors. In the case of air traffic control systems such as Short-Term Conflict Alert (STCA), uncertainty increases errors in alerts of separation loss. Statistical methods that are based on analytical assumptions can provide biased estimates of uncertainties. More accurate analysis can be achieved by using Bayesian Model Averaging, which provides estimates of the posterior probability distribution of a prediction. We propose a new approach to estimate the prediction uncertainty, which is based on observations that the uncertainty can be quantified by variance of predicted outcomes. In our approach, predictions for which variances of posterior probabilities are above a given threshold are assigned to be uncertain. To verify our approach we calculate a probability of alert based on the extrapolation of closest point of approach. Using Heathrow airport flight data we found that alerts are often generated under different conditions, variations in which lead to alert detection errors. Achieving 82.1% accuracy of modelling the STCA system, which is a necessary condition for evaluating the uncertainty in prediction, we found that the proposed method is capable of reducing the uncertain component. Comparison with a bootstrap aggregation method has demonstrated a significant reduction of uncertainty in predictions. Realistic estimates of uncertainties will open up new approaches to improving the performance of alert systems

Uncertain Trees: Dealing with Uncertain Inputs in Regression Trees

Tree-based ensemble methods, as Random Forests and Gradient Boosted Trees, have been successfully used for regression in many applications and research studies. Furthermore, these methods have been extended in order to deal with uncertainty in the output variable, using for example a quantile loss in Random Forests (Meinshausen, 2006). To the best of our knowledge, no extension has been provided yet for dealing with uncertainties in the input variables, even though such uncertainties are common in practical situations. We propose here such an extension by showing how standard regression trees optimizing a quadratic loss can be adapted and learned while taking into account the uncertainties in the inputs. By doing so, one no longer assumes that an observation lies into a single region of the regression tree, but rather that it belongs to each region with a certain probability. Experiments conducted on several data sets illustrate the good behavior of the proposed extension.Comment: 9 page

Assessing Multiple Prior Models of Behaviour under Ambiguity

The recent spate of theoretical models of behaviour under ambiguity can be partitioned into two sets: those involving multiple priors (in which the probabilities of the various events are not known but probabilities can be attached to the various possible values for the probabilities) and those not involving multiple priors. This paper concentrates on the first set and provides an experimental investigation into recently proposed theories. Using an appropriate experimental interface, in which the probabilities on the various possibilities are explicitly stated, we examine the fitted and predictive power of the various theories. We first estimate subject-by-subject, and then we estimateand predict using a mixture model over the contending theories. The individual estimates suggest that 25% of our 149 subjects have behaviour consistent with Expected Utility, 54% with the Smooth Model (of Klibanoff et al, 2005), 12% with Rank Dependent Expected Utility and 9% with the Alpha Model (of Ghirardato et al 2004); these figures are very close to the mixing proportions obtained from the mixture estimates. However, if we classify our subjects through the posterior probabilities (given all the evidence) of each of them being of the various types: using the estimates we get 38%, 19%, 28% and 16% (for EU, Smooth, Rank Dependent and Alpha); while using the predictions 36%, 19%, 33% and 11%. Interestingly the older models (EU and RD) seem to fare relatively better, suggesting that representing ambiguity through multiple priors is perceived by subjects as risk, rather than ambiguityAlpha Model, Ambiguity, Expected Utility, Mixture Models, Rank Dependent Expected Utility, Smooth Model.

Recognizing Uncertainty in Speech

We address the problem of inferring a speaker's level of certainty based on prosodic information in the speech signal, which has application in speech-based dialogue systems. We show that using phrase-level prosodic features centered around the phrases causing uncertainty, in addition to utterance-level prosodic features, improves our model's level of certainty classification. In addition, our models can be used to predict which phrase a person is uncertain about. These results rely on a novel method for eliciting utterances of varying levels of certainty that allows us to compare the utility of contextually-based feature sets. We elicit level of certainty ratings from both the speakers themselves and a panel of listeners, finding that there is often a mismatch between speakers' internal states and their perceived states, and highlighting the importance of this distinction.Comment: 11 page