Robust estimation of risks from small samples

Data-driven risk analysis involves the inference of probability distributions from measured or simulated data. In the case of a highly reliable system, such as the electricity grid, the amount of relevant data is often exceedingly limited, but the impact of estimation errors may be very large. This paper presents a robust nonparametric Bayesian method to infer possible underlying distributions. The method obtains rigorous error bounds even for small samples taken from ill-behaved distributions. The approach taken has a natural interpretation in terms of the intervals between ordered observations, where allocation of probability mass across intervals is well-specified, but the location of that mass within each interval is unconstrained. This formulation gives rise to a straightforward computational resampling method: Bayesian Interval Sampling. In a comparison with common alternative approaches, it is shown to satisfy strict error bounds even for ill-behaved distributions.Comment: 13 pages, 3 figures; supplementary information provided. A revised version of this manuscript has been accepted for publication in Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Science

A nonparametric predictive alternative to the Imprecise Dirichlet Model: the case of a known number of categories

Nonparametric Predictive Inference (NPI) is a general methodology to learn from data in the absence of prior knowledge and without adding unjustified assumptions. This paper develops NPI for multinomial data where the total number of possible categories for the data is known. We present the general upper and lower probabilities and several of their properties. We also comment on differences between this NPI approach and corresponding inferences based on Walley's Imprecise Dirichlet Model

Nonparametric predictive inference for system failure time based on bounds for the signature

System signatures provide a powerful framework for reliability assessment for systems consisting of exchangeable components. The use of signatures in nonparametric predictive inference has been presented and leads to lower and upper survival functions for the system failure time, given failure times of tested components. However, deriving the system signature is computationally complex. This article presents how limited information about the signature can be used to derive bounds on such lower and upper survival functions and related inferences. If such bounds are sufficiently decisive they also indicate that more detailed computation of the system signature is not required

Three-group ROC predictive analysis for ordinal outcomes

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning and credit scoring. The receiver operating characteristic (ROC) surface is a useful tool to assess the ability of a diagnostic test to discriminate among three ordered classes or groups. In this paper, nonparametric predictive inference (NPI) for three-group ROC analysis for ordinal outcomes is presented. NPI is a frequentist statistical method that is explicitly aimed at using few modelling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. This paper also includes results on the volumes under the ROC surfaces and consideration of the choice of decision thresholds for the diagnosis. Two examples are provided to illustrate our method

Nonparametric predictive inference for diagnostic test thresholds

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning and credit scoring. The receiver operating characteristic (ROC) curve and surface are useful tools to assess the ability of diagnostic tests to discriminate between ordered classes or groups. To define these diagnostic tests, selecting the optimal thresholds that maximize the accuracy of these tests is required. One procedure that is commonly used to find the optimal thresholds is by maximizing what is known as Youden’s index. This article presents nonparametric predictive inference (NPI) for selecting the optimal thresholds of a diagnostic test. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. Based on multiple future observations, the NPI approach is presented for selecting the optimal thresholds for two-group and three-group scenarios. In addition, a pairwise approach has also been presented for the three-group scenario. The article ends with an example to illustrate the proposed methods and a simulation study of the predictive performance of the proposed methods along with some classical methods such as Youden index. The NPI-based methods show some interesting results that overcome some of the issues concerning the predictive performance of Youden’s index

Nonparametric Predictive Inference for System Reliability

This thesis provides a new method for statistical inference on system reliability on the basis of limited information resulting from component testing. This method is called Nonparametric Predictive Inference (NPI). We present NPI for system reliability, in particular NPI for k-out-of-m systems, and for systems that consist of multiple ki-out-of-mi subsystems in series configuration. The algorithm for optimal redundancy allocation, with additional components added to subsystems one at a time is presented. We also illustrate redundancy allocation for the same system in case the costs of additional components differ per subsystem. Then NPI is presented for system reliability in a similar setting, but with all subsystems consisting of the same single type of component. As a further step in the development of NPI for system reliability, where more general system structures can be considered, nonparametric predictive inference for reliability of voting systems with multiple component types is presented. We start with a single voting system with multiple component types, then we extend to a series configuration of voting subsystems with multiple component types. Throughout this thesis we assume information from tests of nt components of type t