Strong approximations of level exceedences related to multiple hypothesis testing

Particularly in genomics, but also in other fields, it has become commonplace to undertake highly multiple Student's $t$-tests based on relatively small sample sizes. The literature on this topic is continually expanding, but the main approaches used to control the family-wise error rate and false discovery rate are still based on the assumption that the tests are independent. The independence condition is known to be false at the level of the joint distributions of the test statistics, but that does not necessarily mean, for the small significance levels involved in highly multiple hypothesis testing, that the assumption leads to major errors. In this paper, we give conditions under which the assumption of independence is valid. Specifically, we derive a strong approximation that closely links the level exceedences of a dependent ``studentized process'' to those of a process of independent random variables. Via this connection, it can be seen that in high-dimensional, low sample-size cases, provided the sample size diverges faster than the logarithm of the number of tests, the assumption of independent $t$-tests is often justified.Comment: Published in at http://dx.doi.org/10.3150/09-BEJ220 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm

Multiple Testing under Copula Dependency Structures

The key to multiple testing is to respect the dependencies between the marginal hypotheses tests. Any dependency structure can be modeled by so-called copula functions. This makes copulas an interesting tool in multiple testing. In particular, it is possible to explicitly utilize the dependency structure of the data. This leads to the sub-class of copula-based multiple tests. One family of non-parametric copula estimators is constituted by Bernstein copulas. We extend previous statistical results regarding bivariate Bernstein copulas and study their impact on multiple tests. A related topic is the estimation of the proportion of true null hypotheses pi 0. It is a well known result in multiple hypothesis testing that this proportion is not identified under general dependencies. However, it is possible to estimate pi 0 if structural information about the dependency structure among the p-values is available

Conformal Methods for Quantifying Uncertainty in Spatiotemporal Data: A Survey

Machine learning methods are increasingly widely used in high-risk settings such as healthcare, transportation, and finance. In these settings, it is important that a model produces calibrated uncertainty to reflect its own confidence and avoid failures. In this paper we survey recent works on uncertainty quantification (UQ) for deep learning, in particular distribution-free Conformal Prediction method for its mathematical properties and wide applicability. We will cover the theoretical guarantees of conformal methods, introduce techniques that improve calibration and efficiency for UQ in the context of spatiotemporal data, and discuss the role of UQ in the context of safe decision making

Bayesian Analysis of the Impact of Rainfall Data Product on Simulated Slope Failure for North Carolina Locations

In the past decades, many different approaches have been developed in the literature to quantify the load-carrying capacity and geotechnical stability (or the factor of safety, Fs) of variably saturated hillslopes. Much of this work has focused on a deterministic characterization of hillslope stability. Yet, simulated Fs values are subject to considerable uncertainty due to our inability to characterize accurately the soil mantles properties (hydraulic, geotechnical, and geomorphologic) and spatiotemporal variability of the moisture content of the hillslope interior. This is particularly true at larger spatial scales. Thus, uncertainty-incorporating analyses of physically based models of rain-induced landslides are rare in the literature. Such landslide modeling is typically conducted at the hillslope scale using gauge-based rainfall forcing data with rather poor spatiotemporal coverage. For regional landslide modeling, the specific advantages and/or disadvantages of gauge-only, radar-merged and satellite-based rainfall products are not clearly established. Here, we compare and evaluate the performance of the Transient Rainfall Infiltration and Grid-based Regional Slope-stability analysis (TRIGRS) model for three different rainfall products using 112 observed landslides in the period between 2004 and 2011 from the North Carolina Geological Survey database. Our study includes the Tropical Rainfall Measuring Mission (TRMM) Multi-satellite Precipitation Analysis Version 7 (TMPA V7), the North American Land Data Assimilation System Phase 2 (NLDAS-2) analysis, and the reference truth Stage IV precipitation. TRIGRS model performance was rather inferior with the use of literature values of the geotechnical parameters and soil hydraulic properties from ROSETTA using soil textural and bulk density data from SSURGO (Soil Survey Geographic database). The performance of TRIGRS improved considerably after Bayesian estimation of the parameters with the DiffeRential Evolution Adaptive Metropolis (DREAM) algorithm using Stage IV precipitation data. Hereto, we use a likelihood function that combines binary slope failure information from landslide event and null periods using multivariate frequency distribution-based metrics such as the false discovery and false omission rates. Our results demonstrate that the Stage IV-inferred TRIGRS parameter distributions generalize well to TMPA and NLDAS-2 precipitation data, particularly at sites with considerably larger TMPA and NLDAS-2 rainfall amounts during landslide events than null periods. TRIGRS model performance is then rather similar for all three rainfall products. At higher elevations, however, the TMPA and NLDAS-2 precipitation volumes are insufficient and their performance with the Stage IV-derived parameter distributions indicates their inability to accurately characterize hillslope stability

Uncertainty quantification for the family-wise error rate in multivariate copula models

We derive confidence regions for the realized family-wise error rate (FWER) of certain multiple tests which are empirically calibrated at a given (global) level of significance. To this end, we regard the FWER as a derived parameter of a multivariate parametric copula model. It turns out that the resulting confidence regions are typically very much concentrated around the target FWER level, while generic multiple tests with fixed thresholds are in general not FWER-exhausting. Since FWER level exhaustion and optimization of power are equivalent for the classes of multiple test problems studied in this paper, the aforementioned findings militate strongly in favour of estimating the dependency structure (i. e., copula) and incorporating it in a multivariate multiple test procedure. We illustrate our theoretical results by considering two particular classes of multiple test problems of practical relevance in detail, namely, multiple tests for components of a mean vector and multiple support tests