Multiple Testing for Exploratory Research

Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing.Comment: Published in at http://dx.doi.org/10.1214/11-STS356 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Multiple testing procedures under confounding

While multiple testing procedures have been the focus of much statistical research, an important facet of the problem is how to deal with possible confounding. Procedures have been developed by authors in genetics and statistics. In this chapter, we relate these proposals. We propose two new multiple testing approaches within this framework. The first combines sensitivity analysis methods with false discovery rate estimation procedures. The second involves construction of shrinkage estimators that utilize the mixture model for multiple testing. The procedures are illustrated with applications to a gene expression profiling experiment in prostate cancer.Comment: Published in at http://dx.doi.org/10.1214/193940307000000176 the IMS Collections (http://www.imstat.org/publications/imscollections.htm) by the Institute of Mathematical Statistics (http://www.imstat.org

Peak Detection as Multiple Testing

This paper considers the problem of detecting equal-shaped non-overlapping unimodal peaks in the presence of Gaussian ergodic stationary noise, where the number, location and heights of the peaks are unknown. A multiple testing approach is proposed in which, after kernel smoothing, the presence of a peak is tested at each observed local maximum. The procedure provides strong control of the family wise error rate and the false discovery rate asymptotically as both the signal-to-noise ratio (SNR) and the search space get large, where the search space may grow exponentially as a function of SNR. Simulations assuming a Gaussian peak shape and a Gaussian autocorrelation function show that desired error levels are achieved for relatively low SNR and are robust to partial peak overlap. Simulations also show that detection power is maximized when the smoothing bandwidth is close to the bandwidth of the signal peaks, akin to the well-known matched filter theorem in signal processing. The procedure is illustrated in an analysis of electrical recordings of neuronal cell activity.Comment: 37 pages, 8 figure

Multiple testing with persistent homology

Multiple hypothesis testing requires a control procedure. Simply increasing simulations or permutations to meet a Bonferroni-style threshold is prohibitively expensive. In this paper we propose a null model based approach to testing for acyclicity, coupled with a Family-Wise Error Rate (FWER) control method that does not suffer from these computational costs. We adapt an False Discovery Rate (FDR) control approach to the topological setting, and show it to be compatible both with our null model approach and with previous approaches to hypothesis testing in persistent homology. By extending a limit theorem for persistent homology on samples from point processes, we provide theoretical validation for our FWER and FDR control methods

Testing for Multiple Bubbles

Identifying and dating explosive bubbles when there is periodically collapsing behavior over time has been a major concern in the economics literature and is of great importance for practitioners. The complexity of the nonlinear structure inherent in multiple bubble phenomena within the same sample period makes econometric analysis particularly difficult. The present paper develops new recursive procedures for practical implementation and surveillance strategies that may be employed by central banks and fiscal regulators. We show how the testing procedure and dating algorithm of Phillips, Wu and Yu (2011, PWY) are affected by multiple bubbles and may fail to be consistent. The present paper proposes a generalized version of the sup ADF test of PWY to address this difficulty, derives its asymptotic distribution, introduces a new date-stamping strategy for the origination and termination of multiple bubbles, and proves consistency of this dating procedure. Simulations show that the test significantly improves discriminatory power and leads to distinct power gains when multiple bubbles occur. Empirical applications are conducted to S&P 500 stock market data over a long historical period from January 1871 to December 2010. The new approach identifies many key historical episodes of exuberance and collapse over this period, whereas the strategy of PWY and the CUSUM procedure locate far fewer episodes in the same sample range.Date-stamping strategy, Generalized sup ADF test, Multiple bubbles, Rational bubble, Periodically collapsing bubbles, Sup ADF test