797 research outputs found
On the power of conditional independence testing under model-X
For testing conditional independence (CI) of a response Y and a predictor X
given covariates Z, the recently introduced model-X (MX) framework has been the
subject of active methodological research, especially in the context of MX
knockoffs and their successful application to genome-wide association studies.
In this paper, we study the power of MX CI tests, yielding quantitative
explanations for empirically observed phenomena and novel insights to guide the
design of MX methodology. We show that any valid MX CI test must also be valid
conditionally on Y and Z; this conditioning allows us to reformulate the
problem as testing a point null hypothesis involving the conditional
distribution of X. The Neyman-Pearson lemma then implies that the conditional
randomization test (CRT) based on a likelihood statistic is the most powerful
MX CI test against a point alternative. We also obtain a related optimality
result for MX knockoffs. Switching to an asymptotic framework with arbitrarily
growing covariate dimension, we derive an expression for the limiting power of
the CRT against local semiparametric alternatives in terms of the prediction
error of the machine learning algorithm on which its test statistic is based.
Finally, we exhibit a resampling-free test with uniform asymptotic Type-I error
control under the assumption that only the first two moments of X given Z are
known, a significant relaxation of the MX assumption
Statistical SVMs for robust detection, supervised learning, and universal classification
The support vector machine (SVM) has emerged as one of the most popular approaches to classification and supervised learning. It is a flexible approach for solving the problems posed in these areas, but the approach is not easily adapted to noisy data in which absolute discrimination is not possible. We address this issue in this paper by returning to the statistical setting. The main contribution is the introduction of a statistical support vector machine (SSVM) that captures all of the desirable features of the SVM, along with desirable statistical features of the classical likelihood ratio test. In particular, we establish the following: (i) The SSVM can be designed so that it forms a continuous function of the data, yet also approximates the potentially discontinuous log likelihood ratio test. (ii) Extension to universal detection is developed, in which only one hypothesis is labeled (a semi-supervised learning problem). (iii) The SSVM generalizes the robust hypothesis testing problem based on a moment class. Motivation for the approach and analysis are each based on ideas from information theory. A detailed performance analysis is provided in the special case of i.i.d. observations. This research was partially supported by NSF under grant CCF 07-29031, by UTRC, Motorola, and by the DARPA ITMANET program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF, UTRC, Motorola, or DARPA. I
Universal and Composite Hypothesis Testing via Mismatched Divergence
For the universal hypothesis testing problem, where the goal is to decide
between the known null hypothesis distribution and some other unknown
distribution, Hoeffding proposed a universal test in the nineteen sixties.
Hoeffding's universal test statistic can be written in terms of
Kullback-Leibler (K-L) divergence between the empirical distribution of the
observations and the null hypothesis distribution. In this paper a modification
of Hoeffding's test is considered based on a relaxation of the K-L divergence
test statistic, referred to as the mismatched divergence. The resulting
mismatched test is shown to be a generalized likelihood-ratio test (GLRT) for
the case where the alternate distribution lies in a parametric family of the
distributions characterized by a finite dimensional parameter, i.e., it is a
solution to the corresponding composite hypothesis testing problem. For certain
choices of the alternate distribution, it is shown that both the Hoeffding test
and the mismatched test have the same asymptotic performance in terms of error
exponents. A consequence of this result is that the GLRT is optimal in
differentiating a particular distribution from others in an exponential family.
It is also shown that the mismatched test has a significant advantage over the
Hoeffding test in terms of finite sample size performance. This advantage is
due to the difference in the asymptotic variances of the two test statistics
under the null hypothesis. In particular, the variance of the K-L divergence
grows linearly with the alphabet size, making the test impractical for
applications involving large alphabet distributions. The variance of the
mismatched divergence on the other hand grows linearly with the dimension of
the parameter space, and can hence be controlled through a prudent choice of
the function class defining the mismatched divergence.Comment: Accepted to IEEE Transactions on Information Theory, July 201
Estimating Functions and Equations: An Essay on Historical Developments with Applications to Econometrics
The idea of using estimating functions goes a long way back, at least to Karl Pearson's introduction to the method of moments in 1894. It is now a very active area of research in the statistics literature. One aim of this chapter is to provide an account of the developments relating to the theory of estimating functions. Starting from the simple case of a single parameter under independence, we cover the multiparameter, presence of nuisance parameters and dependent data cases. Application of the estimating functions technique to econometrics is still at its infancy. However, we illustrate how this estimation approach could be used in a number of time series models, such as random coefficient, threshold, bilinear, autoregressive conditional heteroscedasticity models, in models of spatial and longitudinal data, and median regression analysis. The chapter is concluded with some remarks on the place of estimating functions in the history of estimation.
Model Checks Using Residual Marked Empirical Processes
This paper proposes omnibus and directional tests for testing the goodness-of-fit of a parametric regression time series model. We use a general class of residual marked empirical processes as the building-blocks for estimation and testing of such models. First, we establish a weak convergence theorem under mild assumptions, which allows us to study in a unified way the asymptotic null distribution of the test statistics and their asymptotic behavior against Pitman's local alternatives. To approximate the asymptotic null distribution of test statistics we justify theoretically a bootstrap procedure. Also, some asymptotic theory for the estimation of the principal components of the residual marked processes is considered. This asymptotic theory is used to derive optimal directional tests and efficient estimation of regression parameters. Finally, a Monte Carlo study shows that the bootstrap and the asymptotic results provide good approximations for small sample sizes and an empirical application to the Canadian lynx data set is considered.
Testable Forecasts
Predictions about the future are commonly evaluated through statistical tests. As shown by recent literature, many known tests are subject to adverse selection problems and cannot discriminate between forecasters who are competent and forecasters who are uninformed but predict strategically. We consider a framework where forecasters' predictions must be consistent with a paradigm, a set of candidate probability laws for the stochastic process of interest. The paper presents necessary and sufficient conditions on the paradigm under which it is possible to discriminate between informed and uninformed forecasters. We show that optimal tests take the form of likelihood-ratio tests comparing forecasters' predictions against the predictions of a hypothetical Bayesian outside observer. In addition, the paper illustrates a new connection between the problem of testing strategic forecasters and the classical Neyman-Pearson paradigm of hypothesis testing
- …