797 research outputs found

    On the power of conditional independence testing under model-X

    Full text link
    For testing conditional independence (CI) of a response Y and a predictor X given covariates Z, the recently introduced model-X (MX) framework has been the subject of active methodological research, especially in the context of MX knockoffs and their successful application to genome-wide association studies. In this paper, we study the power of MX CI tests, yielding quantitative explanations for empirically observed phenomena and novel insights to guide the design of MX methodology. We show that any valid MX CI test must also be valid conditionally on Y and Z; this conditioning allows us to reformulate the problem as testing a point null hypothesis involving the conditional distribution of X. The Neyman-Pearson lemma then implies that the conditional randomization test (CRT) based on a likelihood statistic is the most powerful MX CI test against a point alternative. We also obtain a related optimality result for MX knockoffs. Switching to an asymptotic framework with arbitrarily growing covariate dimension, we derive an expression for the limiting power of the CRT against local semiparametric alternatives in terms of the prediction error of the machine learning algorithm on which its test statistic is based. Finally, we exhibit a resampling-free test with uniform asymptotic Type-I error control under the assumption that only the first two moments of X given Z are known, a significant relaxation of the MX assumption

    Statistical SVMs for robust detection, supervised learning, and universal classification

    Get PDF
    The support vector machine (SVM) has emerged as one of the most popular approaches to classification and supervised learning. It is a flexible approach for solving the problems posed in these areas, but the approach is not easily adapted to noisy data in which absolute discrimination is not possible. We address this issue in this paper by returning to the statistical setting. The main contribution is the introduction of a statistical support vector machine (SSVM) that captures all of the desirable features of the SVM, along with desirable statistical features of the classical likelihood ratio test. In particular, we establish the following: (i) The SSVM can be designed so that it forms a continuous function of the data, yet also approximates the potentially discontinuous log likelihood ratio test. (ii) Extension to universal detection is developed, in which only one hypothesis is labeled (a semi-supervised learning problem). (iii) The SSVM generalizes the robust hypothesis testing problem based on a moment class. Motivation for the approach and analysis are each based on ideas from information theory. A detailed performance analysis is provided in the special case of i.i.d. observations. This research was partially supported by NSF under grant CCF 07-29031, by UTRC, Motorola, and by the DARPA ITMANET program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF, UTRC, Motorola, or DARPA. I

    Universal and Composite Hypothesis Testing via Mismatched Divergence

    Full text link
    For the universal hypothesis testing problem, where the goal is to decide between the known null hypothesis distribution and some other unknown distribution, Hoeffding proposed a universal test in the nineteen sixties. Hoeffding's universal test statistic can be written in terms of Kullback-Leibler (K-L) divergence between the empirical distribution of the observations and the null hypothesis distribution. In this paper a modification of Hoeffding's test is considered based on a relaxation of the K-L divergence test statistic, referred to as the mismatched divergence. The resulting mismatched test is shown to be a generalized likelihood-ratio test (GLRT) for the case where the alternate distribution lies in a parametric family of the distributions characterized by a finite dimensional parameter, i.e., it is a solution to the corresponding composite hypothesis testing problem. For certain choices of the alternate distribution, it is shown that both the Hoeffding test and the mismatched test have the same asymptotic performance in terms of error exponents. A consequence of this result is that the GLRT is optimal in differentiating a particular distribution from others in an exponential family. It is also shown that the mismatched test has a significant advantage over the Hoeffding test in terms of finite sample size performance. This advantage is due to the difference in the asymptotic variances of the two test statistics under the null hypothesis. In particular, the variance of the K-L divergence grows linearly with the alphabet size, making the test impractical for applications involving large alphabet distributions. The variance of the mismatched divergence on the other hand grows linearly with the dimension of the parameter space, and can hence be controlled through a prudent choice of the function class defining the mismatched divergence.Comment: Accepted to IEEE Transactions on Information Theory, July 201

    Estimating Functions and Equations: An Essay on Historical Developments with Applications to Econometrics

    Get PDF
    The idea of using estimating functions goes a long way back, at least to Karl Pearson's introduction to the method of moments in 1894. It is now a very active area of research in the statistics literature. One aim of this chapter is to provide an account of the developments relating to the theory of estimating functions. Starting from the simple case of a single parameter under independence, we cover the multiparameter, presence of nuisance parameters and dependent data cases. Application of the estimating functions technique to econometrics is still at its infancy. However, we illustrate how this estimation approach could be used in a number of time series models, such as random coefficient, threshold, bilinear, autoregressive conditional heteroscedasticity models, in models of spatial and longitudinal data, and median regression analysis. The chapter is concluded with some remarks on the place of estimating functions in the history of estimation.

    Model Checks Using Residual Marked Empirical Processes

    Get PDF
    This paper proposes omnibus and directional tests for testing the goodness-of-fit of a parametric regression time series model. We use a general class of residual marked empirical processes as the building-blocks for estimation and testing of such models. First, we establish a weak convergence theorem under mild assumptions, which allows us to study in a unified way the asymptotic null distribution of the test statistics and their asymptotic behavior against Pitman's local alternatives. To approximate the asymptotic null distribution of test statistics we justify theoretically a bootstrap procedure. Also, some asymptotic theory for the estimation of the principal components of the residual marked processes is considered. This asymptotic theory is used to derive optimal directional tests and efficient estimation of regression parameters. Finally, a Monte Carlo study shows that the bootstrap and the asymptotic results provide good approximations for small sample sizes and an empirical application to the Canadian lynx data set is considered.

    Testable Forecasts

    Get PDF
    Predictions about the future are commonly evaluated through statistical tests. As shown by recent literature, many known tests are subject to adverse selection problems and cannot discriminate between forecasters who are competent and forecasters who are uninformed but predict strategically. We consider a framework where forecasters' predictions must be consistent with a paradigm, a set of candidate probability laws for the stochastic process of interest. The paper presents necessary and sufficient conditions on the paradigm under which it is possible to discriminate between informed and uninformed forecasters. We show that optimal tests take the form of likelihood-ratio tests comparing forecasters' predictions against the predictions of a hypothetical Bayesian outside observer. In addition, the paper illustrates a new connection between the problem of testing strategic forecasters and the classical Neyman-Pearson paradigm of hypothesis testing
    corecore