Bringing game theory to hypothesis testing: Establishing finite sample bounds on inference

Abstract

Small sample properties are of fundamental interest when only limited data is avail- able. Exact inference is limited by constraints imposed by speci.c nonrandomized tests and of course also by lack of more data. These e¤ects can be separated as we propose to evaluate a test by comparing its type II error to the minimal type II error among all tests for the given sample. Game theory is used to establish this minimal type II error, the associated randomized test is characterized as part of a Nash equilibrium of a .ctitious game against nature. We use this method to investigate sequential tests for the di¤erence between two means when outcomes are constrained to belong to a given bounded set. Tests of inequality and of noninferiority are included. We .nd that inference in terms of type II error based on a balanced sample cannot be improved by sequential sampling or even by observing counter factual evidence providing there is a reasonable gap between the hypotheses.Exact, distribution-free, nonparametric, independent samples, matched pairs, Z test, unavoidable type II error, noninferiority, LeeX