The likelihood ratio test (LRT) and the related F test, do not (even
asymptotically) adhere to their nominal χ2 and F distributions in many
statistical tests common in astrophysics, thereby casting many marginal line or
source detections and non-detections into doubt. Although there are many
legitimate uses of these statistics, in some important cases it can be
impossible to compute the correct false positive rate. For example, it has
become common practice to use the LRT or the F test for detecting a line in a
spectral model or a source above background despite the lack of certain
required regularity conditions. In these and other settings that involve
testing a hypothesis that is on the boundary of the parameter space, {\it
contrary to common practice, the nominal χ2 distribution for the LRT or
the F distribution for the F test should not be used}. In this paper, we
characterize an important class of problems where the LRT and the F test fail
and illustrate this non-standard behavior. We briefly sketch several possible
acceptable alternatives, focusing on Bayesian posterior predictive
probability-values. We present this method in some detail, as it is a simple,
robust, and intuitive approach. This alternative method is illustrated using
the gamma-ray burst of May 8, 1997 (GRB 970508) to investigate the presence of
an Fe K emission line during the initial phase of the observation.Comment: Twenty four pages, seven figures. The Astrophysical Journal, May
2002, to appea