5 research outputs found
The edge of discovery: Controlling the local false discovery rate at the margin
Despite the popularity of the false discovery rate (FDR) as an error control
metric for large-scale multiple testing, its close Bayesian counterpart the
local false discovery rate (lfdr), defined as the posterior probability that a
particular null hypothesis is false, is a more directly relevant standard for
justifying and interpreting individual rejections. However, the lfdr is
difficult to work with in small samples, as the prior distribution is typically
unknown. We propose a simple multiple testing procedure and prove that it
controls the expectation of the maximum lfdr across all rejections;
equivalently, it controls the probability that the rejection with the largest
p-value is a false discovery. Our method operates without knowledge of the
prior, assuming only that the p-value density is uniform under the null and
decreasing under the alternative. We also show that our method asymptotically
implements the oracle Bayes procedure for a weighted classification risk,
optimally trading off between false positives and false negatives. We derive
the limiting distribution of the attained maximum lfdr over the rejections, and
the limiting empirical Bayes regret relative to the oracle procedure
Incentive-Theoretic Bayesian Inference for Collaborative Science
Contemporary scientific research is a distributed, collaborative endeavor,
carried out by teams of researchers, regulatory institutions, funding agencies,
commercial partners, and scientific bodies, all interacting with each other and
facing different incentives. To maintain scientific rigor, statistical methods
should acknowledge this state of affairs. To this end, we study hypothesis
testing when there is an agent (e.g., a researcher or a pharmaceutical company)
with a private prior about an unknown parameter and a principal (e.g., a
policymaker or regulator) who wishes to make decisions based on the parameter
value. The agent chooses whether to run a statistical trial based on their
private prior and then the result of the trial is used by the principal to
reach a decision. We show how the principal can conduct statistical inference
that leverages the information that is revealed by an agent's strategic
behavior -- their choice to run a trial or not. In particular, we show how the
principal can design a policy to elucidate partial information about the
agent's private prior beliefs and use this to control the posterior probability
of the null. One implication is a simple guideline for the choice of
significance threshold in clinical trials: the type-I error level should be set
to be strictly less than the cost of the trial divided by the firm's profit if
the trial is successful
Principal-Agent Hypothesis Testing
Consider the relationship between the FDA (the principal) and a
pharmaceutical company (the agent). The pharmaceutical company wishes to sell a
product to make a profit, and the FDA wishes to ensure that only efficacious
drugs are released to the public. The efficacy of the drug is not known to the
FDA, so the pharmaceutical company must run a costly trial to prove efficacy to
the FDA. Critically, the statistical protocol used to establish efficacy
affects the behavior of a strategic, self-interested pharmaceutical company; a
lower standard of statistical evidence incentivizes the pharmaceutical company
to run more trials for drugs that are less likely to be effective, since the
drug may pass the trial by chance, resulting in large profits. The interaction
between the statistical protocol and the incentives of the pharmaceutical
company is crucial to understanding this system and designing protocols with
high social utility. In this work, we discuss how the principal and agent can
enter into a contract with payoffs based on statistical evidence. When there is
stronger evidence for the quality of the product, the principal allows the
agent to make a larger profit. We show how to design contracts that are robust
to an agent's strategic actions, and derive the optimal contract in the
presence of strategic behavior