Carving model-free inference

In many large-scale experiments, the investigator begins with pilot data to look for promising findings. As fresh data becomes available at a later point of time, or from a different source, she is left with the question of how to use the full data to infer for the selected findings. Compensating for the overoptimism from selection, carving permits a reuse of pilot data for valid inference. The principles of carving are quite appealing in practice: instead of throwing away the pilot samples, carving simply discards the information consumed at the time of selection. However, the theoretical justification for carving is strongly tied to parametric models, an example being the ubiquitous gaussian model. In this paper we develop asymptotic guarantees to substantiate the use of carving beyond gaussian generating models. In simulations and in an application on gene expression data, we find that carving delivers valid and tight confidence intervals in model-free settings.Comment: 50 pages, 2 figures, 7 Table