Modern deep learning systems require huge data sets to achieve impressive
performance, but there is little guidance on how much or what kind of data to
collect. Over-collecting data incurs unnecessary present costs, while
under-collecting may incur future costs and delay workflows. We propose a new
paradigm for modeling the data collection workflow as a formal optimal data
collection problem that allows designers to specify performance targets,
collection costs, a time horizon, and penalties for failing to meet the
targets. Additionally, this formulation generalizes to tasks requiring multiple
data sources, such as labeled and unlabeled data used in semi-supervised
learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which
minimizes expected future collection costs. Finally, we numerically compare our
framework to the conventional baseline of estimating data requirements by
extrapolating from neural scaling laws. We significantly reduce the risks of
failing to meet desired performance targets on several classification,
segmentation, and detection tasks, while maintaining low total collection
costs.Comment: Accepted to NeurIPS 202