We study the compute-optimal trade-off between model and training data set
sizes for large neural networks. Our result suggests a linear relation similar
to that supported by the empirical analysis of chinchilla. While that work
studies transformer-based large language models trained on the MassiveText
corpus gopher, as a starting point for development of a mathematical theory, we
focus on a simpler learning model and data generating process, each based on a
neural network with a sigmoidal output unit and single hidden layer of ReLU
activation units. We introduce general error upper bounds for a class of
algorithms which incrementally update a statistic (for example gradient
descent). For a particular learning model inspired by barron 1993, we establish
an upper bound on the minimal information-theoretically achievable expected
error as a function of model and data set sizes. We then derive allocations of
computation that minimize this bound. We present empirical results which
suggest that this approximation correctly identifies an asymptotic linear
compute-optimal scaling. This approximation also generates new insights. Among
other things, it suggests that, as the input dimension or latent space
complexity grows, as might be the case for example if a longer history of
tokens is taken as input to a language model, a larger fraction of the compute
budget should be allocated to growing the learning model rather than training
data