Recent years have seen the advent of molecular simulation datasets that are
orders of magnitude larger and more diverse. These new datasets differ
substantially in four aspects of complexity: 1. Chemical diversity (number of
different elements), 2. system size (number of atoms per sample), 3. dataset
size (number of data samples), and 4. domain shift (similarity of the training
and test set). Despite these large differences, benchmarks on small and narrow
datasets remain the predominant method of demonstrating progress in graph
neural networks (GNNs) for molecular simulation, likely due to cheaper training
compute requirements. This raises the question -- does GNN progress on small
and narrow datasets translate to these more complex datasets? This work
investigates this question by first developing the GemNet-OC model based on the
large Open Catalyst 2020 (OC20) dataset. GemNet-OC outperforms the previous
state-of-the-art on OC20 by 16% while reducing training time by a factor of 10.
We then compare the impact of 18 model components and hyperparameter choices on
performance in multiple datasets. We find that the resulting model would be
drastically different depending on the dataset used for making model choices.
To isolate the source of this discrepancy we study six subsets of the OC20
dataset that individually test each of the above-mentioned four dataset
aspects. We find that results on the OC-2M subset correlate well with the full
OC20 dataset while being substantially cheaper to train on. Our findings
challenge the common practice of developing GNNs solely on small datasets, but
highlight ways of achieving fast development cycles and generalizable results
via moderately-sized, representative datasets such as OC-2M and efficient
models such as GemNet-OC. Our code and pretrained model weights are
open-sourced