Preventing the performance decay of Transformers on inputs longer than those
used for training has been an important challenge in extending the context
length of these models. Though the Transformer architecture has fundamentally
no limits on the input sequence lengths it can process, the choice of position
encoding used during training can limit the performance of these models on
longer inputs. We propose a novel functional relative position encoding with
progressive interpolation, FIRE, to improve Transformer generalization to
longer contexts. We theoretically prove that this can represent some of the
popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We
next empirically show that FIRE models have better generalization to longer
contexts on both zero-shot language modeling and long text benchmarks