Martins, A.F.T.

Mihaylova, T.

Niculae, V.

English

Latent structure models are a powerful tool for modeling language data: they can mitigate the error propagation and annotation bottleneck in pipeline systems, while simultaneously uncovering linguistic insights about the data. One challenge with end-to-end training of these models is the argmax operation, which has null gradient. In this paper, we focus on surrogate gradients, a popular strategy to deal with this problem. We explore latent structure learning through the angle of pulling back the downstream learning objective. In this paradigm, we discover a principled motivation for both the straight-through estimator (STE) as well as the recently-proposed SPIGOT – a variant of STE for structured models. Our perspective leads to new algorithms in the same family. We empirically compare the known and the novel pulled-back estimators against the popular alternatives, yielding new insight for practitioners and revealing intriguing failure cases

International Migration, Integration and Social Cohesion online publications

Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent Structure Learning

https://pure.uva.nl/ws/files/65724399/2020.emnlp_main.171.pdf

Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent Structure Learning

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications