Learning a set of tasks in sequence remains a challenge for artificial neural
networks, which, in such scenarios, tend to suffer from Catastrophic Forgetting
(CF). The same applies to End-to-End (E2E) Automatic Speech Recognition (ASR)
models, even for monolingual tasks. In this paper, we aim to overcome CF for
E2E ASR by inserting adapters, small architectures of few parameters which
allow a general model to be fine-tuned to a specific task, into our model. We
make these adapters task-specific, while regularizing the parameters of the
model shared by all tasks, thus stimulating the model to fully exploit the
adapters while keeping the shared parameters to work well for all tasks. Our
method outperforms all baselines on two monolingual experiments while being
more storage efficient and without requiring the storage of data from previous
tasks.Comment: Submitted to ICASSP 2023. 5 page