This paper describes Tallinn University of Technology (TalTech) systems
developed for the ASRU MADASR 2023 Challenge. The challenge focuses on
automatic speech recognition of dialect-rich Indian languages with limited
training audio and text data. TalTech participated in two tracks of the
challenge: Track 1 that allowed using only the provided training data and Track
3 which allowed using additional audio data. In both tracks, we relied on
wav2vec2.0 models. Our methodology diverges from the traditional procedure of
finetuning pretrained wav2vec2.0 models in two key points: firstly, through the
implementation of the aligned data augmentation technique to enhance the
linguistic diversity of the training data, and secondly, via the application of
deep prefix tuning for dialect adaptation of wav2vec2.0 models. In both tracks,
our approach yielded significant improvements over the provided baselines,
achieving the lowest word error rates across all participating teams