Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.Comment: Accepted to IEEE ASRU 2023. Internal Review Approved. Revised 2nd version with Andreas and Huck. The first version is in Sep 29th. 8 page

Generation of diffraction-free petallike beams based on stationary phase principle

Lately, a kind of so-called “combined half-integer Bessel-like beams,” in which their subfamilies include radial carpet, petallike, and ringlike vortex beams, has been generated using a radial periodic grating. Different from classical petallike beams, we introduce and generate a kind of new petallike beams, that is, diffraction-free petallike beams (DFPBs). Their diffraction pattern is independent of the propagation distance. Firstly, the generating mechanism of diffraction-free petallike beams is theoretically investigated based on the stationary phase principle. Then, we theoretically construct a family of patterns of DFPBs. Subsequently, the DFPBs is generated by using an amplitude radial grating with a sinusoidal transmission function and two phase modulation elements — phase plate and axicon. Experimental results well confirm the theoretical predictions, thereby showing that the stationary phase principle is an effective approach to generating various diffraction-free combined half-integer Bessel-like beams based on radial grating, including diffraction-free radial carpet, petallike, and ringlike vortex beams