5 research outputs found
Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
We propose a neural language modeling system based on low-rank adaptation
(LoRA) for speech recognition output rescoring. Although pretrained language
models (LMs) like BERT have shown superior performance in second-pass
rescoring, the high computational cost of scaling up the pretraining stage and
adapting the pretrained models to specific domains limit their practical use in
rescoring. Here we present a method based on low-rank decomposition to train a
rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
of the pretrained parameters. These inserted matrices are optimized through a
discriminative training objective along with a correlation-based regularization
loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
evaluated on LibriSpeech and internal datasets with decreased training times by
factors between 5.4 and 3.6.Comment: Accepted to IEEE ASRU 2023. Internal Review Approved. Revised 2nd
version with Andreas and Huck. The first version is in Sep 29th. 8 page
Generation of diffraction-free petallike beams based on stationary phase principle
Lately, a kind of so-called “combined half-integer Bessel-like beams,” in which their subfamilies include radial carpet, petallike, and ringlike vortex beams, has been generated using a radial periodic grating. Different from classical petallike beams, we introduce and generate a kind of new petallike beams, that is, diffraction-free petallike beams (DFPBs). Their diffraction pattern is independent of the propagation distance. Firstly, the generating mechanism of diffraction-free petallike beams is theoretically investigated based on the stationary phase principle. Then, we theoretically construct a family of patterns of DFPBs. Subsequently, the DFPBs is generated by using an amplitude radial grating with a sinusoidal transmission function and two phase modulation elements — phase plate and axicon. Experimental results well confirm the theoretical predictions, thereby showing that the stationary phase principle is an effective approach to generating various diffraction-free combined half-integer Bessel-like beams based on radial grating, including diffraction-free radial carpet, petallike, and ringlike vortex beams