Transfer Learning for Speech and Language Processing

Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of `model adaptation'. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the `transfer' can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field.Comment: 13 pages, APSIPA 201

Lyapunov exponent, universality and phase transition for products of random matrices

Products of $M$ i.i.d. random matrices of size $N \times N$ are related to classical limit theorems in probability theory ($N=1$ and large $M$), to Lyapunov exponents in dynamical systems (finite $N$ and large $M$), and to universality in random matrix theory (finite $M$ and large $N$). Under the two different limits of $M \to \infty$ and $N \to \infty$, the local singular value statistics display Gaussian and random matrix theory universality, respectively. However, it is unclear what happens if both $M$ and $N$ go to infinity. This problem, proposed by Akemann, Burda, Kieburg \cite{Akemann-Burda-Kieburg14} and Deift \cite{Deift17}, lies at the heart of understanding both kinds of universal limits. In the case of complex Gaussian random matrices, we prove that there exists a crossover phenomenon as the relative ratio of $M$ and $N$ changes from $0$ to $\infty$: sine and Airy kernels from the Gaussian Unitary Ensemble (GUE) when $M/N \to 0$, Gaussian fluctuation when $M/N \to \infty$, and new critical phenomena when $M/N \to \gamma \in (0,\infty)$. Accordingly, we further prove that the largest singular value undergoes a phase transition between the Gaussian and GUE Tracy-Widom distributions.Comment: Therems 1.1 stated in a more intuitive way; proofs extensively revised; convergence in trace norm for the critical and subcritical cases added; 35 pages, 3 figure

On Explicit Probability Densities Associated with Fuss-Catalan Numbers

In this note we give explicitly a family of probability densities, the moments of which are Fuss-Catalan numbers. The densities appear naturally in random matrices, free probability and other contexts.Comment: 4 page

A note on eigenvalues of random block Toeplitz matrices with slowly growing bandwidth

This paper can be thought of as a remark of \cite{llw}, where the authors studied the eigenvalue distribution $\mu_{X_N}$ of random block Toeplitz band matrices with given block order $m$. In this note we will give explicit density functions of $\lim\limits_{N\to\infty}\mu_{X_N}$ when the bandwidth grows slowly. In fact, these densities are exactly the normalized one-point correlation functions of $m\times m$ Gaussian unitary ensemble (GUE for short). The series $\{\lim\limits_{N\to\infty}\mu_{X_N}|m\in\mathbb{N}\}$ can be seen as a transition from the standard normal distribution to semicircle distribution. We also show a similar relationship between GOE and block Toeplitz band matrices with symmetric blocks.Comment: 6 page