Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition

While word error rates of automatic speech recognition (ASR) systems have consistently fallen, natural language understanding (NLU) applications built on top of ASR systems still attribute significant numbers of failures to low-quality speech recognition results. Existing assistant systems collect large numbers of these unsuccessful interactions, but these systems usually fail to learn from these interactions, even in an offline fashion. In this work, we introduce CLC: Contrastive Learning for Conversations, a family of methods for contrastive fine-tuning of models in a self-supervised fashion, making use of easily detectable artifacts in unsuccessful conversations with assistants. We demonstrate that our CLC family of approaches can improve the performance of ASR models on OD3, a new public large-scale semi-synthetic meta-dataset of audio task-oriented dialogues, by up to 19.2%. These gains transfer to real-world systems as well, where we show that CLC can help to improve performance by up to 6.7% over baselines. We make OD3 publicly available at https://github.com/amazon-science/amazon-od3 .Comment: To appear in ICASSP 202

A Riemannian Primal-dual Algorithm Based on Proximal Operator and its Application in Metric Learning

In this paper, we consider optimizing a smooth, convex, lower semicontinuous function in Riemannian space with constraints. To solve the problem, we first convert it to a dual problem and then propose a general primal-dual algorithm to optimize the primal and dual variables iteratively. In each optimization iteration, we employ a proximal operator to search optimal solution in the primal space. We prove convergence of the proposed algorithm and show its non-asymptotic convergence rate. By utilizing the proposed primal-dual optimization technique, we propose a novel metric learning algorithm which learns an optimal feature transformation matrix in the Riemannian space of positive definite matrices. Preliminary experimental results on an optimal fund selection problem in fund of funds (FOF) management for quantitative investment showed its efficacy.Comment: 8 pages, 2 figures, published as a conference paper in 2019 International Joint Conference on Neural Networks (IJCNN

Non-Parametric Manifold Learning

We introduce an estimator for distances in a compact Riemannian manifold M based on graph Laplacian estimates of the Laplace-Beltrami operator. We upper bound the l2-loss for the ratio of the estimator over the true manifold distance, or more precisely an approximation of manifold distance in non-commutative geometry (cf. [Connes and Suijelekom, 2020]), in terms of spectral errors in the graph Laplacian estimates and, implicitly, several geometric properties of the manifold. We consequently obtain a consistency result for the estimator for samples equidistributed from a strictly positive density on M and graph Laplacians which spectrally converge, in a suitable sense, to the Laplace-Beltrami operator. The estimator resembles, and in fact its convergence properties are derived from, a special case of the Kontorovic dual reformulation of Wasserstein distance known as Connes' Distance Formula

Low-Rank Subspace Override for Unsupervised Domain Adaptation

Current supervised learning models cannot generalize well across domain boundaries, which is a known problem in many applications, such as robotics or visual classification. Domain adaptation methods are used to improve these generalization properties. However, these techniques suffer either from being restricted to a particular task, such as visual adaptation, require a lot of computational time and data, which is not always guaranteed, have complex parameterization, or expensive optimization procedures. In this work, we present an approach that requires only a well-chosen snapshot of data to find a single domain invariant subspace. The subspace is calculated in closed form and overrides domain structures, which makes it fast and stable in parameterization. By employing low-rank techniques, we emphasize on descriptive characteristics of data. The presented idea is evaluated on various domain adaptation tasks such as text and image classification against state of the art domain adaptation approaches and achieves remarkable performance across all tasks

Manifold Learning by Mixture Models of VAEs for Inverse Problems

Representing a manifold of very high-dimensional data with generative models has been shown to be computationally efficient in practice. However, this requires that the data manifold admits a global parameterization. In order to represent manifolds of arbitrary topology, we propose to learn a mixture model of variational autoencoders. Here, every encoder-decoder pair represents one chart of a manifold. We propose a loss function for maximum likelihood estimation of the model weights and choose an architecture that provides us the analytical expression of the charts and of their inverses. Once the manifold is learned, we use it for solving inverse problems by minimizing a data fidelity term restricted to the learned manifold. To solve the arising minimization problem we propose a Riemannian gradient descent algorithm on the learned manifold. We demonstrate the performance of our method for low-dimensional toy examples as well as for deblurring and electrical impedance tomography on certain image manifolds

Generalized Bures-Wasserstein Geometry for Positive Definite Matrices

This paper proposes a generalized Bures-Wasserstein (BW) Riemannian geometry for the manifold of symmetric positive definite matrices. We explore the generalization of the BW geometry in three different ways: 1) by generalizing the Lyapunov operator in the metric, 2) by generalizing the orthogonal Procrustes distance, and 3) by generalizing the Wasserstein distance between the Gaussians. We show that they all lead to the same geometry. The proposed generalization is parameterized by a symmetric positive definite matrix $\mathbf{M}$ such that when $\mathbf{M} = \mathbf{I}$, we recover the BW geometry. We derive expressions for the distance, geodesic, exponential/logarithm maps, Levi-Civita connection, and sectional curvature under the generalized BW geometry. We also present applications and experiments that illustrate the efficacy of the proposed geometry

BIOMOLECULAR FUNCTION FROM STRUCTURAL SNAPSHOTS

Biological molecules can assume a continuous range of conformations during function. Near equilibrium, the Boltzmann relation connects a particular conformation\u27s free energy to the conformation\u27s occupation probability, thus giving rise to one or more energy landscapes. Biomolecular function proceeds along minimum-energy pathways on such landscapes. Consequently, a comprehensive understanding of biomolecular function often involves the determination of the free-energy landscapes and the identification of functionally relevant minimum-energy conformational paths on these landscapes. Specific techniques are necessary to determine continuous conformational spectra and identify functionally relevant conformational trajectories from a collection of raw single-particle snapshots from, e.g. cryogenic electron microscopy (cryo-EM) or X-ray diffraction. To assess the capability of different algorithms to recover conformational landscapes, we:• Measure, compare, and benchmark the performance of four leading data-analytical approaches to determine the accuracy with which energy landscapes are recovered from simulated cryo-EM data. Our simulated data are derived from projection directions along the great circle, emanating from a known energy landscape. • Demonstrate the ability to recover a biomolecule\u27s energy landscapes and functional pathways of biomolecules extracted from collections of cryo-EM snapshots. Structural biology applications in drug discovery and molecular medicine highlight the importance of the free-energy landscapes of the biomolecules more crucial than ever. Recently several data-driven machine learning algorithms have emerged to extract energy landscapes and functionally relevant continuous conformational pathways from single-particle data (Dashti et al., 2014; Dashti et al., 2020; Mashayekhi,et al., 2022). In a benchmarking study, the performance of several advanced data-analytical algorithms was critically assessed (Dsouza et al., 2023). In this dissertation, we have benchmarked the performance of four leading algorithms in extracting energy landscapes and functional pathways from single-particle cryo-EM snapshots. In addition, we have significantly improved the performance of the ManifoldEM algorithm, which has demonstrated the highest performance. Our contributions can be summarized as follows.: • Expert user supervision is required in one of the main steps of the ManifoldEM framework wherein the algorithm needs to propagate the conformational information through all angular space. We have succeeded in introducing an automated approach, which eliminates the need for user involvement. • The quality of the energy landscapes extracted by ManifoldEM from cryo-EM data has been improved, as the accuracy scores demonstrate this improvement. These measures have substantially enhanced ManifoldEM’s ability to recover the conformational motions of biomolecules by extracting the energy landscape from cryo-EM data.In line with the primary goal of our research, we aimed to extend the automated method across the entire angular sphere rather than a great circle. During this endeavor, we encountered challenges, particularly with some projection directions not following the proposed model. Through methodological adjustments and sampling optimization, we improved the projection direction\u27s conformity to the model. However, a small subset of Projection directions (5 %) remained challenging. We also recommended the use of specific methodologies, namely feature extraction and edge detection algorithms, to enhance the precision in quantifying image differentiation, a crucial component of our automated model. we also suggested that integrating different techniques might potentially resolve challenges associated with certain projection directions. We also applied ManifoldEM to experimental cryo-EM images of the SARS-CoV-2 spike protein in complex with the ACE2 receptor. By introducing several improvements, such as the incorporation of an adaptive mask and cosine curve fitting, we enhanced the framework\u27s output quality. This enhancement can be quantified by observing the removal of the artifact from the energy landscape, especially if the post-enhancement landscape differs from the artifact-affected one. These modifications, specifically aimed at addressing challenges from Nonlinear Laplacian Spectral Analysis (NLSA) (Giannakis et al., 2012), are intended for application in upcoming cryo-EM studies utilizing ManifoldEM. In the closing sections of this dissertation, a summary and a projection of future research directions are provided. While initial automated methods have been explored, there remains room for refinement. We have offered numerous methodological suggestions oriented toward addressing solutions to the challenge of conformational information propagation. Key methodologies discussed include Manifold Alignment, Canonical Correlation Analysis, and Multi-View Diffusion Maps. These recommendations are aimed to inform and guide subsequent developments in the ManifoldEM suite