End-to-End Zero-Shot Voice Conversion with Location-Variable Convolutions

Zero-shot voice conversion is becoming an increasingly popular research direction, as it promises the ability to transform speech to match the vocal identity of any speaker. However, relatively little work has been done on end-to-end methods for this task, which are appealing because they remove the need for a separate vocoder to generate audio from intermediate features. In this work, we propose LVC-VC, an end-to-end zero-shot voice conversion model that uses location-variable convolutions (LVCs) to jointly model the conversion and speech synthesis processes with a small number of parameters. LVC-VC utilizes carefully designed input features that have disentangled content and speaker style information, and the neural vocoder-like architecture learns to combine them to perform voice conversion while simultaneously synthesizing audio. Experiments show that our model achieves competitive or better voice conversion performance compared to several baselines while maintaining intelligibility particularly well

Spectral Entropy Feature in Full-Combination Multi-stream for Robust ASR

In a recent paper, we reported promising automatic speech recognition results obtained by appending spectral entropy features to PLP features. In the present paper, spectral entropy features are used along with PLP features in the framework of multi-stream combination. In a full-combination multi-stream hidden Markov model/artificial neural network (HMM/ANN) hybrid system, we train a separate multi-layered perceptron (MLP) for PLP features, for spectral entropy features and for both combined by concatenation. The output posteriors from these three MLPs are combined with weights inversely proportional to the entropies of their respective posterior distributions. We show that on the Numbers95 database, this approach yields a significant improvement under both clean and noisy conditions as compared to simply appending the features. Further, in the framework of a Tandem HMM/ANN system, we apply the same inverse entropy weighting to combine the outputs of the MLPs before the softmax non-linearity. Feeding the combined and decorrelated MLP outputs to the HMM gives a 9.2\% relative error reduction as compared to the baseline

Multi-stream adaptive evidence combination for noise robust ASR

In this paper we develop different mathematical models in the framework of the multi-stream paradigm for noise robust ASR, and discuss their close relationship with human speech perception. Largely inspired by Fletcher's "product-of-errors" rule in psychoacoustics, multi-band ASR aims for robustness to data mismatch through the exploitation of spectral redundancy, while making minimum assumptions about noise type. Previous ASR tests have shown that independent sub-band processing can lead to decreased recognition performance with clean speech. We have overcome this problem by considering every combination of data sub-bands as an independent data stream. After introducing the background to multi-band ASR, we show how this "full combination" approach can be formalised, in the context of HMM/ANN based ASR, by introducing a latent variable to specify which data sub-bands in each data frame are free from data mismatch. This enables us to decompose the posterior probability for each phoneme into a reliability weighted integral over all possible positions of clean data. This approach offers great potential for adaptation to rapidly changing and unpredictable noise

Discriminative connectionist approaches for automatic speech recognition in cars

The first part of this thesis is devoted to the evaluation of approaches which exploit the inherent redundancy of the speech signal to improve the noise robustness. On the basis of this evaluation on the AURORA 2000 database, we further study in detail two of the evaluated approaches. The first of these approaches is the hybrid RBF/HMM approach, which is an attempt to combine the superior classification performance of radial basis functions (RBFs) with the ability of HMMs to model time variation. The second approach is using neural networks to non-linearly reduce the dimensionality of large feature vectors including context frames. We propose the use of different MLP topologies for that purpose. Experiments on the AURORA 2000 database reveal that the performance of the first approach is similar to the performance of systems based on SCHMMs. The second approach cannot outperform the performance of linear discriminant analysis (LDA) on a database recorded in real car environments, but it is on average significantly better than LDA on the AURORA 2000 database.Im ersten Teil dieser Arbeit werden bestehende Verfahren zur Erhöhung der Robustheit von Spracherkennungssystemen in lauten Umgebungen evaluiert, die auf der Ausnutzung der Redundanz im Sprachsignal basieren. Auf der Grundlage dieser Evaluation auf der AURORA 2000 Datenbank werden zwei spezielle Ansätze weiter ausgearbeitet und detalliert analysiert. Der erste dieser Ansätze verbindet die herausragende Klassifikationsleistung von neuronalen Netzen mit radialen Basisfunktionen (RBF) mit der Fähigkeit von Hidden-Markov-Modellen (HMM), Zeitveränderlichkeiten zu modellieren. In einem zweiten Ansatz werden NN zur nichtlinearen Dimensionsreduktion hochdimensionaler Kontextvektoren in unterschiedlichen Netzwerk-Topologien untersucht. In Experimenten konnte gezeigt werden, dass der erste dieser Ansätze für die AURORA-Datenbank eine ähnliche Leistungsfähigkeit wie semikontinuierliche HMM (SCHMM) aufweist. Der zweite Ansatz erzielt auf einer im Kraftfahrzeug aufgenommenen Datenbank keine Verbesserung gegenüber den klassischen linearen Ansätzen zu Dimensionsreduktion (LDA), erweist sich aber auf der AURORA-Datenbank als signifikan

De novo design of multi-domain metalloenzymes

The course of evolution required the recombination of protein domains to perform ever-growing complex functions. The presence of an additional domain in a multi-domain protein expands, alters, or modulates the functionality with respect to the isolated one-domain protein (1). In particular, small molecule binding domains have shown a strong propensity to form multi-domain proteins and regulate enzymatic, transport, and signal-transducing domains (2). This modulation is referred to as allostery (from Greek, other solid body), as the properties of a functional site are affected by a small molecule bound to a distinctive protein site (3). Taking inspiration from Nature, artificial proteins have been engineered combining different domains to develop bioinspired molecular machines, able to respond to external stimuli (4). This Ph.D. project, born from the collaboration of the Artificial Metallo-Enzyme Group and the DeGradoLab, was devoted to the development of a multi-domain protein. This represents the first example of an artificial multi-domain protein, in which allostery was designed completely from scratch (5,6). DF (Due Ferri), a diiron phenol oxidase domain, and PS (Porphyrin-binding Sequence), a zinc porphyrin binding domain, were selected as starting proteins to be combined and give DFP (Due Ferri Porphyrin).7 The multiple junctions were exploited to link the two domains, and obtain a more extensive structural coupling between them. While the two metalloproteins present the same kind of domain, the two four-helix bundles are characterized by different geometrical parameters. Therefore, a structural-based methodology was firstly developed in order to identify the best colocalization and helical junctions to accommodate the changes in interhelical separation and registry between the bundles. The x-ray structure of the first analogue, DFP1, was determined, bound to its metal cofactors. The superposition of the 120 residues comprising binding sites gave an excellent fit to the design model, with an overall backbone RMSD of less than 1.4 Å. However, DFP1 was designed to maximize structural stability with a tight and uniform packing, which hindered the access to organic substrates at the DF domain and, thus, its functional characterization. The channel-lining residues of the dimetal-binding site in DF domain were mutated in Gly residues to create a pocket for a substrate. The introduction of helix-breaking residues, that gave oligomerization promiscuity, required also the mutation of DF loop, leading to the final candidate DFP3. An extensive spectroscopic characterization was performed to investigate the functional properties of the multi-domain proteins. DFP3 was demonstrated to bind the designed zinc porphyrin ZnP (Zn-meso-(trifluoromethyl)porphin) at the PS domain with nanomolar affinity. The strong negative Cotton Effect in the ZnP Soret region confirmed the tight and single-mode binding in the rigid asymmetric protein core. On the other side of the multi-domain metalloprotein, cobalt binding experiments confirmed the preservation of the DF penta-coordinating environment. Indeed, the dizinc form was able to stabilize the semiquinone form of 3,5-ditertbutylcatechol/quinone couple, and DFP3 showed ferroxidase and phenoloxidase activities. Although these reactivities were still present upon ZnP binding, a modulation effect was observed. The catalytic characterization of 4-aminophenol oxidation demonstrated a Michaelis-Menten mechanism in the phenoloxidase activity, and high-lightened a 4-fold tighter Km and a 7-fold decrease in kcat upon binding of ZnP. Molecular Dynamics simulations suggested that the presence of ZnP restrains the conformational freedom of a second-shell Tyr, that have been previously shown to largely affect the reactivity of the diiron center. Subsequently, the binding fitness of the zinc porphyrin was changed to investigate the bidirectionality of the allosteric regulation. In the presence of the different zinc porphyrin ZnDP (ZnDP, Zn-Deuteroporphyrin IX), DFP3 resulted to be more flexible, as demonstrated by thermal and chemical denaturations. Nevertheless, the dizinc center continued to stabilize the seminiquinone, and the ferroxidase and phenol oxidase activities were still modulated by the presence of ZnDP. DFP3 showed an excellent affinity for ZnDP, only one order lower in magnitude compared to the designed ZnP. More importantly, the ZnDP affinity was modulated by the presence of zinc bound to DFP3, showing a 3-fold decrease in KD, and demonstrating the presence of a back-regulation. In final instance, the photosensitizing properties of zinc porphyrin-DFP3 complexes were tested in the oxidation of the biological redox cofactor NADH. The photocatalytic characterization highlighted the paramount role of the protein scaffold not only in increasing the reaction rate, but also in protecting the zinc porphyrins from highly reactive species. The lower binding fitness DFP3 towards ZnDP hindered this protection, enabling a major permeability of these species and leading to the zinc porphyrin photobleaching. Although only a preliminary characterization of photocatalysis has been performed, the high reactivity and versatility of such systems are a promising starting point for the de novo design of artificial photosystems for the storage of light energy in chemical fuels.   References (1) Bashton, M. & Chothia, C. The Generation of New Protein Functions by the Combination of Domains. Structure 15, 85–99 (2007). (2) Anantharaman, V., Koonin, E. V. & Aravind, L. Regulatory potential, phyletic distribution and evolution of ancient, intracellular small-molecule-binding domains11Edited by F. Cohen. J. Mol. Biol. 307, 1271–1292 (2001). (3) Monod, J., Wyman, J. & Changeux, J.-P. On the nature of allosteric transitions: A plausible model. J. Mol. Biol. 12, 88–118 (1965). (4) Ostermeier, M. Engineering allosteric protein switches by domain insertion. Protein Eng. Des. Sel. 18, 359–364 (2005). (5) Researchers design allosteric protein from scratch. Chemical & Engineering News https://cen.acs.org/biological-chemistry/Researchers-design-allosteric-protein-scratch/98/i48. 6. Pirro, F. et al. Allosteric cooperation in a de novo-designed two-domain protein. Proc. Natl. Acad. Sci. 117, 33246–33253 (2020). (7) Lombardi, A., Pirro, F., Maglio, O., Chino, M. & DeGrado, W. F. De Novo Design of Four-Helix Bundle Metalloproteins: One Scaffold, Diverse Reactivities. Acc. Chem. Res. 52, 1148–1159 (2019)