Proximal Methods for Hierarchical Sparse Coding

Sparse coding consists in representing signals as sparse linear combinations of atoms selected from a dictionary. We consider an extension of this framework where the atoms are further assumed to be embedded in a tree. This is achieved using a recently introduced tree-structured sparse regularization norm, which has proven useful in several applications. This norm leads to regularized problems that are difficult to optimize, and we propose in this paper efficient algorithms for solving them. More precisely, we show that the proximal operator associated with this norm is computable exactly via a dual approach that can be viewed as the composition of elementary proximal operators. Our procedure has a complexity linear, or close to linear, in the number of atoms, and allows the use of accelerated gradient techniques to solve the tree-structured sparse approximation problem at the same computational cost as traditional ones using the L1-norm. Our method is efficient and scales gracefully to millions of variables, which we illustrate in two types of applications: first, we consider fixed hierarchical dictionaries of wavelets to denoise natural images. Then, we apply our optimization tools in the context of dictionary learning, where learned dictionary elements naturally organize in a prespecified arborescent structure, leading to a better performance in reconstruction of natural image patches. When applied to text documents, our method learns hierarchies of topics, thus providing a competitive alternative to probabilistic topic models

Learning visual representations with neural networks for video captioning and image generation

La recherche sur les réseaux de neurones a permis de réaliser de larges progrès durant la dernière décennie. Non seulement les réseaux de neurones ont été appliqués avec succès pour résoudre des problèmes de plus en plus complexes; mais ils sont aussi devenus l’approche dominante dans les domaines où ils ont été testés tels que la compréhension du langage, les agents jouant à des jeux de manière automatique ou encore la vision par ordinateur, grâce à leurs capacités calculatoires et leurs efficacités statistiques. La présente thèse étudie les réseaux de neurones appliqués à des problèmes en vision par ordinateur, où les représentations sémantiques abstraites jouent un rôle fondamental. Nous démontrerons, à la fois par la théorie et par l’expérimentation, la capacité des réseaux de neurones à apprendre de telles représentations à partir de données, avec ou sans supervision. Le contenu de la thèse est divisé en deux parties. La première partie étudie les réseaux de neurones appliqués à la description de vidéo en langage naturel, nécessitant l’apprentissage de représentation visuelle. Le premier modèle proposé permet d’avoir une attention dynamique sur les différentes trames de la vidéo lors de la génération de la description textuelle pour de courtes vidéos. Ce modèle est ensuite amélioré par l’introduction d’une opération de convolution récurrente. Par la suite, la dernière section de cette partie identifie un problème fondamental dans la description de vidéo en langage naturel et propose un nouveau type de métrique d’évaluation qui peut être utilisé empiriquement comme un oracle afin d’analyser les performances de modèles concernant cette tâche. La deuxième partie se concentre sur l’apprentissage non-supervisé et étudie une famille de modèles capables de générer des images. En particulier, l’accent est mis sur les “Neural Autoregressive Density Estimators (NADEs), une famille de modèles probabilistes pour les images naturelles. Ce travail met tout d’abord en évidence une connection entre les modèles NADEs et les réseaux stochastiques génératifs (GSN). De plus, une amélioration des modèles NADEs standards est proposée. Dénommés NADEs itératifs, cette amélioration introduit plusieurs itérations lors de l’inférence du modèle NADEs tout en préservant son nombre de paramètres. Débutant par une revue chronologique, ce travail se termine par un résumé des récents développements en lien avec les contributions présentées dans les deux parties principales, concernant les problèmes d’apprentissage de représentation sémantiques pour les images et les vidéos. De prometteuses directions de recherche sont envisagées.The past decade has been marked as a golden era of neural network research. Not only have neural networks been successfully applied to solve more and more challenging real- world problems, but also they have become the dominant approach in many of the places where they have been tested. These places include, for instance, language understanding, game playing, and computer vision, thanks to neural networks’ superiority in computational efficiency and statistical capacity. This thesis applies neural networks to problems in computer vision where high-level and semantically meaningful representations play a fundamental role. It demonstrates both in theory and in experiment the ability to learn such representations from data with and without supervision. The main content of the thesis is divided into two parts. The first part studies neural networks in the context of learning visual representations for the task of video captioning. Models are developed to dynamically focus on different frames while generating a natural language description of a short video. Such a model is further improved by recurrent convolutional operations. The end of this part identifies fundamental challenges in video captioning and proposes a new type of evaluation metric that may be used experimentally as an oracle to benchmark performance. The second part studies the family of models that generate images. While the first part is supervised, this part is unsupervised. The focus of it is the popular family of Neural Autoregressive Density Estimators (NADEs), a tractable probabilistic model for natural images. This work first makes a connection between NADEs and Generative Stochastic Networks (GSNs). The standard NADE is improved by introducing multiple iterations in its inference without increasing the number of parameters, which is dubbed iterative NADE. With a historical view at the beginning, this work ends with a summary of recent development for work discussed in the first two parts around the central topic of learning visual representations for images and videos. A bright future is envisioned at the end

The Influence of Structural Constraints on Protein Evolution

Few mathematical models of sequence evolution incorporate parameters describingprotein structure, despite its high conservation, essential functional role and the increasingavailability of structural data. The primary goal of my PhD project was to create astructurally aware amino acid substitution model in which proteins are represented usingan expanded alphabet that relays both amino acid identity and structural information.Each character in this alphabet specifies an amino acid as well as information aboutthe rotamer configuration of its side chain: the discrete geometric pattern of permittedside chain atomic positions, as defined by the dihedral angles between covalently linkedatoms. I generated a 55-state “Dayhoff-like” substitution model (RAM55) by assigningrotamer states in 79,558 structures (∼50%of all PDBe entries) and identifying substitu-tions between closely related sequences. RAM55’s rotamer state exchange patterns clearlyshow that the evolutionary properties of amino acids depend strongly upon side chain ge-ometry. Exploiting knowledge of these patterns assists in phylogenetic analyses: I showthat RAM55 performs as well as or better than traditional 20-state models on simulatedand empirical data for divergence time estimation, tree inference, side chain configurationprediction and ancestral sequence reconstruction.Further, encoding observed characters in an alignment as ambiguous representations ofcharacters in a larger state-space allows the application of RAM55 to 20-state amino aciddata for which structures are not known. Adding structural information to as few as12.5%of the sequences in an amino acid alignment results in excellent ancestral reconstructionperformance compared to a benchmark that considers the full rotamer state information.This strategy significantly expands the applicability of RAM55 to real-world scenarioswhere structure might only be available for some of the sequences of interest.Thus, not only is rotamer configuration a valuable source of information for phylo-genetic studies, but modelling the concomitant evolution of sequence and structure mayhave important implications for understanding protein folding and function

MAUVE Scores for Generative Models: Theory and Practice

Generative artificial intelligence has made significant strides, producing text indistinguishable from human prose and remarkably photorealistic images. Automatically measuring how close the generated data distribution is to the target distribution is central to diagnosing existing models and developing better ones. We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images. These scores are statistical summaries of divergence frontiers capturing two types of errors in generative modeling. We explore three approaches to statistically estimate these scores: vector quantization, non-parametric estimation, and classifier-based estimation. We provide statistical bounds for the vector quantization approach. Empirically, we find that the proposed scores paired with a range of $f$-divergences and statistical estimation methods can quantify the gaps between the distributions of human-written text and those of modern neural language models by correlating with human judgments and identifying known properties of the generated texts. We demonstrate in the vision domain that MAUVE can identify known properties of generated images on par with or better than existing metrics. In conclusion, we present practical recommendations for using MAUVE effectively with language and image modalities.Comment: Published in Journal of Machine Learning Researc

The Role of Mutations in Protein Structural Dynamics and Function: A Multi-scale Computational Approach

abstract: Proteins are a fundamental unit in biology. Although proteins have been extensively studied, there is still much to investigate. The mechanism by which proteins fold into their native state, how evolution shapes structural dynamics, and the dynamic mechanisms of many diseases are not well understood. In this thesis, protein folding is explored using a multi-scale modeling method including (i) geometric constraint based simulations that efficiently search for native like topologies and (ii) reservoir replica exchange molecular dynamics, which identify the low free energy structures and refines these structures toward the native conformation. A test set of eight proteins and three ancestral steroid receptor proteins are folded to 2.7Å all-atom RMSD from their experimental crystal structures. Protein evolution and disease associated mutations (DAMs) are most commonly studied by in silico multiple sequence alignment methods. Here, however, the structural dynamics are incorporated to give insight into the evolution of three ancestral proteins and the mechanism of several diseases in human ferritin protein. The differences in conformational dynamics of these evolutionary related, functionally diverged ancestral steroid receptor proteins are investigated by obtaining the most collective motion through essential dynamics. Strikingly, this analysis shows that evolutionary diverged proteins of the same family do not share the same dynamic subspace. Rather, those sharing the same function are simultaneously clustered together and distant from those functionally diverged homologs. This dynamics analysis also identifies 77% of mutations (functional and permissive) necessary to evolve new function. In silico methods for prediction of DAMs rely on differences in evolution rate due to purifying selection and therefore the accuracy of DAM prediction decreases at fast and slow evolvable sites. Here, we investigate structural dynamics through computing the contribution of each residue to the biologically relevant fluctuations and from this define a metric: the dynamic stability index (DSI). Using DSI we study the mechanism for three diseases observed in the human ferritin protein. The T30I and R40G DAMs show a loss of dynamic stability at the C-terminus helix and nearby regulatory loop, agreeing with experimental results implicating the same regulatory loop as a cause in cataracts syndrome.Dissertation/ThesisPh.D. Physics 201

ICR ANNUAL REPORT 2022 (Volume 29)[All Pages]

This Annual Report covers from 1 January to 31 December 202

Nuclear receptors in the Pacific oyster, Crassostrea gigas, as screening tool for determining response to environmental contaminants.

Marine environments are under constant pressure from anthropogenic pollution. Chemical pollutants are introduced into the aquatic environment through waste disposal, sewage, land runoff and environmental exploitation (harbours, fisheries, tourism) leading to disastrous effects on the marine wildlife. Developmental malformations, reproduction failure including sex changes and high death rates are commonly observed in aquatic animal populations around the world. Unfortunately, the underlying molecular mechanisms of these pollution effects, in particular for marine invertebrate species, are often unknown. One proposed mechanism through which environmental pollution affects wildlife, is the disruption of nuclear receptors (NRs), ligand-binding transcription factors in animals. Environmental pollutants can directly interact with nuclear receptors, inducing incorrect signals for gene expression and subsequently disrupt developmental and physiological processes. Elucidation of the exact mechanism in invertebrates, however, is sparse due to limited understanding of invertebrate endocrinology and molecular regulatory mechanisms. Here, I have investigated the presence, expression and function of NRs in the Pacific oyster, Crassostrea gigas, and explored their interrelation with known environmental pollutants. Using a suite of molecular techniques and bioinformatics tools I demonstrate that the Pacific oyster possesses a large variety of NR homologs (43 NRs), which display individual expression profiles during embryo/larval development and supposedly fulfil distinct functions in developmental and physiological processes. Functional studies on a small subset of oyster NRs provided evidence for their ability to regulate gene expression, including interactions with DNA, other NRs or small molecules (ligand-binding). Oyster receptors also show a high likeliness to be disrupted by environmental pollutants. Computational docking showed that the retinoid X receptor ortholog, CgRXR, is able to bind and be activated by 9-cis retinoic acid and by the well-known environmental contaminant tributyltin. A potential interaction between tributyltin and the peroxisome proliferator-activated receptor ortholog CgPPAR has also been found. In addition, exposure of oyster embryos to retinoic acids and tributyltin resulted in shell deformations and developmental failure. In contrast, computer modelling of another putative target for pollutants, the retinoic acid receptor ortholog CgRAR, did not indicate interactions with common retinoic acids, supporting a recently developed theory of loss of retinoid binding in molluscan RARs. Sequence analyses revealed six residues in the receptor sequence, which prevent the successful interaction with retinoid ligands. In conclusion, this investigative work aids the understanding of fundamental processes in invertebrates, such as gene expression and endocrinology, as well as further understanding and prediction of effects of environmental pollutants on marine invertebrates.Funded by the University of Exeter and the Centre for Environment, Fisheries and Aquaculture Scienc

Probabilistic methods in the analysis of protein interaction networks

Imperial Users onl

2014 Summer Research Symposium Abstract Book

2014 Summer volume of abstracts for science research projects conducted by students at Trinity College