Disentangled generative models for robust prediction of system dynamics

The use of deep neural networks for modelling system dynamics is increasingly popular, but long-term prediction accuracy and out-of-distribution generalization still present challenges. In this study, we address these challenges by considering the parameters of dynamical systems as factors of variation of the data and leverage their ground-truth values to disentangle the representations learned by generative models. Our experimental results in phase-space and observation-space dynamics, demonstrate the effectiveness of latent-space supervision in producing disentangled representations, leading to improved long-term prediction accuracy and out-of-distribution robustness

On Contrastive Learning for Likelihood-free Inference

Likelihood-free methods perform parameter inference in stochastic simulator models where evaluating the likelihood is intractable but sampling synthetic data is possible. One class of methods for this likelihood-free problem uses a classifier to distinguish between pairs of parameter-observation samples generated using the simulator and pairs sampled from some reference distribution, which implicitly learns a density ratio proportional to the likelihood. Another popular class of methods fits a conditional distribution to the parameter posterior directly, and a particular recent variant allows for the use of flexible neural density estimators for this task. In this work, we show that both of these approaches can be unified under a general contrastive learning scheme, and clarify how they should be run and compared.Comment: Appeared at ICML 202

Advances in Probabilistic Meta-Learning and the Neural Process Family

A natural progression in machine learning research is to automate and learn from data increasingly many components of our learning agents.Meta-learning is a paradigm that fully embraces this perspective, and can be intuitively described as embodying the idea of learning to learn. A goal of meta-learning research is the development of models to assist users in navigating the intricate space of design choices associated with specifying machine learning solutions. This space is particularly formidable when considering deep learning approaches, which involve myriad design choices interacting in complex fashions to affect the performance of the resulting agents. Despite the impressive successes of deep learning in recent years, this challenge remains a significant bottleneck in deploying neural network based solutions in several important application domains. But how can we reason about and design solutions to this daunting task? This thesis is concerned with a particular perspective for meta-learning in supervised settings. We view supervised learning algorithms as mappings that take data sets to predictive models, and consider meta-learning as learning to approximate functions of this form. In particular, we are interested in meta-learners that (i) employ neural networks to approximate these functions in an end-to-end manner, and (ii) provide predictive distributions rather than single predictors. The former is motivated by the success of neural networks as function approximators, and the latter by our interest in the few-shot learning scenario. The introductory chapters of this thesis formalise this notion, and use it to provide a tutorial introducing the Neural Process Family (NPF), a class of models introduced by Garnelo et al (2018) satisfying the above-mentioned modelling desiderata. We then present our own technical contributions to the NPF. First, we focus on fundamental properties of the model-class, such as expressivity and limiting behaviours of the associated training procedures. Next, we study the role of translation equivariance in the NPF. Considering the intimate relationship between the NPF and the representation of functions operating on sets, we extend the underlying theory of DeepSets to include translation equivariance. We then develop novel members of the NPF endowed with this important inductive bias. Through extensive empirical evaluation, we demonstrate that, in many settings, they significantly outperform their non-equivariant counterparts. Finally, we turn our attention to the development of Neural Processes for few-shot image-classification. We introduce models that navigate the important tradeoffs associated with this setting, and describe the specification of their central components. We demonstrate that the resulting models---CNAPs---achieve state-of-the-art performance on a challenging benchmark called Meta-Dataset, while adapting faster and with less computational overhead than their best-performing competitors

Detecting Causality in complex systems

In this work, recent and classical results on causality detection and predicability of a complex system have been reviewed critically. In the first part, I have extensively studied the Convergent Cross Mapping method [1] and I testedin cases of particular interest. I have also confronted this approach with the classical Granger framework for causality [2]. A study with a simulated Lotka-Volterra model has shown certain limits of this method, and I have obtained counterintuitive results. In the second part of the work, I have made a description of the Analog method [3] to perform prediction on complex systems. I have also studied the theorem on which this approach is rooted, the Takens’ theorem. Moreover, we have investigated the main limitation of this approach, known as the curse of dimensionality: when the system increases in dimensionality the number of data needed to obtain sufficiently accurate predictions scales exponentially with dimension. Additionally, finding the effective dimension of a complex systemstill an open problem. I have presented methods to estimate the effective dimension of the attractor of dynamical systems known as Grassberger-Procaccia algorithm and his extensions [4]. Finally, I have tested all these data driven machineries to a well-studied dynamical system, i.e. the Lorenz system, finding that the theoretical results are confirmed and the data needed scales as an exponential law N~Єd.ope

Automated Telescience: Active Machine Learning Of Remote Dynamical Systems

Automated science is an emerging field of research and technology that aims to extend the role of computers in science from a tool that stores and analyzes data to one that generates hypotheses and designs experiments. Despite the tremendous discoveries and advancements brought forth by the scientific method, it is a process that is fundamentally driven by human insight and ingenuity. Automated science aims to develop algorithms, protocols and design philosophies that are capable of automating the scientific process. This work presents advances the field of automated science and the specific contributions of this work fall into three categories: coevolutionary search methods and applications, inferring the underlying structure of dynamical systems, and remote controlled automated science. First, a collection of coevolutionary search methods and applications are presented. These approaches include: a method to reduce the computational overhead of evolutionary algorithms via trainer selection strategies in a rank predictor framework, an approach for optimal experiment design for nonparametric models using Shannon information, and an application of coevolutionary algorithms to infer kinematic poses from RGBD images. Second, three algorithms are presented that infer the underlying structure of dynamical systems: a method to infer discrete-continuous hybrid dynamical systems from unlabeled data, an approach to discovering ordinary differential equations of arbitrary order, and a principle to uncover the existence and dynamics of hidden state variables that correspond to physical quantities from nonlinear differential equations. All of these algorithms are able to uncover structure in an unsupervised manner without any prior domain knowledge. Third, a remote controlled, distributed system is demonstrated to autonomously generate scientific models by perturbing and observing a system in an intelligent fashion. By automating the components of physical experimentation, scientific modeling and experimental design, models of luminescent chemical reactions and multi-compartmental pharmacokinetic systems were discovered without any human intervention, which illustrates how a set of distributed machines can contribute scientific knowledge while scaling beyond geographic constraints

Representation and decision making in the immune system

The immune system has long been attributed cognitive capacities such as "recognition" of pathogenic agents; "memory" of previous infections; "regulation" of a cavalry of detector and effector cells; and "adaptation" to a changing environment and evolving threats. Ostensibly, in preventing disease the immune system must be capable of discriminating states of pathology in the organism; identifying causal agents or ``pathogens''; and correctly deploying lethal effector mechanisms. What is more, these behaviours must be learnt insomuch as the paternal genes cannot encode the pathogenic environment of the child. Insights into the mechanisms underlying these phenomena are of interest, not only to immunologists, but to computer scientists pushing the envelope of machine autonomy. This thesis approaches these phenomena from the perspective that immunological processes are inherently inferential processes. By considering the immune system as a statistical decision maker, we attempt to build a bridge between the traditionally distinct fields of biological modelling and statistical modelling. Through a mixture of novel theoretical and empirical analysis we assert the efficacy of competitive exclusion as a general principle that benefits both. For the immunologist, the statistical modelling perspective allows us to better determine that which is phenomenologically sufficient from the mass of observational data, providing quantitative insight that may offer relief from existing dichotomies. For the computer scientist, the biological modelling perspective results in a theoretically transparent and empirically effective numerical method that is able to finesse the trade-off between myopic greediness and intractability in domains such as sparse approximation, continuous learning and boosting weak heuristics. Together, we offer this as a modern reformulation of the interface between computer science and immunology, established in the seminal work of Perelson and collaborators, over 20 years ago.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Abstract economic modeling : a semantic-philosophical definition of economic models

As fundações análogas das ideias de Kuhn e da visão pragmática das teorias favorecem uma união de pensamentos. Na concepção de Kuhn após a Estrutura das Revoluções Científicas, a filosofia da linguagem – especialmente as teorias de uso da linguagem – e suas ramificações nas ciências cognitivas são formas efetivas de julgar problemas científicos. Baseados nessas novas ideias, os interpretes de Kuhn propuseram a teoria psicológica dos Enquadramentos Dinâmicos como uma forma funcional de reavaliar a evolução científica. Uma aplicação dessa teoria para reler as definições pragmáticas de modelos foi realizada nessa dissertação, expondo a incomparabilidade entre estudos de caso, o que impede o avanço das discussões. Consequentemente, a criação de definições comparáveis é necessária para o desenvolvimento dos debates pragmáticos. Inspirada em Sugden (2000;2009), a solução proposta foi a criação de paradigmas plausíveis. Seguindo esta linha de raciocínio, um exame da história do pensamento econômico foi realizado buscando uma fundação crível para a definição de modelos econômicos abstratos. A pesquisa identificou os trabalhos de Tinbergen (1935) e de Von Neumann (1945) como os primeiros a usarem o termo ‘modelo’ em sentido abstrato e, portanto, como uma fundação sólida para um paradigma definidor do termo modelo econômico no período que transcorre de 1930 à 1950. Em seguida, a combinação da teoria dos Enquadramentos Dinâmicos e dos exemplares resultou na definição de modelos econômicos contendo cinco características: adaptabilidade, neutralidade, estrutura matemática, simplificação e objetivo. Uma avaliação subsequente da disseminação do termo de 1930 até 1950 sugere que os exemplares escolhidos são uma fundação plausível, ainda que a definição não tenha sido instantânea nem completamente disseminada entre os economistas.The analogous foundations of Kuhnian ideas and of The Pragmatic View of Theories favor a union of thoughts. In Kuhn’s renewed ideas, philosophy of language – especially use theories - and its ramifications in cognitive sciences are an effective form of judging scientific conundrums. Based on this insight, Kuhn’s interpreters proposed the psychological theory of Dynamic Frames as a functional form of reviewing scientific evolution. An application of Dynamic Frames was realized to reread pragmatic definitions of models, exposing the incomparability between case-studies, which hampers the development of discussions. Consequently, the creation of comparable definitions is necessary for the advancement of pragmatic debates. Inspired by Sugden (2002; 2009), the proposed solution was the creation of plausible paradigms. Following this mode of reasoning, an examination of history of economic thought was realized searching for a credible foundation for the definition of abstract economic models. The exploration suggested Tinbergen’s (1935) and Von Neumann’s (1945) works as the first ones to use the term “model” in an abstract sense and thus as a solid foundation for a paradigm intended to define economic models. The following combination of Dynamic Frames ideas and the exemplars resulted in a definition of models containing five characteristics: adaptability; neutrality; mathematical structure; simplification; and objective. A subsequent examination of the dissemination of the term from 1930s to 1950s suggested the exemplars were a plausible foundation, even though the definition was neither instantly nor completely disseminated among economists

Révision automatique de théories écologiques

À l’origine, ce sont des difficultés en biologie évolutive qui ont motivé cette thèse. Après des décennies à tenter de trouver une théorie basée sur la sélection capable de prédire la diversité génomique, les théoriciens n’ont pas trouvé d’alternatives pratiques à la théorie neutre. Après avoir étudié la relation entre la spéciation et la diversité (Annexes A, B, C), j’ai conclu que l’approche traditionnelle pour construire des théories serait difficile à appliquer au problème de la biodiversité. Prenons par exemple le problème de la diversité génomique, la difficulté n’est pas que l’on ignore les mécanismes impliqués, mais qu’on ne réussit pas à construire de théories capable d’intégrer ces mécanismes. Les techniques en intelligence artificielle, à l’inverse, réussissent souvent à construire des modèles prédictifs efficaces justement là où les théories traditionnelles échouent. Malheureusement, les modèles bâtis par les intelligences arti- ficielles ne sont pas clairs. Un réseau de neurones peut avoir jusqu’à un milliard de paramètres. L’objectif principal de ma thèse est d’étudier des algorithmes capable de réviser les théories écologiques. L’intégration d’idées venant de différentes branches de l’écologie est une tâche difficile. Le premier défi de ma thèse est de trouver sous quelle représentation formelle les théories écologiques doivent être encodées. Contrairement aux mathématiques, nos théories sont rarement exactes. Il y a à la fois de l’incertitude dans les données que l’on collecte, et un flou dans nos théories (on ne s’attend pas à que la théorie de niche fonctionne 100% du temps). Contrairement à la physique, où un petit nombre de forces dominent la théorie, l’écologie a un très grand nombre de théories. Le deuxième défi est de trouver comment ces théories peuvent être révisées automatiquement. Ici, le but est d’avoir la clarté des théories traditionnelles et la capacité des algorithmes en intelligence artificielle de trouver de puissants modèles prédictifs à partir de données. Les tests sont faits sur des données d’interactions d’espèces

La résilience des réseaux complexes

Les systèmes réels subissant des perturbations par l’interaction avec leur environnement sont susceptibles d’être entraînés vers des transitions irréversibles de leur principal état d’activité. Avec la croissance de l’empreinte humaine mondiale sur les écosystèmes, la caractérisation de la résilience de ces systèmes complexes est un enjeu majeur du 21e siècle. Cette thèse s’intéresse aux systèmes complexes pour lesquels il existe un réseau d’interactions et où les composantes sont des variables dynamiques. L’étude de leur résilience exige la description de leurs états dynamiques qui peuvent avoir jusqu’à plusieurs milliers de dimensions. Cette thèse propose trois nouvelles méthodes permettant de faire des mesures de la dynamique en fonction de la structure du réseau. L’originalité de ce travail vient de la diversité des approches présentées pour traiter la résilience, en débutant avec des outils basés sur des modèles dynamiques définis et en terminant avec d’autres n’exploitant que des données récoltées. D’abord, une solution exacte à une dynamique de cascade (modèle de feu de forêt) est développée et accompagnée d’un algorithme optimisé. Comme sa portée pratique s’arrête aux petits réseaux, cette méthode signale les limitations d’une approche avec un grand nombre de dimensions. Ensuite, une méthode de réduction dimensionnelle est introduite pour établir les bifurcations dynamiques d’un système. Cette contribution renforce les fondements théoriques et élargit le domaine d’applications de méthodes existantes. Enfin, le problème de retracer l’origine structurelle d’une perturbation est traité au moyen de l’apprentissage automatique. La validité de l’outil est supportée par une analyse numérique sur des dynamiques de propagation, de populations d’espèces et de neurones. Les principaux résultats indiquent que de fines anomalies observées dans la dynamique d’un système peuvent être détectées et suffisent pour retracer la cause de la perturbation. L’analyse témoigne également du rôle que l’apprentissage automatique pourrait jouer dans l’étude de la résilience de systèmes réels.Real complex systems are often driven by external perturbations toward irreversible transitions of their dynamical state. With the rise of the human footprint on ecosystems, these perturbations will likely become more persistent so that characterizing resilience of complex systems has become a major issue of the 21st century. This thesis focuses on complex systems that exhibit networked interactions where the components present dynamical states. Studying the resilience of these networks demands depicting their dynamical portraits which may feature thousands of dimensions. In this thesis, three contrasting methods are introduced for studying the dynamical properties as a function of the network structure. Apart from the methods themselves, the originality of the thesis lies in the wide vision of resilience analysis, opening with model-based approaches and concluding with data-driven tools. We begin by developing an exact solution to binary cascades on networks (forest fire type) and follow with an optimized algorithm. Because its practical range is restricted to small networks, this method highlights the limitations of using model-based and highly dimensional tools. Wethen introduce a dimension reduction method to predict dynamical bifurcations of networked systems. This contribution builds up on theoretical foundations and expands possible applications of existing frameworks. Finally, we examine the task of extracting the structural causesof perturbations using machine learning. The validity of the developed tool is supported by an extended numerical analysis of spreading, population, and neural dynamics. The results indicate that subtle dynamical anomalies may suffice to infer the causes of perturbations. It also shows the leading role that machine learning may have to play in the future of resilience of real complex systems