Variational optimization of probability measure spaces resolves the chain store paradox

In game theory, players have continuous expected payoff functions and can use fixed point theorems to locate equilibria. This optimization method requires that players adopt a particular type of probability measure space. Here, we introduce alternate probability measure spaces altering the dimensionality, continuity, and differentiability properties of what are now the game's expected payoff functionals. Optimizing such functionals requires generalized variational and functional optimization methods to locate novel equilibria. These variational methods can reconcile game theoretic prediction and observed human behaviours, as we illustrate by resolving the chain store paradox. Our generalized optimization analysis has significant implications for economics, artificial intelligence, complex system theory, neurobiology, and biological evolution and development.Comment: 11 pages, 5 figures. Replaced for minor notational correctio

Variational optimization of probability measure spaces resolves the chain store paradox

In game theory, players have continuous expected payoff functions and can use fixed point theorems to locate equilibria. This optimization method requires that players adopt a particular type of probability measure space. Here, we introduce alternate probability measure spaces altering the dimensionality, continuity, and differentiability properties of what are now the game's expected payoff functionals. Optimizing such functionals requires generalized variational and functional optimization methods to locate novel equilibria. These variational methods can reconcile game theoretic prediction and observed human behaviours, as we illustrate by resolving the chain store paradox. Our generalized optimization analysis has significant implications for economics, artificial intelligence, complex system theory, neurobiology, and biological evolution and development.optimization; probability measure space; noncooperative game; chain store paradox

On Monte Carlo time-dependent variational principles

[no abstract

Image Segmentation using PDE, Variational, Morphological and Probabilistic Methods

The research in this dissertation has focused upon image segmentation and its related areas, using the techniques of partial differential equations, variational methods, mathematical morphological methods and probabilistic methods. An integrated segmentation method using both curve evolution and anisotropic diffusion is presented that utilizes both gradient and region information in images. A bottom-up image segmentation method is proposed to minimize the Mumford-Shah functional. Preferential image segmentation methods are presented that are based on the tree of shapes in mathematical morphologies and the Kullback-Leibler distance in information theory. A thorough evaluation of the morphological preferential image segmentation method is provided, and a web interface is described. A probabilistic model is presented that is based on particle filters for image segmentation. These methods may be incorporated as components of an integrated image processed system. The system utilizes Internet Protocol (IP) cameras for data acquisition. It utilizes image databases to provide prior information and store image processing results. Image preprocessing, image segmentation and object recognition are integrated in one stage in the system, using various methods developed in several areas. Interactions between data acquisition, integrated image processing and image databases are handled smoothly. A framework of the integrated system is implemented using Perl, C++, MySQL and CGI. The integrated system works for various applications such as video tracking, medical image processing and facial image processing. Experimental results on this applications are provided in the dissertation. Efficient computations such as multi-scale computing and parallel computing using graphic processors are also presented

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Quantum Markov processes and applications in many-body systems

Gegenstand der vorliegenden Arbeit ist die Untersuchung von quantenmechanischen und klassischen Markov-Prozessen und deren Anwendung im Bereich der stark korrelierten Vielteilchensysteme. Unter einem Markov-Prozess versteht man eine spezielle Art eines stochastischen Prozesses, dessen weitere dynamische Entwicklung unabhängig ist von der Vorgeschichte seiner Entwicklung und nur von der derzeitigen Konfiguration abhängt. Die Anwendung von Markov-Prozessen im Bereich der statistischen Mechanik von klassischen Vielteilchensystemen hat eine lange Geschichte. Markov-Prozesse dienen nicht nur der Beschreibung der Dynamik von stochastischen Systemen, sondern liefern vielmehr auch eine sehr praktische Methode, mit dereren Hilfe grundlegende Eigenschaften komplexer Vielteilchenprobleme in Form eines probabilistischen Algorithmus berechnet werden können. Ziel dieser Arbeit ist es das Verhalten von quantenmechanischen Markov Prozessen, dies sind Markov-Prozesse, welchen ein quantenmechanischer Konfigurationsraum zu Grunde liegt, zu untersuchen und mit deren Hilfe komplexe Vielteilchensysteme besser zu verstehen. Darüber hinaus formulieren wir einen Quantenalgorithmus, mit dessen Hilfe es möglich ist, die thermischen- und Grundzustandseigenschaften von quantenmechanischen Vielteilchensystemen zu berechnen. Nachdem wir eine kurze Einführung in das Feld der quantenmechanischen Markov-Prozesse gegeben haben, untersuchen wir deren Konvergenzeigenschaften. Wir finden Schranken für die Konvergenzraten der quantenmechanischen Prozesse, basierend auf einer Verallgemeinerung von geometrischen Schranken, welche für klassische Prozesse gefunden wurden. Wir verallgemeinern ein Abstandsmaß, die Chi-Quadrat Divergenz für nicht kommutative Wahrscheinlichkeitsräume, welches unseren Untersuchungen zu Grunde liegt. Diese Divergenz ermöglicht auch eine Verallgemeinerung der detaillierten Balance für quantenmechanische Prozesse. Danach konstruieren wir den Quantenalgorithmus, der als natürliche Verallgemeinerung des Metropolisalgorithmus für quantenmechanische Hamiltonoperatoren verstanden werden kann. Wir beabsichtigen damit zu zeigen, dass ein Quantencomputer in der Lage ist, als universeller Quantensimulator zu fungieren, welcher nicht nur die Dynamik eines Quantensystems beschreiben kann, sondern auch den Zugang zu statischen Berechnungen ermöglicht. Danach untersuchen wir die Korrelationseigenschaften von klassischen Nichtgleichgewichtszuständen mit Methoden der Quanteninformationstheorie. Wir konstruieren eine Klasse von Matrix-Produkt-Zuständen, deren Korrelationen anhand von klassischen Markov-Prozessen verstanden werden können. Schließlich untersuchen wir die Transporteigenschaften eines stationären Nichtgleichgewichtszustandes. Die dynamische Gleichung ist so konstruiert, dass der Transport je nach Parameterwahl entweder hauptsächlich stochastisch oder hauptsächlich kohärent stattfindet. Wir können somit die unterschiedlichen Formen des Transports innerhalb eines Modells miteinander vergleichen.This thesis is concerned with the investigation of quantum as well as classical Markov processes and their application in the field of strongly correlated many-body systems. A Markov process is a special kind of stochastic process, which is determined by an evolution that is independent of its history and only depends on the current state of the system. The application of Markov processes has a long history in the field of statistical mechanics and classical many-body theory. Not only are Markov processes used to describe the dynamics of stochastic systems, but they predominantly also serve as a practical method that allows for the computation of fundamental properties of complex many-body systems by means of probabilistic algorithms. The aim of this thesis is to investigate the properties of quantum Markov processes, i.e. Markov processes taking place in a quantum mechanical state space, and to gain a better insight into complex many-body systems by means thereof. Moreover, we formulate a novel quantum algorithm which allows for the computation of the thermal and ground states of quantum many-body systems. After a brief introduction to quantum Markov processes we turn to an investigation of their convergence properties. We find bounds on the convergence rate of the quantum process by generalizing geometric bounds found for classical processes. We generalize a distance measure that serves as the basis for our investigations, the chi-square divergence, to non-commuting probability spaces. This divergence allows for a convenient generalization of the detailed balance condition to quantum processes. We then devise the quantum algorithm that can be seen as the natural generalization of the ubiquitous Metropolis algorithm to simulate quantum many-body Hamiltonians. By this we intend to provide further evidence, that a quantum computer can serve as a fully-fledged quantum simulator, which is not only capable of describing the dynamical evolution of quantum systems, but also gives access to the computation of their static properties. After this, we turn to an investigation of classical non-equilibrium steady states with methods derived from quantum information theory. We construct a special class of matrix product states that exhibit correlations which can best be understood in terms of classical Markov processes. Finally, we investigate the transport properties of non-equilibrium steady states. The dynamical equations are constructed in such a manner that they allow for both stochastic as well as coherent transport in the same formal framework. It is therefore possible to compare different forms of transport within the same model

A theoretical and computational study of the mechanics of biomembranes at multiple scales

Lipid membranes are thin objects that form the main separation structure in cells. They have remarkable mechanical properties; while behaving as a solid shell against bending, they exhibit in-plane fluidity. These two aspects of their mechanics are not only interesting from a physical viewpoint, but also fundamental for their biological function. Indeed, the equilibrium shapes of different organelles in the cell rely on the bending elasticity of lipid membranes. On the other hand, the in-plane fluidity of the membrane is essential in functions such as cell motility, mechano-adaptation, or for the lateral diffusion of proteins and other membrane inclusions. The bending rigidity of membranes can be motivated from microscopic models that account for the stress distribution across the membrane thickness. In particular, the microscopic stress across the membrane is routinely computed from molecular dynamics simulations to investigate how different microscopic features, such as the addition of anesthetics or cholesterol, affect their effective mechanical response. The microscopic stress bridges the gap between the statistical mechanics of a set of point particles, the atoms in a molecular dynamics simulation, and continuum mechanics models. However, we lack an unambiguous definition of the microscopic stress, and different definitions of the microscopic stress suggest different connections between molecular and continuum models. In the first Part of this Thesis, we show that many of the existing definitions of the microscopic stress do not satisfy the most basic balance laws of continuum mechanics, and thus are not physically meaningful. This striking issue has motivated us to propose a new definition of the microscopic stress that complies with these fundamental balance laws. Furthermore, we provide a freely available implementation of our stress definition that can be computed from molecular dynamics simulations (mdstress.org). Our definition of the stress along with our implementation provides a foundation for a meaningful analysis of molecular dynamics simulations from a continuum viewpoint. In addition to lipid membranes, we show the application of our methodology to other important systems, such as defective crystals or fibrous proteins. In the second part of the Thesis, we focus on the continuum modeling of lipid membranes. Because these membranes are continuously brought out-of-equilibrium by biological activity, it is important to go beyond curvature elasticity and describe the internal mechanisms associated with bilayer fluidity. We develop a three-dimensional and non-linear theory and a simulation methodology for the mechanics of lipid membranes, which have been lacking in the field. We base our approach on a general framework for the mechanics of dissipative systems, Onsager's variational principle, and on a careful formulation of the kinematics and balance principles for fluid surfaces. For the simulation of our models, we follow a finite element approach that, however, requires of unconventional dicretization methods due to the non-linear coupling between shape changes and tangent flows on fluid surfaces. Our formulation provides the basis for further investigations of the out-of-equilibrium chemo-mechanics of lipid membranes and other fluid surfaces, such as the cell cortex.Las membranas lipídicas son estructuras delgadas que forman la separación fundamental de las células. Tienen propiedades físicas notables: mientras que se comportan como láminas delgadas sólidas frente a curvatura, presentan fluidez interfacial. Estos dos aspectos de su mecánica son interesantes desde un punto de vista físico e ingenieril, pero además son fundamentales para su función biológica. Las formas de equilibrio de diferentes organelos celulares dependen de la elasticidad frente a curvatura de la membrana lipídica. Por otro lado, la fluidez interfacial es esencial en funciones como la movilidad celular, la adaptación mecánica a deformaciones, o para la difusión lateral de proteínas. La elasticidad frente a curvatura de las membranas lipídicas puede motivarse a través de modelos microscópicos que tienen en cuenta la distribución de esfuerzos a lo largo del espesor de la membrana. En particular, el tensor de esfuerzos microscópico se calcula habitualmente en simulaciones de dinámica molecular a lo largo del espesor de la membrana para investigar cómo diferentes características microscópicas, como la adición de anestésicos o colesterol, afecta la respuesta mecánica efectiva. El tensor de esfuerzos microscópico tiende un puente entre la mecánica estadística de un conjunto de partículas puntuales, los átomos de una simulación de dinámica molecular, y modelos de mecánica de medios continuos. Sin embargo, no disponemos de una definición única del tensor de esfuerzos microscópico, y diferentes definiciones dan lugar a diferentes interpretaciones de la conexión entre modelos moleculares y continuos. En la primera parte de la tesis, mostramos que muchas de las definiciones del tensor de esfuerzos microscópico no satisfacen las leyes más básicas de la mecánica de medios continuos, y por tanto no son físicamente relevantes. Este problema nos ha motivado a proponer una nueva definición del tensor de esfuerzos microscópicos que cumpla las leyes fundamentales de la mecánica de medios continuos por construcción. Además, hemos desarrollado (y puesto a disposición del público libremente) una implementación numérica de nuestra definición del tensor de esfuerzos microscópico que puede calcularse mediante simulaciones de dinámica molecular (mdstress.org). Nuestra definición del tensor de esfuerzos, así como nuestra implementación del mismo, proporcionan una base sólida para el análisis de simulaciones de dinámica molecular desde un punto de vista continuo. Además de membranas lipídicas, mostramos la aplicación de nuestro método en otros sistemas relevantes, como cristales con defectos o proteínas fibrosas. En la segunda parte de esta tesis nos hemos focalizado en el modelado continuo de membranas lipídicas. Ya que estas membranas están constantemente sufriendo actividad biológica que las lleva fuera de equilibrio, es importante tener en cuenta no sólo la elasticidad de curvatura, sino también los grados de libertad internos asociados a la fluidez de la membrana. Para ello, desarrollamos un nuevo marco teórico y computacional general, tridimensional y no-lineal, para la mecánica de membranas lipídicas. Nuestro enfoque se basa en un marco general para la mecánica de sistemas disipativos, el principio variacional de Onsager, y en una formulación cuidadosa de la cinemática y las ecuaciones de balance para superficies fluídas. Para la simulación de nuestros modelos, seguimos una aproximación basada en elementos finitos que, sin embargo, requiere de métodos no convencionales debido al acoplamiento no-lineal entre cambios de forma y los campos de velocidad tangentes en superficies fluídas. Nuestra formulación proporciona la base para futuras investigaciones de la quimiomecánica fuera de equilibrio de membranas lipídicas y otras superficies fluídas, como el cortex celula