Formal verification of deep reinforcement learning agents

Deep reinforcement learning has been successfully applied to many control tasks, but the application of such controllers in safety-critical scenarios has been limited due to safety concerns. Rigorous testing of these controllers is challenging, particularly when they operate in uncertain environments. In this thesis we develop novel verification techniques to give the user stronger guarantees over the performance of the trained agents that they would be able to obtain by testing, under different degrees and sources of uncertainty. In particular, we tackle three different sources of uncertainty to the agent and offer different algorithms to provide strong guarantees to the user. The first one is input noise: sensors in the real world always provide imperfect data. The second source of uncertainty comes from the actuators: once an agent decides to take a specific action, faulty actuators and or hardware problems could still prevent the agent from acting upon the decisions given by the controller. The last source of uncertainty is the policy: the set of decisions the controller takes when operating in the environment. Agents may act probabilistically for a number of reasons, such as dealing with adversaries in a competitive environment or addressing partial observability of the environment. In this thesis, we develop formal models of controllers executing under uncertainty, and propose new verification techniques based on abstract interpretation for their analysis. We cover different horizon lengths, i.e., the number of steps into the future that we analyse, and present methods for both finite-horizon and infinite-horizon verification. We perform both probabilistic and non-probabilistic analysis of the models constructed, depending on the methodology adopted. We implement and evaluate our methods on controllers trained for several benchmark control problems

Identifying Network Correlates of Memory Consolidation

Neuronal spiking activity carries information about our experiences in the waking world but exactly how the brain can quickly and efficiently encode sensory information into a useful neural code and then subsequently consolidate that information into memory remains a mystery. While neuronal networks are known to play a vital role in these processes, detangling the properties of network activity from the complex spiking dynamics observed is a formidable challenge, requiring collaborations across scientific disciplines. In this work, I outline my contributions in computational modeling and data analysis toward understanding how network dynamics facilitate memory consolidation. For experimental perspective, I investigate hippocampal recordings of mice that are subjected to contextual fear conditioning and subsequently undergo sleep-dependent fear memory consolidation. First, I outline the development of a functional connectivity algorithm which rapidly and robustly assesses network structure based on neuronal spike timing. I show that the relative stability of these functional networks can be used to identify global network dynamics, revealing that an increase in functional network stability correlates with successful fear memory consolidation in vivo. Using an attractor-based model to simulate memory encoding and consolidation, I go on to show that dynamics associated with a second-order phase transition, at a critical point in phase-space, are necessary for recruiting additional neurons into network dynamics associated with memory consolidation. I show that successful consolidation subsequently shifts dynamics away from a critical point and towards sub-critical dynamics. Investigations of in vivo spiking dynamics likewise revealed that hippocampal dynamics during non-rapid-eye-movement (NREM) sleep show features of being near a critical point and that fear memory consolidation leads to a shift in dynamics. Finally, I investigate the role of NREM sleep in facilitating memory consolidation using a conductance-based model of neuronal activity that can easily switch between modes of activity loosely representing waking and NREM sleep. Analysis of model simulations revealed that oscillations associated with NREM sleep promote a phase-based coding of information; neurons with high firing rates during periods of wake lead spiking activity during NREM oscillations. I show that when phase-coding is active in both simulations and in vivo, synaptic plasticity selectively strengthens the input to neurons firing late in the oscillation while simultaneously reducing input to neurons firing early in the oscillation. The effect is a net homogenization of firing rates observed in multiple other studies, and subsequently leads to recruitment of new neurons into a memory engram and information transfer from fast firing neurons to slow firing neurons. Taken together, my work outlines important, newly-discovered features of neuronal network dynamics related to memory encoding and consolidation: networks near criticality promote recruitment of additional neurons into stable firing patterns through NREM-associated oscillations and subsequently consolidates information into memories through phase-based coding.PHDBiophysicsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/162991/1/qmskill_1.pd

Learning from Integral Losses in Physics Informed Neural Networks

This work proposes a solution for the problem of training physics informed networks under partial integro-differential equations. These equations require infinite or a large number of neural evaluations to construct a single residual for training. As a result, accurate evaluation may be impractical, and we show that naive approximations at replacing these integrals with unbiased estimates lead to biased loss functions and solutions. To overcome this bias, we investigate three types of solutions: the deterministic sampling approach, the double-sampling trick, and the delayed target method. We consider three classes of PDEs for benchmarking; one defining a Poisson problem with singular charges and weak solutions, another involving weak solutions on electro-magnetic fields and a Maxwell equation, and a third one defining a Smoluchowski coagulation problem. Our numerical results confirm the existence of the aforementioned bias in practice, and also show that our proposed delayed target approach can lead to accurate solutions with comparable quality to ones estimated with a large number of samples. Our implementation is open-source and available at https://github.com/ehsansaleh/btspinn

Reservoir Computing: computation with dynamical systems

In het onderzoeksgebied Machine Learning worden systemen onderzocht die kunnen leren op basis van voorbeelden. Binnen dit onderzoeksgebied zijn de recurrente neurale netwerken een belangrijke deelgroep. Deze netwerken zijn abstracte modellen van de werking van delen van de hersenen. Zij zijn in staat om zeer complexe temporele problemen op te lossen maar zijn over het algemeen zeer moeilijk om te trainen. Recentelijk zijn een aantal gelijkaardige methodes voorgesteld die dit trainingsprobleem elimineren. Deze methodes worden aangeduid met de naam Reservoir Computing. Reservoir Computing combineert de indrukwekkende rekenkracht van recurrente neurale netwerken met een eenvoudige trainingsmethode. Bovendien blijkt dat deze trainingsmethoden niet beperkt zijn tot neurale netwerken, maar kunnen toegepast worden op generieke dynamische systemen. Waarom deze systemen goed werken en welke eigenschappen bepalend zijn voor de prestatie is evenwel nog niet duidelijk. Voor dit proefschrift is onderzoek gedaan naar de dynamische eigenschappen van generieke Reservoir Computing systemen. Zo is experimenteel aangetoond dat de idee van Reservoir Computing ook toepasbaar is op niet-neurale netwerken van dynamische knopen. Verder is een maat voorgesteld die gebruikt kan worden om het dynamisch regime van een reservoir te meten. Tenslotte is een adaptatieregel geïntroduceerd die voor een breed scala reservoirtypes de dynamica van het reservoir kan afregelen tot het gewenste dynamisch regime. De technieken beschreven in dit proefschrift zijn gedemonstreerd op verschillende academische en ingenieurstoepassingen

Neuronal Models of Motor Sequence Learning in the Songbird

Communication of complex content is an important ability in our everyday life. For communication to be possible, several requirements need to be met: The individual communicated to has to learn to associate a certain meaning with a given sound. In the brain, this sound is represented as a spatio-temporal pattern of spikes, which will thus have to be associated with a different spike pattern representing its meaning. In this thesis, models for associative learning in spiking neurons are introduced in chapters 6 and 7. There, a new biologically plausible learning mechanism is proposed, where a property of the neuronal dynamics - the hyperpolarization of a neuron after each spike it produces - is coupled with a homeostatic plasticity mechanism, which acts to balance inputs into the neuron. In chapter 6, the mechanism used is a version of spike timing dependent plasticity (STDP), a property that was experimentally observed: The direction and amplitude of synaptic change depends on the precise timing of pre- and postsynaptic spiking activity. This mechanism is applied to associative learning of output spikes in response to purely spatial spiking patterns. In chapter 7, a new learning rule is introduced, which is derived from the objective of a balanced membrane potential. This learning rule is shown to be equivalent to a version of STDP and applied to associative learning of precisely timed output spikes in response to spatio-temporal input patterns. The individual communicating has to learn to reproduce certain sounds (which can be associated with a given meaning). To that end, a memory of the sound sequence has to be formed. Since sound sequences are represented as sequences of activation patterns in the brain, learning of a given sequence of spike patterns is an interesting problem for theoretical considerations Here, it is shown that the biologically plausible learning mechanism introduced for associative learning enables recurrently coupled networks of spiking neurons to learn to reproduce given sequences of spikes. These results are presented in chapter 9. Finally, the communicator has to translate the sensory memory into motor actions that serve to reproduce the target sound. This process is investigated in the framework of inverse model learning, where the learner learns to invert the action-perception cycle by mapping perceptions back onto the actions that caused them. Two different setups for inverse model learning are investigated: In chapter 5, a simple setup for inverse model learning is coupled with the learning algorithm used for Perceptron learning in chapter 6 and it is shown that models of the sound generation and perception process, which are non-linear and non-local in time, can be inverted, if the width of the distribution of time delays of self-generated inputs caused by an individual motor spike is not too large. This limitation is mitigated by the model introduced in chapter 8. Both these models have experimentally testable consequences, namely a dip in the autocorrelation function of the spike times in the motor population of the duration of the loop delay, i.e. the time it takes for a motor activation to cause a sound and thus a sensory activation and the time that this sensory activation takes to be looped back to the motor population. Furthermore, both models predict neurons, which are active during the sound generation and during the passive playback of the sound with a time delay equivalent to the loop delay. Finally, the inverse model presented in chapter 8 additionally predicts mirror neurons without a time delay. Both types of mirror neurons have been observed in the songbird [GKGH14, PPNM08], a popular animal model for vocal imitation learning

Системи диференциални уравнения и невронни мрежи със закъснения и импулси

Department of Mathematics & Statistics, College of Science, Sultan Qaboos University, Muscat, Sultanate of Oman и ИМИ-БАН, 16.06.2014 г., присъждане на научна степен "доктор на науките" на Валерий Ковачев по научна специалност 01.01.13. математическо моделиране и приложение на математиката. [Covachev Valery Hristov; Ковачев Валерий Христов

Qualitative Studies of Nonlinear Hybrid Systems

A hybrid system is a dynamical system that exhibits both continuous and discrete dynamic behavior. Hybrid systems arise in a wide variety of important applications in diverse areas, ranging from biology to computer science to air traffic dynamics. The interaction of continuous- and discrete-time dynamics in a hybrid system often leads to very rich dynamical behavior and phenomena that are not encountered in purely continuous- or discrete-time systems. Investigating the dynamical behavior of hybrid systems is of great theoretical and practical importance. The objectives of this thesis are to develop the qualitative theory of nonlinear hybrid systems with impulses, time-delay, switching modes, and stochastic disturbances, to develop algorithms and perform analysis for hybrid systems with an emphasis on stability and control, and to apply the theory and methods to real-world application problems. Switched nonlinear systems are formulated as a family of nonlinear differential equations, called subsystems, together with a switching signal that selects the continuous dynamics among the subsystems. Uniform stability is studied emphasizing the situation where both stable and unstable subsystems are present. Uniformity of stability refers to both the initial time and a family of switching signals. Stabilization of nonlinear systems via state-dependent switching signal is investigated. Based on assumptions on a convex linear combination of the nonlinear vector fields, a generalized minimal rule is proposed to generate stabilizing switching signals that are well-defined and do not exhibit chattering or Zeno behavior. Impulsive switched systems are hybrid systems exhibiting both impulse and switching effects, and are mathematically formulated as a switched nonlinear system coupled with a sequence of nonlinear difference equations that act on the switched system at discrete times. Impulsive switching signals integrate both impulsive and switching laws that specify when and how impulses and switching occur. Invariance principles can be used to investigate asymptotic stability in the absence of a strict Lyapunov function. An invariance principle is established for impulsive switched systems under weak dwell-time signals. Applications of this invariance principle provide several asymptotic stability criteria. Input-to-state stability notions are formulated in terms of two different measures, which not only unify various stability notions under the stability theory in two measures, but also bridge this theory with the existent input/output theories for nonlinear systems. Input-to-state stability results are obtained for impulsive switched systems under generalized dwell-time signals. Hybrid time-delay systems are hybrid systems with dependence on the past states of the systems. Switched delay systems and impulsive switched systems are special classes of hybrid time-delay systems. Both invariance property and input-to-state stability are extended to cover hybrid time-delay systems. Stochastic hybrid systems are hybrid systems subject to random disturbances, and are formulated using stochastic differential equations. Focused on stochastic hybrid systems with time-delay, a fundamental theory regarding existence and uniqueness of solutions is established. Stabilization schemes for stochastic delay systems using state-dependent switching and stabilizing impulses are proposed, both emphasizing the situation where all the subsystems are unstable. Concerning general stochastic hybrid systems with time-delay, the Razumikhin technique and multiple Lyapunov functions are combined to obtain several Razumikhin-type theorems on both moment and almost sure stability of stochastic hybrid systems with time-delay. Consensus problems in networked multi-agent systems and global convergence of artificial neural networks are related to qualitative studies of hybrid systems in the sense that dynamic switching, impulsive effects, communication time-delays, and random disturbances are ubiquitous in networked systems. Consensus protocols are proposed for reaching consensus among networked agents despite switching network topologies, communication time-delays, and measurement noises. Focused on neural networks with discontinuous neuron activation functions and mixed time-delays, sufficient conditions for existence and uniqueness of equilibrium and global convergence and stability are derived using both linear matrix inequalities and M-matrix type conditions. Numerical examples and simulations are presented throughout this thesis to illustrate the theoretical results

Mean field modelling of human EEG: application to epilepsy

Aggregated electrical activity from brain regions recorded via an electroencephalogram (EEG), reveal that the brain is never at rest, producing a spectrum of ongoing oscillations that change as a result of different behavioural states and neurological conditions. In particular, this thesis focusses on pathological oscillations associated with absence seizures that typically affect 2–16 year old children. Investigation of the cellular and network mechanisms for absence seizures studies have implicated an abnormality in the cortical and thalamic activity in the generation of absence seizures, which have provided much insight to the potential cause of this disease. A number of competing hypotheses have been suggested, however the precise cause has yet to be determined. This work attempts to provide an explanation of these abnormal rhythms by considering a physiologically based, macroscopic continuum mean-field model of the brain's electrical activity. The methodology taken in this thesis is to assume that many of the physiological details of the involved brain structures can be aggregated into continuum state variables and parameters. The methodology has the advantage to indirectly encapsulate into state variables and parameters, many known physiological mechanisms underlying the genesis of epilepsy, which permits a reduction of the complexity of the problem. That is, a macroscopic description of the involved brain structures involved in epilepsy is taken and then by scanning the parameters of the model, identification of state changes in the system are made possible. Thus, this work demonstrates how changes in brain state as determined in EEG can be understood via dynamical state changes in the model providing an explanation of absence seizures. Furthermore, key observations from both the model and EEG data motivates a number of model reductions. These reductions provide approximate solutions of seizure oscillations and a better understanding of periodic oscillations arising from the involved brain regions. Local analysis of oscillations are performed by employing dynamical systems theory which provide necessary and sufficient conditions for their appearance. Finally local and global stability is then proved for the reduced model, for a reduced region in the parameter space. The results obtained in this thesis can be extended and suggestions are provided for future progress in this area

Computer Aided Verification

This open access two-volume set LNCS 11561 and 11562 constitutes the refereed proceedings of the 31st International Conference on Computer Aided Verification, CAV 2019, held in New York City, USA, in July 2019. The 52 full papers presented together with 13 tool papers and 2 case studies, were carefully reviewed and selected from 258 submissions. The papers were organized in the following topical sections: Part I: automata and timed systems; security and hyperproperties; synthesis; model checking; cyber-physical systems and machine learning; probabilistic systems, runtime techniques; dynamical, hybrid, and reactive systems; Part II: logics, decision procedures; and solvers; numerical programs; verification; distributed systems and networks; verification and invariants; and concurrency