Learning with Surprise:Theory and Applications

Everybody knows what it feels to be surprised. Surprise raises our attention and is crucial for learning. It is a ubiquitous concept whose traces have been found in both neuroscience and machine learning. However, a comprehensive theory has not yet been developed that addresses fundamental problems about surprise: (1) surprise is difficult to quantify. How should we measure the level of surprise when we encounter an unexpected event? What is the link between surprise and startle responses in behavioral biology? (2) the key role of surprise in learning is somewhat unclear. We believe that surprise drives attention and modifies learning; but, how should surprise be incorporated, in general paradigms of learning? and (3) can we develop a biologically plausible theory that explains how surprise can be neurally calculated and implemented in the brain? I propose a theoretical framework to address the above issues about surprise. There are three components to this framework: (1) a subjective confidence-adjusted measure of surprise, that can be used for quantification purposes, (2) a surprise-minimization learning rule that models the role of surprise in learning by balancing the relative contribution of new and old data for inference about the world, and (3) a surprise-modulated Hebbian plasticity rule that can be implemented in both artificial and spiking neural networks. The proposed online rule links surprise to the activity of the neuromodulatory system in the brain, and belongs to the class of neo-Hebbian plasticity rules. My work on the foundations of surprise provides a suitable framework for future studies on learning with surprise. Reinforcement learning methods can be enhanced by incorporating the proposed theory of surprise. The theory could ultimately become interesting for the analysis of fMRI and EEG data. It may also inspire new synaptic plasticity rules that are under the simultaneous control of reward and surprise. Moreover, the proposed theory can be used to make testable predictions about the time course of the neural substrate of surprise (e.g., noradrenaline), and suggests behavioral experiments that can be performed on real animals for studying surprise-related neural activity

Balancing New Against Old Information: The Role of Surprise

Surprise is a widely used concept describing a range of phenomena from unexpected events to behavioral responses. We propose a measure of surprise, to arrive at a new framework for surprise-driven learning. There are two components to this framework: (i) a confidence-adjusted surprise measure to capture environmental statistics as well as subjective beliefs, (ii) a surprise-minimization learning rule, or SMiLe-rule, which dynamically adjusts the balance between new and old information without making prior assumptions about the temporal statistics of the environment. We apply our framework to a dynamic decision-making task and a maze exploration task to demonstrate that it is suitable for learning in complex environments, even if the environment undergoes gradual or sudden changes. Our proposed surprise-modulated belief update algorithm provides a framework to study the behavior of humans and animals encountering surprising events

Surprise minimization as a learning strategy in neural networks

Visual tracking concerns the problem of following an arbitrary object in a video sequence. In this thesis, we examine how to use stereo images to extend existing visual tracking algorithms, which methods exists to obtain information from stereo images, and how the results change as the parameters to each tracker vary. For this purpose, four abstract approaches are identified, with five distinct implementations. Each tracker implementation is an extension of a baseline algorithm, MOSSE. The free parameters of each model are optimized with respect to two different evaluation strategies called nor- and wir-tests, and four different objective functions, which are then fixed when comparing the models against each other. The results are created on single target tracks extracted from the KITTI tracking dataset, and the optimization results show that none of the objective functions are sensitive to the exposed parameters under the joint selection of model and dataset. The evaluation results also shows that none of the extensions improve the results of the baseline tracker