Hedonic Quality or Reward? A Study of Basic Pleasure in Homeostasis and Decision Making of a Motivated Autonomous Robot

© The Author (s) 2016. Published by SAGE. This is an Open Access article distributed under the terms of the Creative Commons Attribution 3.0 License (http://www.creativecommons.org/licenses/by/3.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).We present a robot architecture and experiments to investigate some of the roles that pleasure plays in the decision making (action selection) process of an autonomous robot that must survive in its environment. We have conducted three sets of experiments to assess the effect of different types of pleasure---related versus unrelated to the satisfaction of physiological needs---under different environmental circumstances. Our results indicate that pleasure, including pleasure unrelated to need satisfaction, has value for homeostatic management in terms of improved viability and increased flexibility in adaptive behavior.Peer reviewedFinal Published versio

A flexible component-based robot control architecture for hormonal modulation of behaviour and affect

This document is the Accepted Manuscritpt of a paper published in Proceedings of 18th Annual Conference, TAROS 2017, Guildford, UK, July 19–21, 2017. Under embargo. Embargo end date: 20 July 2018. The final publication is available at Springer via https://link.springer.com/chapter/10.1007%2F978-3-319-64107-2_36. © 2017 Springer, Cham.In this paper we present the foundations of an architecture that will support the wider context of our work, which is to explore the link between affect, perception and behaviour from an embodied perspective and assess their relevance to Human Robot Interaction (HRI). Our approach builds upon existing affect-based architectures by combining artificial hormones with discrete abstract components that are designed with the explicit consideration of influencing, and being receptive to, the wider affective state of the robot

Dendritic Morphology Predicts Pattern Recognition Performance in Multi-compartmental Model Neurons with and without Active Conductances

This is an Open Access article published under the Creative Commons Attribution license CC BY 4.0 which allows users to read, copy, distribute and make derivative works, as long as the author of the original work is citedIn this paper we examine how a neuron’s dendritic morphology can affect its pattern recognition performance. We use two different algorithms to systematically explore the space of dendritic morphologies: an algorithm that generates all possible dendritic trees with 22 terminal points, and one that creates representative samples of trees with 128 terminal points. Based on these trees, we construct multi-compartmental models. To assess the performance of the resulting neuronal models, we quantify their ability to discriminate learnt and novel input patterns. We find that the dendritic morphology does have a considerable effect on pattern recognition performance and that the neuronal performance is inversely correlated with the mean depth of the dendritic tree. The results also reveal that the asymmetry index of the dendritic tree does not correlate with the performance for the full range of tree morphologies. The performance of neurons with dendritic tapering is best predicted by the mean and variance of the electrotonic distance of their synapses to the soma. All relationships found for passive neuron models also hold, even in more accentuated form, for neurons with active membranesPeer reviewedFinal Published versio

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications

Using Strategic Movement to Calibrate a Neural Compass: A Spiking Network for Tracking Head Direction in Rats and Robots

The head direction (HD) system in mammals contains neurons that fire to represent the direction the animal is facing in its environment. The ability of these cells to reliably track head direction even after the removal of external sensory cues implies that the HD system is calibrated to function effectively using just internal (proprioceptive and vestibular) inputs. Rat pups and other infant mammals display stereotypical warm-up movements prior to locomotion in novel environments, and similar warm-up movements are seen in adult mammals with certain brain lesion-induced motor impairments. In this study we propose that synaptic learning mechanisms, in conjunction with appropriate movement strategies based on warm-up movements, can calibrate the HD system so that it functions effectively even in darkness. To examine the link between physical embodiment and neural control, and to determine that the system is robust to real-world phenomena, we implemented the synaptic mechanisms in a spiking neural network and tested it on a mobile robot platform. Results show that the combination of the synaptic learning mechanisms and warm-up movements are able to reliably calibrate the HD system so that it accurately tracks real-world head direction, and that calibration breaks down in systematic ways if certain movements are omitted. This work confirms that targeted, embodied behaviour can be used to calibrate neural systems, demonstrates that ‘grounding’ of modelled biological processes in the real world can reveal underlying functional principles (supporting the importance of robotics to biology), and proposes a functional role for stereotypical behaviours seen in infant mammals and those animals with certain motor deficits. We conjecture that these calibration principles may extend to the calibration of other neural systems involved in motion tracking and the representation of space, such as grid cells in entorhinal cortex

Convergence among Non-Sister Dendritic Branches: An Activity-Controlled Mean to Strengthen Network Connectivity

The manner by which axons distribute synaptic connections along dendrites remains a fundamental unresolved issue in neuronal development and physiology. We found in vitro and in vivo indications that dendrites determine the density, location and strength of their synaptic inputs by controlling the distance of their branches from those of their neighbors. Such control occurs through collective branch convergence, a behavior promoted by AMPA and NMDA glutamate receptor activity. At hubs of convergence sites, the incidence of axo-dendritic contacts as well as clustering levels, pre- and post-synaptic protein content and secretion capacity of synaptic connections are higher than found elsewhere. This coupling between synaptic distribution and the pattern of dendritic overlapping results in ‘Economical Small World Network’, a network configuration that enables single axons to innervate multiple and remote dendrites using short wiring lengths. Thus, activity-mediated regulation of the proximity among dendritic branches serves to pattern and strengthen neuronal connectivity

Odor supported place cell model and goal navigation in rodents

Experiments with rodents demonstrate that visual cues play an important role in the control of hippocampal place cells and spatial navigation. Nevertheless, rats may also rely on auditory, olfactory and somatosensory stimuli for orientation. It is also known that rats can track odors or self-generated scent marks to find a food source. Here we model odor supported place cells by using a simple feed-forward network and analyze the impact of olfactory cues on place cell formation and spatial navigation. The obtained place cells are used to solve a goal navigation task by a novel mechanism based on self-marking by odor patches combined with a Q-learning algorithm. We also analyze the impact of place cell remapping on goal directed behavior when switching between two environments. We emphasize the importance of olfactory cues in place cell formation and show that the utility of environmental and self-generated olfactory cues, together with a mixed navigation strategy, improves goal directed navigation

Order in Spontaneous Behavior

Brains are usually described as input/output systems: they transform sensory input into motor output. However, the motor output of brains (behavior) is notoriously variable, even under identical sensory conditions. The question of whether this behavioral variability merely reflects residual deviations due to extrinsic random noise in such otherwise deterministic systems or an intrinsic, adaptive indeterminacy trait is central for the basic understanding of brain function. Instead of random noise, we find a fractal order (resembling Lévy flights) in the temporal structure of spontaneous flight maneuvers in tethered Drosophila fruit flies. Lévy-like probabilistic behavior patterns are evolutionarily conserved, suggesting a general neural mechanism underlying spontaneous behavior. Drosophila can produce these patterns endogenously, without any external cues. The fly's behavior is controlled by brain circuits which operate as a nonlinear system with unstable dynamics far from equilibrium. These findings suggest that both general models of brain function and autonomous agents ought to include biologically relevant nonlinear, endogenous behavior-initiating mechanisms if they strive to realistically simulate biological brains or out-compete other agents

Impact of Dendritic Size and Dendritic Topology on Burst Firing in Pyramidal Cells

Neurons display a wide range of intrinsic firing patterns. A particularly relevant pattern for neuronal signaling and synaptic plasticity is burst firing, the generation of clusters of action potentials with short interspike intervals. Besides ion-channel composition, dendritic morphology appears to be an important factor modulating firing pattern. However, the underlying mechanisms are poorly understood, and the impact of morphology on burst firing remains insufficiently known. Dendritic morphology is not fixed but can undergo significant changes in many pathological conditions. Using computational models of neocortical pyramidal cells, we here show that not only the total length of the apical dendrite but also the topological structure of its branching pattern markedly influences inter- and intraburst spike intervals and even determines whether or not a cell exhibits burst firing. We found that there is only a range of dendritic sizes that supports burst firing, and that this range is modulated by dendritic topology. Either reducing or enlarging the dendritic tree, or merely modifying its topological structure without changing total dendritic length, can transform a cell's firing pattern from bursting to tonic firing. Interestingly, the results are largely independent of whether the cells are stimulated by current injection at the soma or by synapses distributed over the dendritic tree. By means of a novel measure called mean electrotonic path length, we show that the influence of dendritic morphology on burst firing is attributable to the effect both dendritic size and dendritic topology have, not on somatic input conductance, but on the average spatial extent of the dendritic tree and the spatiotemporal dynamics of the dendritic membrane potential. Our results suggest that alterations in size or topology of pyramidal cell morphology, such as observed in Alzheimer's disease, mental retardation, epilepsy, and chronic stress, could change neuronal burst firing and thus ultimately affect information processing and cognition