Scale-free memory model for multiagent reinforcement learning. Mean field approximation and rock-paper-scissors dynamics

A continuous time model for multiagent systems governed by reinforcement learning with scale-free memory is developed. The agents are assumed to act independently of one another in optimizing their choice of possible actions via trial-and-error search. To gain awareness about the action value the agents accumulate in their memory the rewards obtained from taking a specific action at each moment of time. The contribution of the rewards in the past to the agent current perception of action value is described by an integral operator with a power-law kernel. Finally a fractional differential equation governing the system dynamics is obtained. The agents are considered to interact with one another implicitly via the reward of one agent depending on the choice of the other agents. The pairwise interaction model is adopted to describe this effect. As a specific example of systems with non-transitive interactions, a two agent and three agent systems of the rock-paper-scissors type are analyzed in detail, including the stability analysis and numerical simulation. Scale-free memory is demonstrated to cause complex dynamics of the systems at hand. In particular, it is shown that there can be simultaneously two modes of the system instability undergoing subcritical and supercritical bifurcation, with the latter one exhibiting anomalous oscillations with the amplitude and period growing with time. Besides, the instability onset via this supercritical mode may be regarded as "altruism self-organization". For the three agent system the instability dynamics is found to be rather irregular and can be composed of alternate fragments of oscillations different in their properties.Comment: 17 pages, 7 figur

Genes, Neurons, and Circuits Underlying the Reproductive Behavior in Drosophila (Advanced Studies on Sustainable Animal Production: Interrelationships among Human, Animal and Environment, 8th International Symposium of Integrated Field Science)

Oral Sessio

Silica-based monolithic sensing plates for waveguide-mode sensors

We developed a monolithic sensing plate for a waveguide-mode sensor. The plate consists of a SiO2 glass substrate and a thin silicon layer the surface of which is thermally oxidized to form a SiO2 glass waveguide. We confirmed that the sensing plate is suitable for high-sensitivity detection of molecular adsorption at the waveguide surface. In addition, a significant enhancement of the sensitivity of the sensor was achieved by perforating the waveguide with holes with diameters of a few tens of nanometers by selective etching of latent tracks created by swift heavy-ion irradiation. Possible strategies for optimizing the plate are discussed

Nedd4-2-dependent regulation of astrocytic Kir4.1 and Connexin43 controls neuronal network activity

Nedd4-2 is an E3 ubiquitin ligase in which missense mutation is related to familial epilepsy, indicating its critical role in regulating neuronal network activity. However, Nedd4-2 substrates involved in neuronal network function have yet to be identified. Using mouse lines lacking Nedd4-1 and Nedd4-2, we identified astrocytic channel proteins inwardly rectifying K+ channel 4.1 (Kir4.1) and Connexin43 as Nedd4-2 substrates. We found that the expression of Kir4.1 and Connexin43 is increased upon conditional deletion of Nedd4-2 in astrocytes, leading to an elevation of astrocytic membrane ion permeability and gap junction activity, with a consequent reduction of γ-oscillatory neuronal network activity. Interestingly, our biochemical data demonstrate that missense mutations found in familial epileptic patients produce gain-of-function of the Nedd4-2 gene product. Our data reveal a process of coordinated astrocytic ion channel proteostasis that controls astrocyte function and astrocyte-dependent neuronal network activity and elucidate a potential mechanism by which aberrant Nedd4-2 function leads to epilepsy

Functional Evolution of Duplicated Odorant-Binding Protein Genes, Obp57d and Obp57e, in Drosophila

Odorant-binding proteins (OBPs) are extracellular proteins found in insect chemosensilla, where they participate in the sensing of odors, tastes, and pheromones. Although a large number of OBP genes have been identified in insect genomes, their molecular functions and biological roles have been clarified in limited cases. Two OBP genes, Obp57d and Obp57e, were involved in the evolution of host-plant preference in Drosophila sechellia. Comparative analyses of the Obp57d/e genomic sequences from 27 closely related species suggested that the two genes arose by tandem gene duplication and functionally diverged from each other. In this study, the functional evolution of Obp57d and Obp57e was examined by in vitro binding assays using recombinant proteins synthesized in a bacterial system. Compared to the ancestral Dpse\OBP57de, Dmel\OBP57d was more specialized to tridecanoic acid while Dmel\OBP57e was generalized regarding their binding affinity, suggesting that the two OBP genes underwent subfunctionalization and neofunctionalization. A behavioral analysis using knockout flies supported that the biological role is different between OBP57d and OBP57e in vivo. Site-directed mutagenesis of the evolutionarily conserved amino acids revealed that these residues play an important role in protein folding. These findings provide a clue to understanding how the repertoire of OBP genes is maintained in a genome under natural selection

Peripheral, Central and Behavioral Responses to the Cuticular Pheromone Bouquet in Drosophila melanogaster Males

Pheromonal communication is crucial with regard to mate choice in many animals including insects. Drosophila melanogaster flies produce a pheromonal bouquet with many cuticular hydrocarbons some of which diverge between the sexes and differently affect male courtship behavior. Cuticular pheromones have a relatively high weight and are thought to be — mostly but not only — detected by gustatory contact. However, the response of the peripheral and central gustatory systems to these substances remains poorly explored. We measured the effect induced by pheromonal cuticular mixtures on (i) the electrophysiological response of peripheral gustatory receptor neurons, (ii) the calcium variation in brain centers receiving these gustatory inputs and (iii) the behavioral reaction induced in control males and in mutant desat1 males, which show abnormal pheromone production and perception. While male and female pheromones induced inhibitory-like effects on taste receptor neurons, the contact of male pheromones on male fore-tarsi elicits a long-lasting response of higher intensity in the dedicated gustatory brain center. We found that the behavior of control males was more strongly inhibited by male pheromones than by female pheromones, but this difference disappeared in anosmic males. Mutant desat1 males showed an increased sensitivity of their peripheral gustatory neurons to contact pheromones and a behavioral incapacity to discriminate sex pheromones. Together our data indicate that cuticular hydrocarbons induce long-lasting inhibitory effects on the relevant taste pathway which may interact with the olfactory pathway to modulate pheromonal perception

Hedonic Taste in Drosophila Revealed by Olfactory Receptors Expressed in Taste Neurons

Taste and olfaction are each tuned to a unique set of chemicals in the outside world, and their corresponding sensory spaces are mapped in different areas in the brain. This dichotomy matches categories of receptors detecting molecules either in the gaseous or in the liquid phase in terrestrial animals. However, in Drosophila olfactory and gustatory neurons express receptors which belong to the same family of 7-transmembrane domain proteins. Striking overlaps exist in their sequence structure and in their expression pattern, suggesting that there might be some functional commonalities between them. In this work, we tested the assumption that Drosophila olfactory receptor proteins are compatible with taste neurons by ectopically expressing an olfactory receptor (OR22a and OR83b) for which ligands are known. Using electrophysiological recordings, we show that the transformed taste neurons are excited by odor ligands as by their cognate tastants. The wiring of these neurons to the brain seems unchanged and no additional connections to the antennal lobe were detected. The odor ligands detected by the olfactory receptor acquire a new hedonic value, inducing appetitive or aversive behaviors depending on the categories of taste neurons in which they are expressed i.e. sugar- or bitter-sensing cells expressing either Gr5a or Gr66a receptors. Taste neurons expressing ectopic olfactory receptors can sense odors at close range either in the aerial phase or by contact, in a lipophilic phase. The responses of the transformed taste neurons to the odorant are similar to those obtained with tastants. The hedonic value attributed to tastants is directly linked to the taste neurons in which their receptors are expressed