Scale-free memory model for multiagent reinforcement learning. Mean field approximation and rock-paper-scissors dynamics

A continuous time model for multiagent systems governed by reinforcement learning with scale-free memory is developed. The agents are assumed to act independently of one another in optimizing their choice of possible actions via trial-and-error search. To gain awareness about the action value the agents accumulate in their memory the rewards obtained from taking a specific action at each moment of time. The contribution of the rewards in the past to the agent current perception of action value is described by an integral operator with a power-law kernel. Finally a fractional differential equation governing the system dynamics is obtained. The agents are considered to interact with one another implicitly via the reward of one agent depending on the choice of the other agents. The pairwise interaction model is adopted to describe this effect. As a specific example of systems with non-transitive interactions, a two agent and three agent systems of the rock-paper-scissors type are analyzed in detail, including the stability analysis and numerical simulation. Scale-free memory is demonstrated to cause complex dynamics of the systems at hand. In particular, it is shown that there can be simultaneously two modes of the system instability undergoing subcritical and supercritical bifurcation, with the latter one exhibiting anomalous oscillations with the amplitude and period growing with time. Besides, the instability onset via this supercritical mode may be regarded as "altruism self-organization". For the three agent system the instability dynamics is found to be rather irregular and can be composed of alternate fragments of oscillations different in their properties.Comment: 17 pages, 7 figur

An evolutionary perspective on caching by corvids

A principal finding in the food-caching literature is that species differences in hoarding propensity are positively correlated with species differences in degree of adaptations to caching behaviour, such as performance on spatial memory tasks and hippocampal volume. However, there are examples that do not fit this pattern. We argue that these examples can be better understood by considering the phylogenetic relatedness between species. We reconstruct the ancestral state for caching behaviour in corvids and assess when transitions in caching behaviour occurred within the corvid phylogeny. Our analysis shows that the common ancestor of all corvids was a moderate cacher. This result suggests that corvids followed a bi-directional evolutionary trajectory in which caching was secondarily lost twice and there were at least two independent transitions from moderate to specialized caching. The independent evolution of specialized cachers in the two groups must, therefore, be a case of convergent evolution. This is exemplified by the fact that specialized cachers show structurally different adaptations serving the same function to intense caching, such as different pouches to transport food. Finally, we argue that convergent evolution may have led to adaptations in memory and hippocampus that serve the same function but differ in design, and that these different adaptations may explain the examples that do not fit the pattern predicted by the adaptive specialization hypothesis