Reinforcement Learning Dynamics in Social Dilemmas

In this paper we replicate and advance Macy and Flache\'s (2002; Proc. Natl. Acad. Sci. USA, 99, 7229–7236) work on the dynamics of reinforcement learning in 2�2 (2-player 2-strategy) social dilemmas. In particular, we provide further insight into the solution concepts that they describe, illustrate some recent analytical results on the dynamics of their model, and discuss the robustness of such results to occasional mistakes made by players in choosing their actions (i.e. trembling hands). It is shown here that the dynamics of their model are strongly dependent on the speed at which players learn. With high learning rates the system quickly reaches its asymptotic behaviour; on the other hand, when learning rates are low, two distinctively different transient regimes can be clearly observed. It is shown that the inclusion of small quantities of randomness in players\' decisions can change the dynamics of the model dramatically.Reinforcement Learning; Replication; Game Theory; Social Dilemmas; Agent-Based; Slow Learning

Techniques to Understand Computer Simulations: Markov Chain Analysis

The aim of this paper is to assist researchers in understanding the dynamics of simulation models that have been implemented and can be run in a computer, i.e. computer models. To do that, we start by explaining (a) that computer models are just input-output functions, (b) that every computer model can be re-implemented in many different formalisms (in particular in most programming languages), leading to alternative representations of the same input-output relation, and (c) that many computer models in the social simulation literature can be usefully represented as time-homogeneous Markov chains. Then we argue that analysing a computer model as a Markov chain can make apparent many features of the model that were not so evident before conducting such analysis. To prove this point, we present the main concepts needed to conduct a formal analysis of any time-homogeneous Markov chain, and we illustrate the usefulness of these concepts by analysing 10 well-known models in the social simulation literature as Markov chains. These models are: • Schelling\'s (1971) model of spatial segregation • Epstein and Axtell\'s (1996) Sugarscape • Miller and Page\'s (2004) standing ovation model • Arthur\'s (1989) model of competing technologies • Axelrod\'s (1986) metanorms models • Takahashi\'s (2000) model of generalized exchange • Axelrod\'s (1997) model of dissemination of culture • Kinnaird\'s (1946) truels • Axelrod and Bennett\'s (1993) model of competing bimodal coalitions • Joyce et al.\'s (2006) model of conditional association In particular, we explain how to characterise the transient and the asymptotic dynamics of these computer models and, where appropriate, how to assess the stochastic stability of their absorbing states. In all cases, the analysis conducted using the theory of Markov chains has yielded useful insights about the dynamics of the computer model under study.Computer Modelling, Simulation, Markov, Stochastic Processes, Analysis, Re-Implementation

Forecasting VARMA processes using VAR models and subspace-based state space models

VAR modelling is a frequent technique in econometrics for linear processes. VAR modelling offers some desirable features such as relatively simple procedures for model specification (order selection) and the possibility of obtaining quick non-iterative maximum likelihood estimates of the system parameters. However, if the process under study follows a finite-order VARMA structure, it cannot be equivalently represented by any finite-order VAR model. On the other hand, a finite-order state space model can represent a finite-order VARMA process exactly, and, for state-space modelling, subspace algorithms allow for quick and non-iterative estimates of the system parameters, as well as for simple specification procedures. Given the previous facts, we check in this paper whether subspace-based state space models provide better forecasts than VAR models when working with VARMA data generating processes. In a simulation study we generate samples from different VARMA data generating processes, obtain VAR-based and state-space-based models for each generating process and compare the predictive power of the obtained models. Different specification and estimation algorithms are considered; in particular, within the subspace family, the CCA (Canonical Correlation Analysis) algorithm is the selected option to obtain state-space models. Our results indicate that when the MA parameter of an ARMA process is close to 1, the CCA state space models are likely to provide better forecasts than the AR models. We also conduct a practical comparison (for two cointegrated economic time series) of the predictive power of Johansen restricted-VAR (VEC) models with the predictive power of state space models obtained by the CCA subspace algorithm, including a density forecasting analysis.subspace algorithms; VAR; forecasting; cointegration; Johansen; CCA

"Test two, choose the better" leads to high cooperation in the Centipede game

Explaining cooperative experimental evidence in the Centipede game constitutes a challenge for rational game theory. Traditional analyses of Centipede based on backward induction predict uncooperative behavior. Furthermore, analyses based on learning or adaptation under the assumption that those strategies that are more successful in a population tend to spread at a higher rate usually make the same prediction. In this paper we consider an adaptation model in which agents in a finite population do adopt those strategies that turn out to be most successful, according to their own experience. However, this behavior leads to an equilibrium with high levels of cooperation and whose qualitative features are consistent with experimental evidence.Financial support from the Spanish State Research Agency (PID2020-118906GB-I00 / AEI / 10.13039/501100011033), from “Junta de Castilla y León - Consejería de Educación” through BDNS 425389, from the Spanish Ministry of Science, Innovation and Universities (PRX18-00182, PRX19/00113), and from the Fulbright Program (PRX19/00113), is gratefully acknowledged

Two-twistor particle models and free massive higher spin fields

We present D=3 and D=4 models for massive particles moving in a new type of enlarged spacetime, with D-1 additional vector coordinates, which after quantization lead to the towers of massive higher spin (HS) free fields. Two classically equivalent formulations are presented: one with a hybrid spacetime/bispinor geometry and a second described by a free two-twistor dynamics with constraints. After quantization in the D=3 and D=4 cases, the wave functions are given as functions on the SL(2,R) and SL(2,C) group manifolds respectively, and describe arbitrary on-shell momenta and spin degrees of freedom. Finally, the D=6 case and possible supersymmetric extensions are mentioned.Comment: 37 pages, plain latex, v2. Text in Secs. 1 nd 4 enlarged, references added. Version to appear in JHE

Characterization of digital dispersive spectrometers by low coherence interferometry

We propose a procedure to determine the spectral response of digital dispersive spectrometers without previous knowledge of any parameter of the system. The method consists of applying the Fourier transform spectroscopy technique to each pixel of the detection plane, a CCD camera, to obtain its individual spectral response. From this simple procedure, the system-point spread function and the effect of the finite pixel width are taken into account giving rise to a response matrix that fully characterizes the spectrometer. Using the response matrix information we find the resolving power of a given spectrometer, predict in advance its response to any virtual input spectrum and improve numerically the spectrometer's resolution. We consider that the presented approach could be useful in most spectroscopic branches such as in computational spectroscopy, optical coherence tomography, hyperspectral imaging, spectral interferometry and analytical chemistry, among others.Fil: Martínez Matos, Ó.. Universidad Complutense de Madrid; EspañaFil: Rickenstorff, C.. Universidad Complutense de Madrid; EspañaFil: Zamora, S.. Universidad Complutense de Madrid; EspañaFil: Izquierdo, J. G.. Universidad Complutense de Madrid; EspañaFil: Vaveliuk, Pablo. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - La Plata. Centro de Investigaciones Ópticas. Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones Ópticas. Universidad Nacional de La Plata. Centro de Investigaciones Ópticas; Argentin

Best experienced payoff dynamics and cooperation in the centipede game

We study population game dynamics under which each revising agent tests each of his strategies a fixed number of times, with each play of each strategy being against a newly drawn opponent, and chooses the strategy whose total payoff was highest. In the centipede game, these best experienced payoff dynamics lead to cooperative play. When strategies are tested once, play at the almost globally stable state is concentrated on the last few nodes of the game, with the proportions of agents playing each strategy being largely independent of the length of the game. Testing strategies many times leads to cyclical play.U.S. National Science Foundation (Grants SES-1458992 and SES- 1728853), the U.S. Army Research Office (Grants W911NF-17-1-0134 MSN201957), Project ECO2017-83147- C2-2-P (MINECO/AEI/FEDER, UE), and the Spanish Ministerio de Educación, Cultura, y Deporte (Grants PRX15/00362 and PRX16/00048