A Java Reinforcement Learning Module for the Recursive Porous Agent Simulation Toolkit : facilitating study and experimentation with reinforcement learning in social science multi-agent simulations

This work details a machine learning tool developed to support computational, agent-based simulation research in the social sciences. Specifically, the Java Reinforcement Learning Module (JReLM) is a platform for implementing reinforcement learning algorithms for use in agent-based simulations. The module was designed for use with the Recursive Porous Agent Simulation Toolkit (Repast), an agent-based simulation platform popular in computational social science research. Background, architecture, and implementation of JReLM are discussed within. This includes explanation of pre-implemented tools and algorithms available for immediate use in Repast simulations. In addition, an account of JReLM\u27s use in an agent-based computational economics simulation is included as an illustrative application. Directions for further development and future use in ongoing agent-based computational economics work are discussed as well

Learning by Doing vs. Learning from Others in a Principal-Agent Model

We introduce learning in a principal-agent model of stochastic output sharing under moral hazard. Without knowing the agents' preferences and technology the principal tries to learn the optimal agency contract. We implement two learning paradigms - social (learning from others) and individual (learning by doing). We use a social evolutionary learning algorithm (SEL) to represent social learning. Within the individual learning paradigm, we investigate the performance of reinforcement learning (RL), experience-weighted attraction learning (EWA), and individual evolutionary learning (IEL). Overall, our results show that learning in the principal-agent environment is very difficult. This is due to three main reasons: (1) the stochastic environment, (2) a discontinuity in the payoff space in a neighborhood of the optimal contract due to the participation constraint and (3) incorrect evaluation of foregone payoffs in the sequential game principal-agent setting. The first two factors apply to all learning algorithms we study while the third is the main contributor for the failure of the EWA and IEL models. Social learning (SEL), especially combined with selective replication, is much more successful in achieving convergence to the optimal contract than the canonical versions of individual learning from the literature. A modified version of the IEL algorithm using realized payoff evaluation performs better than the other individual learning models; however, it still falls short of the social learning's ability to converge to the optimal contract.learning, principal-agent model, moral hazard

Market Power and Efficiency in a Computational Electricity Market with Discriminatory Double-Auction Pricing

This study reports experimental market power and efficiency outcomes for a computational wholesale electricity market operating in the short run under systematically varied concentration and capacity conditions. The pricing of electricity is determined by means of a clearinghouse double auction with discriminatory mid-point pricing. Buyers and sellers use Roth-Erev individual reinforcement learning to determine their price and quantity offers in each auction round. It is shown that market microstructure is strongly predictive for the relative market power of buyers and sellers, and that high market efficiency is generally attained. These findings are robust for tested changes in individual learning parameters. It is also shown that similar relative market power findings are obtained if the electricity buyer and seller populations instead each engage in social mimicry learning via a genetic algorithm. However, market efficiency is substantially reduced.Wholesale electricity market, Electricity restructuring, Double auction, Market power, Efficiency, Concentration, Capacity, Agent-based computational economics, Roth-Erev reinforcement learning, Genetic algorithm social learning.