Learning Representations in Model-Free Hierarchical Reinforcement Learning

Common approaches to Reinforcement Learning (RL) are seriously challenged by large-scale applications involving huge state spaces and sparse delayed reward feedback. Hierarchical Reinforcement Learning (HRL) methods attempt to address this scalability issue by learning action selection policies at multiple levels of temporal abstraction. Abstraction can be had by identifying a relatively small set of states that are likely to be useful as subgoals, in concert with the learning of corresponding skill policies to achieve those subgoals. Many approaches to subgoal discovery in HRL depend on the analysis of a model of the environment, but the need to learn such a model introduces its own problems of scale. Once subgoals are identified, skills may be learned through intrinsic motivation, introducing an internal reward signal marking subgoal attainment. In this paper, we present a novel model-free method for subgoal discovery using incremental unsupervised learning over a small memory of the most recent experiences (trajectories) of the agent. When combined with an intrinsic motivation learning mechanism, this method learns both subgoals and skills, based on experiences in the environment. Thus, we offer an original approach to HRL that does not require the acquisition of a model of the environment, suitable for large-scale applications. We demonstrate the efficiency of our method on two RL problems with sparse delayed feedback: a variant of the rooms environment and the first screen of the ATARI 2600 Montezuma's Revenge game

Balancing Selection Pressures, Multiple Objectives, and Neural Modularity to Coevolve Cooperative Agent Behavior

Previous research using evolutionary computation in Multi-Agent Systems indicates that assigning fitness based on team vs.\ individual behavior has a strong impact on the ability of evolved teams of artificial agents to exhibit teamwork in challenging tasks. However, such research only made use of single-objective evolution. In contrast, when a multiobjective evolutionary algorithm is used, populations can be subject to individual-level objectives, team-level objectives, or combinations of the two. This paper explores the performance of cooperatively coevolved teams of agents controlled by artificial neural networks subject to these types of objectives. Specifically, predator agents are evolved to capture scripted prey agents in a torus-shaped grid world. Because of the tension between individual and team behaviors, multiple modes of behavior can be useful, and thus the effect of modular neural networks is also explored. Results demonstrate that fitness rewarding individual behavior is superior to fitness rewarding team behavior, despite being applied to a cooperative task. However, the use of networks with multiple modules allows predators to discover intelligent behavior, regardless of which type of objectives are used

SOAP Services with Clarens: Guide for Developers and Administrators

The Clarens application server enables secure, asynchronous SOAP services to run on a Grid cluster such as one of those of the TeraGrid. There is a Client, who wants to use the service and understands the application domain enough to form a reasonable service request; a Developer, who is a power-user of the TeraGrid, who understands both Clarens and the application domain, and creates and deploys a service on a TeraGrid head node; and there is a Root system administrator, who controls the Clarens installation and the cluster on which it runs. The purpose of this document is to provide all of the information a service developer needs to know in order to deploy a Clarens service, with information also provided for the system administrator of the Clarens installation. First we discuss how each of the three roles see the service

Dynamical effects of exchange symmetry breaking in mixtures of interacting bosons

In a double-well potential, a Bose-Einstein condensate exhibits Josephson oscillations or self-trapping, depending on its initial preparation and on the ratio of inter-particle interaction to inter-well tunneling. Here, we elucidate the role of the exchange symmetry for the dynamics with a mixture of two distinguishable species with identical physical properties, i.e. which are governed by an isospecific interaction and external potential. In the mean-field limit, the spatial population imbalance of the mixture can be described by the dynamics of a single species in an effective potential with modified properties or, equivalently, with an effective total particle number. The oscillation behavior can be tuned by populating the second species while maintaining the spatial population imbalance and all other parameters constant. In the corresponding many-body approach, the single-species description approximates the full counting statistics well also outside the realm of spin-coherent states. The method is extended to general Bose-Hubbard systems and to their classical mean-field limits, which suggests an effective single-species description of multicomponent Bose gases with weakly an-isospecific interactions.Comment: amended and expanded, accepted for Phys. Rev. A, 14 pages, 7 figure

The evolution of planetary nebulae VII. Modelling planetary nebulae of distant stellar systems

By means of hydrodynamical models we do the first investigations of how the properties of planetary nebulae are affected by their metal content and what can be learned from spatially unresolved spectrograms of planetary nebulae in distant stellar systems. We computed a new series of 1D radiation-hydrodynamics planetary nebulae model sequences with central stars of 0.595 M_sun surrounded by initial envelope structures that differ only by their metal content. At selected phases along the evolutionary path, the hydrodynamic terms were switched off, allowing the models to relax for fixed radial structure and radiation field into their equilibrium state with respect to energy and ionisation. The analyses of the line spectra emitted from both the dynamical and static models enabled us to systematically study the influence of hydrodynamics as a function of metallicity and evolution. We also recomputed selected sequences already used in previous publications, but now with different metal abundances. These sequences were used to study the expansion properties of planetary nebulae close to the bright cut-off of the planetary nebula luminosity function. Our simulations show that the metal content strongly influences the expansion of planetary nebulae: the lower the metal content, the weaker the pressure of the stellar wind bubble, but the faster the expansion of the outer shell because of the higher electron temperature. This is in variance with the predictions of the interacting-stellar-winds model (or its variants) according to which only the central-star wind is thought to be responsible for driving the expansion of a planetary nebula. Metal-poor objects around slowly evolving central stars become very dilute and are prone to depart from thermal equilibrium because then adiabatic expansion contributes to gas cooling. ...abridged abstract.Comment: 35 pages, 43 figures, accepted for publication by A&

Was there an ‘Industrious Revolution’ before the Industrial Revolution? An Empirical Exercise for England, c. 1300-1830

It is conventionally assumed that the pre-modern working year was fixed and that consumption varied with changes in wages and prices. This is challenged by the twin theories of the ‘industrious’ revolution and the consumer revolution, positing a longer working year as people earned surplus money to buy novel goods. In this study, we turn the conventional view on its head, fixing consumption rather than labour input. Specifically, we use a basket of basic consumption goods and compute the working year of rural and urban day labourers required to achieve that. By comparing with independent estimates of the actual working year, we find two ‘industrious’ revolutions among rural workers; both, however, are attributable to economic hardship, and we detect no signs of a consumer revolution. For urban labourers, by contrast, a growing gap between their actual working year and the work required to buy the basket provides great scope for a consumer revolution.Consumer Revolution; Cost-of-Living Index; Day Wages; ‘Industrious’ Revolution; Industrial Revolution; Labour Supply; Standard of Living