XCS with eligibility traces

Belgium Herbarium image of Meise Botanic Garden

A brief history of learning classifier systems: from CS-1 to XCS and its variants

© 2015, Springer-Verlag Berlin Heidelberg. The direction set by Wilson’s XCS is that modern Learning Classifier Systems can be characterized by their use of rule accuracy as the utility metric for the search algorithm(s) discovering useful rules. Such searching typically takes place within the restricted space of co-active rules for efficiency. This paper gives an overview of the evolution of Learning Classifier Systems up to XCS, and then of some of the subsequent developments of Wilson’s algorithm to different types of learning

Fuzzy and tile coding approximation techniques for coevolution in reinforcement learning

PhDThis thesis investigates reinforcement learning algorithms suitable for learning in large state space problems and coevolution. In order to learn in large state spaces, the state space must be collapsed to a computationally feasible size and then generalised about. This thesis presents two new implementations of the classic temporal difference (TD) reinforcement learning algorithm Sarsa that utilise fuzzy logic principles for approximation, FQ Sarsa and Fuzzy Sarsa. The effectiveness of these two fuzzy reinforcement learning algorithms is investigated in the context of an agent marketplace. It presents a practical investigation into the design of fuzzy membership functions and tile coding schemas. A critical analysis of the fuzzy algorithms to a related technique in function approximation, a coarse coding approach called tile coding is given in the context of three different simulation environments; the mountain-car problem, a predator/prey gridworld and an agent marketplace. A further comparison between Fuzzy Sarsa and tile coding in the context of the nonstationary environments of the agent marketplace and predator/prey gridworld is presented. This thesis shows that the Fuzzy Sarsa algorithm achieves a significant reduction of state space over traditional Sarsa, without loss of the finer detail that the FQ Sarsa algorithm experiences. It also shows that Fuzzy Sarsa and gradient descent Sarsa(λ) with tile coding learn similar levels of distinction against a stationary strategy. Finally, this thesis demonstrates that Fuzzy Sarsa performs better in a competitive multiagent domain than the tile coding solution

Mass variance from archival X-ray properties of dark energy survey year-1 galaxy clusters

For abstract see published article

Sussing merger trees: the influence of the halo finder

Merger tree codes are routinely used to follow the growth and merger of dark matter haloes in simulations of cosmic structure formation. Whereas in Srisawat et. al. we compared the trees built using a wide variety of such codes, here we study the influence of the underlying halo catalogue upon the resulting trees. We observe that the specifics of halo finding itself greatly influences the constructed merger trees. We find that the choices made to define the halo mass are of prime importance. For instance, amongst many potential options different finders select self-bound objects or spherical regions of defined overdensity, decide whether or not to include substructures within the mass returned and vary in their initial particle selection. The impact of these decisions is seen in tree length (the period of time a particularly halo can be traced back through the simulation), branching ratio (essentially the merger rate of subhaloes) and mass evolution. We therefore conclude that the choice of the underlying halo finder is more relevant to the process of building merger trees than the tree builder itself. We also report on some built-in features of specific merger tree codes that (sometimes) help to improve the quality of the merger trees produced

Modèle informatique du coapprentissage des ganglions de la base et du cortex : l'apprentissage par renforcement et le développement de représentations

Tout au long de la vie, le cerveau développe des représentations de son environnement permettant à l’individu d’en tirer meilleur profit. Comment ces représentations se développent-elles pendant la quête de récompenses demeure un mystère. Il est raisonnable de penser que le cortex est le siège de ces représentations et que les ganglions de la base jouent un rôle important dans la maximisation des récompenses. En particulier, les neurones dopaminergiques semblent coder un signal d’erreur de prédiction de récompense. Cette thèse étudie le problème en construisant, à l’aide de l’apprentissage machine, un modèle informatique intégrant de nombreuses évidences neurologiques. Après une introduction au cadre mathématique et à quelques algorithmes de l’apprentissage machine, un survol de l’apprentissage en psychologie et en neuroscience et une revue des modèles de l’apprentissage dans les ganglions de la base, la thèse comporte trois articles. Le premier montre qu’il est possible d’apprendre à maximiser ses récompenses tout en développant de meilleures représentations des entrées. Le second article porte sur l'important problème toujours non résolu de la représentation du temps. Il démontre qu’une représentation du temps peut être acquise automatiquement dans un réseau de neurones artificiels faisant office de mémoire de travail. La représentation développée par le modèle ressemble beaucoup à l’activité de neurones corticaux dans des tâches similaires. De plus, le modèle montre que l’utilisation du signal d’erreur de récompense peut accélérer la construction de ces représentations temporelles. Finalement, il montre qu’une telle représentation acquise automatiquement dans le cortex peut fournir l’information nécessaire aux ganglions de la base pour expliquer le signal dopaminergique. Enfin, le troisième article évalue le pouvoir explicatif et prédictif du modèle sur différentes situations comme la présence ou l’absence d’un stimulus (conditionnement classique ou de trace) pendant l’attente de la récompense. En plus de faire des prédictions très intéressantes en lien avec la littérature sur les intervalles de temps, l’article révèle certaines lacunes du modèle qui devront être améliorées. Bref, cette thèse étend les modèles actuels de l’apprentissage des ganglions de la base et du système dopaminergique au développement concurrent de représentations temporelles dans le cortex et aux interactions de ces deux structures.Throughout lifetime, the brain develops abstract representations of its environment that allow the individual to maximize his benefits. How these representations are developed while trying to acquire rewards remains a mystery. It is reasonable to assume that these representations arise in the cortex and that the basal ganglia are playing an important role in reward maximization. In particular, dopaminergic neurons appear to code a reward prediction error signal. This thesis studies the problem by constructing, using machine learning tools, a computational model that incorporates a number of relevant neurophysiological findings. After an introduction to the machine learning framework and to some of its algorithms, an overview of learning in psychology and neuroscience, and a review of models of learning in the basal ganglia, the thesis comprises three papers. The first article shows that it is possible to learn a better representation of the inputs while learning to maximize reward. The second paper addresses the important and still unresolved problem of the representation of time in the brain. The paper shows that a time representation can be acquired automatically in an artificial neural network acting like a working memory. The representation learned by the model closely resembles the activity of cortical neurons in similar tasks. Moreover, the model shows that the reward prediction error signal could accelerate the development of the temporal representation. Finally, it shows that if such a learned representation exists in the cortex, it could provide the necessary information to the basal ganglia to explain the dopaminergic signal. The third article evaluates the explanatory and predictive power of the model on the effects of differences in task conditions such as the presence or absence of a stimulus (classical versus trace conditioning) while waiting for the reward. Beyond making interesting predictions relevant to the timing literature, the paper reveals some shortcomings of the model that will need to be resolved. In summary, this thesis extends current models of reinforcement learning of the basal ganglia and the dopaminergic system to the concurrent development of representation in the cortex and to the interactions between these two regions