XCS Classifier System with Experience Replay

XCS constitutes the most deeply investigated classifier system today. It bears strong potentials and comes with inherent capabilities for mastering a variety of different learning tasks. Besides outstanding successes in various classification and regression tasks, XCS also proved very effective in certain multi-step environments from the domain of reinforcement learning. Especially in the latter domain, recent advances have been mainly driven by algorithms which model their policies based on deep neural networks -- among which the Deep-Q-Network (DQN) is a prominent representative. Experience Replay (ER) constitutes one of the crucial factors for the DQN's successes, since it facilitates stabilized training of the neural network-based Q-function approximators. Surprisingly, XCS barely takes advantage of similar mechanisms that leverage stored raw experiences encountered so far. To bridge this gap, this paper investigates the benefits of extending XCS with ER. On the one hand, we demonstrate that for single-step tasks ER bears massive potential for improvements in terms of sample efficiency. On the shady side, however, we reveal that the use of ER might further aggravate well-studied issues not yet solved for XCS when applied to sequential decision problems demanding for long-action-chains

Theoretical XCS parameter settings of learning accurate classifiers

XCS is the most popular type of Learning Classifier System, but setting optimum parameter values is more of an art than a science. Early theoretical work required the impractical assumption that classifier parameters had fully converged with infinite update times. The aim of this work is to derive a theoretical condition to mathematically guarantee that XCS identifies maximally accurate classifiers, such that subsequent deletion methods can be used optimally, in as few updates as possible. Consequently, our theory provides a universally usable setup guide for three important parameter settings; the learning rate, the accuracy update and the threshold for subsumption deletion. XCS with our best parameter settings solves the 70-bit multiplexer problem with only 21% of instances that the standard XCS setup needs. On a highly class-imbalanced multiplexer problem with inaccurate classifiers having more than 99.99% classification accuracy, our theory enables XCS to identify only 100% accurate classifiers as accurate and thus obtain the optimal performance.</p