Learning to play using low-complexity rule-based policies: Illustrations through Ms. Pac-Man

In this article we propose a method that can deal with certain combinatorial reinforcement learning tasks. We demonstrate the approach in the popular Ms. Pac-Man game. We define a set of high-level observation and action modules, from which rule-based policies are constructed automatically. In these policies, actions are temporally extended, and may work concurrently. The policy of the agent is encoded by a compact decision list. The components of the list are selected from a large pool of rules, which can be either hand-crafted or generated automatically. A suitable selection of rules is learnt by the cross-entropy method, a recent global optimization algorithm that fits our framework smoothly. Cross-entropy-optimized policies perform better than our hand-crafted policy, and reach the score of average human players. We argue that learning is successful mainly because (i) policies may apply concurrent actions and thus the policy space is sufficiently rich, (ii) the search is biased towards low-complexity policies and therefore, solutions with a compact description can be found quickly if they exist

An Experimental Evaluation of Position Estimation Methods for Person Localization in Wireless Sensor Networks

In this paper, the localization of persons by means of a Wireless Sensor Network (WSN) is considered. Persons carry on-body sensor nodes and move within a WSN. The location of each person is calculated on this node and communicated through the network to a central data sink for visualization. Applications of such a system could be found in mass casualty events, firefighter scenarios, hospitals or retirement homes for example. For the location estimation on the sensor node, three derivatives of the Kalman Filter and a closed-form solution (CFS) are applied, compared, and evaluated in a real-world scenario. A prototype 65-node ZigBee WSN is implemented and data are collected in in- and outdoor environments with differently positioned on-body nodes. The described estimators are then evaluated off-line on the experimentally collected data. The goal of this paper is to present a comprehensive real-world evaluation of methods for person localization in a WSN based on received signal strength (RSS) range measurements. It is concluded that person localization in in- and outdoor environments is possible under the considered conditions with the considered filters. The compared methods allow for suffciently accurate localization results and are robust against inaccurate range measurements

Activity of cortical and thalamic neurons during the slow (<1 Hz) rhythm in the mouse in vivo

During NREM sleep and under certain types of anaesthesia, the mammalian brain exhibits a distinctive slow (<1 Hz) rhythm. At the cellular level, this rhythm correlates with so-called UP and DOWN membrane potential states. In the neocortex, these UP and DOWN states correspond to periods of intense network activity and widespread neuronal silence, respectively, whereas in thalamocortical (TC) neurons, UP/DOWN states take on a more stereotypical oscillatory form, with UP states commencing with a low-threshold Ca2+ potential (LTCP). Whilst these properties are now well recognised for neurons in cats and rats, whether or not they are also shared by neurons in the mouse is not fully known. To address this issue, we obtained intracellular recordings from neocortical and TC neurons during the slow (<1 Hz) rhythm in anaesthetised mice. We show that UP/DOWN states in this species are broadly similar to those observed in cats and rats, with UP states in neocortical neurons being characterised by a combination of action potential output and intense synaptic activity, whereas UP states in TC neurons always commence with an LTCP. In some neocortical and TC neurons, we observed ‘spikelets’ during UP states, supporting the possible presence of electrical coupling. Lastly, we show that, upon tonic depolarisation, UP/DOWN states in TC neurons are replaced by rhythmic high-threshold bursting at ~5 Hz, as predicted by in vitro studies. Thus, UP/DOWN state generation appears to be an elemental and conserved process in mammals that underlies the slow (<1 Hz) rhythm in several species, including humans