Count-Based Exploration in Feature Space for Reinforcement Learning

We introduce a new count-based optimistic exploration algorithm for Reinforcement Learning (RL) that is feasible in environments with high-dimensional state-action spaces. The success of RL algorithms in these domains depends crucially on generalisation from limited training experience. Function approximation techniques enable RL agents to generalise in order to estimate the value of unvisited states, but at present few methods enable generalisation regarding uncertainty. This has prevented the combination of scalable RL algorithms with efficient exploration strategies that drive the agent to reduce its uncertainty. We present a new method for computing a generalised state visit-count, which allows the agent to estimate the uncertainty associated with any state. Our \phi-pseudocount achieves generalisation by exploiting same feature representation of the state space that is used for value function approximation. States that have less frequently observed features are deemed more uncertain. The \phi-Exploration-Bonus algorithm rewards the agent for exploring in feature space rather than in the untransformed state space. The method is simpler and less computationally expensive than some previous proposals, and achieves near state-of-the-art results on high-dimensional RL benchmarks.Comment: Conference: Twenty-sixth International Joint Conference on Artificial Intelligence (IJCAI-17), 8 pages, 1 figur

Neutron matter at zero temperature with auxiliary field diffusion Monte Carlo

The recently developed auxiliary field diffusion Monte Carlo method is applied to compute the equation of state and the compressibility of neutron matter. By combining diffusion Monte Carlo for the spatial degrees of freedom and auxiliary field Monte Carlo to separate the spin-isospin operators, quantum Monte Carlo can be used to simulate the ground state of many nucleon systems (A\alt 100). We use a path constraint to control the fermion sign problem. We have made simulations for realistic interactions, which include tensor and spin--orbit two--body potentials as well as three-nucleon forces. The Argonne $v_8'$ and $v_6'$ two nucleon potentials plus the Urbana or Illinois three-nucleon potentials have been used in our calculations. We compare with fermion hypernetted chain results. We report results of a Periodic Box--FHNC calculation, which is also used to estimate the finite size corrections to our quantum Monte Carlo simulations. Our AFDMC results for $v_6$ models of pure neutron matter are in reasonably good agreement with equivalent Correlated Basis Function (CBF) calculations, providing energies per particle which are slightly lower than the CBF ones. However, the inclusion of the spin--orbit force leads to quite different results particularly at relatively high densities. The resulting equation of state from AFDMC calculations is harder than the one from previous Fermi hypernetted chain studies commonly used to determine the neutron star structure.Comment: 15 pages, 15 tables and 5 figure