Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds

We describe a method to use discrete human feedback to enhance the performance of deep learning agents in virtual three-dimensional environments by extending deep-reinforcement learning to model the confidence and consistency of human feedback. This enables deep reinforcement learning algorithms to determine the most appropriate time to listen to the human feedback, exploit the current policy model, or explore the agent's environment. Managing the trade-off between these three strategies allows DRL agents to be robust to inconsistent or intermittent human feedback. Through experimentation using a synthetic oracle, we show that our technique improves the training speed and overall performance of deep reinforcement learning in navigating three-dimensional environments using Minecraft. We further show that our technique is robust to highly innacurate human feedback and can also operate when no human feedback is given

Appendix A. An explanation of the fitting of the cause-specific mortality rates model with the MARK program.

An explanation of the fitting of the cause-specific mortality rates model with the MARK program

Appendix B. The parameter index matrices of the cause-specific mortality rates model used in the MARK program.

The parameter index matrices of the cause-specific mortality rates model used in the MARK program

General pattern of transitions between breeding states.

<p>, breeder with previous experiences and , non-breeder with previous experiences for . The transition probabilities are expressed in terms of , the probability of surviving to the next breeding season, and , the probability, conditional on survival, of breeding the next season (the 's will be age-dependent in practice).</p

Appendix D. Rank and identifiability of our multi-event models.

Rank and identifiability of our multi-event models

Appendix B. Decomposition of the transition and event matrices of our model.

Decomposition of the transition and event matrices of our model

Appendix F. Goodness-of-fit tests.

Goodness-of-fit tests

Breeding probability as a function of age and experience for greater flamingos breeding in the Camargue, south of France.

<p>full squares: no previous breeding episode; full triangles: one previous breeding episode; full circles: 2 or more previous breeding episodes. The curve for inexperienced individuals is that obtained with a normal-year first-year survival of 0.632; the dashed curve below is for a value of 0.763 of the same parameter corresponding to an absence of emigration (see text for details).</p

Prediction of the role of experience in the increase of breeding probability with age.

<p>Under a pure restraint hypothesis, breeding probability is hypothesized to increase as a response to the decline in residual reproductive value with age <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0051016#pone.0051016-Pianka1" target="_blank">[61]</a>. Under a pure constraint hypothesis, breeding probability increases through improved skills <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0051016#pone.0051016-Forslund1" target="_blank">[2]</a>.</p

Appendix G. Complete results of model selection.

Complete results of model selection