Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds

We describe a method to use discrete human feedback to enhance the performance of deep learning agents in virtual three-dimensional environments by extending deep-reinforcement learning to model the confidence and consistency of human feedback. This enables deep reinforcement learning algorithms to determine the most appropriate time to listen to the human feedback, exploit the current policy model, or explore the agent's environment. Managing the trade-off between these three strategies allows DRL agents to be robust to inconsistent or intermittent human feedback. Through experimentation using a synthetic oracle, we show that our technique improves the training speed and overall performance of deep reinforcement learning in navigating three-dimensional environments using Minecraft. We further show that our technique is robust to highly innacurate human feedback and can also operate when no human feedback is given

General pattern of transitions between breeding states.

<p>, breeder with previous experiences and , non-breeder with previous experiences for . The transition probabilities are expressed in terms of , the probability of surviving to the next breeding season, and , the probability, conditional on survival, of breeding the next season (the 's will be age-dependent in practice).</p

Prediction of the role of experience in the increase of breeding probability with age.

<p>Under a pure restraint hypothesis, breeding probability is hypothesized to increase as a response to the decline in residual reproductive value with age <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0051016#pone.0051016-Pianka1" target="_blank">[61]</a>. Under a pure constraint hypothesis, breeding probability increases through improved skills <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0051016#pone.0051016-Forslund1" target="_blank">[2]</a>.</p

Change in breeding probability estimates in presence of an additive random individual effect.

<p>Comparison of the estimates of breeding probabilities under the main model (thick line, full symbols) and those for an average individual under the additive random individual effect model (thin line, empty symbols). Squares are for inexperienced individuals, triangles for individuals with one experience, and circles for individuals with 2+ experiences.</p

Breeding probability as a function of age and experience for greater flamingos breeding in the Camargue, south of France.

<p>full squares: no previous breeding episode; full triangles: one previous breeding episode; full circles: 2 or more previous breeding episodes. The curve for inexperienced individuals is that obtained with a normal-year first-year survival of 0.632; the dashed curve below is for a value of 0.763 of the same parameter corresponding to an absence of emigration (see text for details).</p

Appendix A. Computation of proportions of the mixture.

Computation of proportions of the mixture

Data and analyses associated with "Dermal mycobacteriosis and warming sea surface temperatures are associated with elevated mortality of striped bass in Chesapeake Bay"

<div>The following data and analyses (R-code) are associated with the manuscript:</div><div><br></div><div>Groner ML, Hoenig JM, Pradel R, Choquet R, Vogelbein, WK, Gauthier, DT, Friedrichs MAM. Dermal mycobacteriosis and warming sea surface temperatures are associated with elevated mortality of striped bass in Chesapeake Bay. 2018 Ecology and Evolution</div><div><br></div><div><br></div><div><b>Data</b></div><div>1) SB recap data for esurge.txt: Dataset on striped bass mark-recapture from 2005-2013 as described in the manuscript. Formatted for analysis in ESURGE<br></div><div> $COV:tag- tag number of unique fish</div><div> H:year- year of study 1-2005, 2-2006, etc.</div><div> COV:rel_fl- fork length at the time of tagging</div><div>$COV:realage- age of fish at the time of tagging</div><div> RC: indicates of data is right-censored (i.e., fish was removed at capture)</div><div><br></div><div> *Note that this data was modified from the dataset available on dryad (https://datadryad.org/resource/doi:10.5061/dryad.f56v8/3), in order to fit the E-SURGE format.</div><div><br></div><div>2) SB Survival best-fit MMSMR.csv: Estimated survival of striped bass by year and disease state from the best-fit MMSMR model</div><div><br></div><div>3) SB state Transitions best-fit MMSMR.csv: Estimated transitions betwee disease states from the best-fit MMSMR model</div><div><br></div><div>4) SB Encounter rates best-fit MMSMR.csv: Estimated catchability of striped bass by year and disease state from the best-fit MMSMR model</div><div><br></div><div>5) SB Survival temp-dependent.csv: Survival rates of striped bass as a function of disease state and average summer SST</div><div><br></div><div>6) Environmental covariates.csv: yearly averages for average summer SST, #days > 25 C, #days < 5.1 mg/L Dissolved oxygen, average water flow</div><div><b><br></b></div><div><b>R-code:</b></div><div><div>1) Graph SB best-fit MMSMR output.R: graphs results from best fit MMSMR (not including environmental covariates) uses datasets 2,3,4 listed above</div><div><br></div><div>2) MMSMR results with environmental covariates.R: </div><div>graphs results from best model in table 3 (include environmental covariates of survival) uses datasets 5, 6 listed above</div><div><br></div><div>3) Temperature-dependent population projection.R: </div><div>projects and graphs a cohort of 10,000 3+ year old striped bass given different average summer SSTs</div></div><div><br></div><div><br></div

Appendix A. Population growth rate bias and mean-square error in the heterogeneous model and the homogeneous model fitted to simulated data.

Population growth rate bias and mean-square error in the heterogeneous model and the homogeneous model fitted to simulated data

Appendix B. Survival and seniority bias and mean-square error in the model with heterogeneity vs. a model with homogeneity.

Survival and seniority bias and mean-square error in the model with heterogeneity vs. a model with homogeneity

Appendix B. Mark-recovery-recapture analyses of the wild boar data.

Mark-recovery-recapture analyses of the wild boar data