Deep Q-learning from Demonstrations

Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages small sets of demonstration data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator's actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 83 million steps to catch up to DQfD's performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 11 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.Comment: Published at AAAI 2018. Previously on arxiv as "Learning from Demonstrations for Real World Reinforcement Learning

The cell biology of smell

The olfactory system detects and discriminates myriad chemical structures across a wide range of concentrations. To meet this task, the system utilizes a large family of G protein–coupled receptors—the odorant receptors—which are the chemical sensors underlying the perception of smell. Interestingly, the odorant receptors are also involved in a number of developmental decisions, including the regulation of their own expression and the patterning of the olfactory sensory neurons' synaptic connections in the brain. This review will focus on the diverse roles of the odorant receptor in the function and development of the olfactory system

Quantitative and functional interrogation of parent-of-origin allelic expression biases in the brain

The maternal and paternal genomes play different roles in mammalian brains as a result of genomic imprinting, an epigenetic regulation leading to differential expression of the parental alleles of some genes. Here we investigate genomic imprinting in the cerebellum using a newly developed Bayesian statistical model that provides unprecedented transcript-level resolution. We uncover 160 imprinted transcripts, including 41 novel and independently validated imprinted genes. Strikingly, many genes exhibit parentally biased—rather than monoallelic—expression, with different magnitudes according to age, organ, and brain region. Developmental changes in parental bias and overall gene expression are strongly correlated, suggesting combined roles in regulating gene dosage. Finally, brain-specific deletion of the paternal, but not maternal, allele of the paternally-biased Bcl-x, (Bcl2l1) results in loss of specific neuron types, supporting the functional significance of parental biases. These findings reveal the remarkable complexity of genomic imprinting, with important implications for understanding the normal and diseased brain. DOI: http://dx.doi.org/10.7554/eLife.07860.00