Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.Comment: Accepted to NeurIPS Deep RL Workshop 201

Metabolic state alters economic decision making under risk in humans

Background: Animals' attitudes to risk are profoundly influenced by metabolic state (hunger and baseline energy stores). Specifically, animals often express a preference for risky (more variable) food sources when below a metabolic reference point (hungry), and safe (less variable) food sources when sated. Circulating hormones report the status of energy reserves and acute nutrient intake to widespread targets in the central nervous system that regulate feeding behaviour, including brain regions strongly implicated in risk and reward based decision-making in humans. Despite this, physiological influences per se have not been considered previously to influence economic decisions in humans. We hypothesised that baseline metabolic reserves and alterations in metabolic state would systematically modulate decision-making and financial risk-taking in humans. Methodology/Principal Findings: We used a controlled feeding manipulation and assayed decision-making preferences across different metabolic states following a meal. To elicit risk-preference, we presented a sequence of 200 paired lotteries, subjects' task being to select their preferred option from each pair. We also measured prandial suppression of circulating acyl-ghrelin (a centrally-acting orexigenic hormone signalling acute nutrient intake), and circulating leptin levels (providing an assay of energy reserves). We show both immediate and delayed effects on risky decision-making following a meal, and that these changes correlate with an individual's baseline leptin and changes in acyl-ghrelin levels respectively. Conclusions/Significance: We show that human risk preferences are exquisitely sensitive to current metabolic state, in a direction consistent with ecological models of feeding behaviour but not predicted by normative economic theory. These substantive effects of state changes on economic decisions perhaps reflect shared evolutionarily conserved neurobiological mechanisms. We suggest that this sensitivity in human risk-preference to current metabolic state has significant implications for both real-world economic transactions and for aberrant decision-making in eating disorders and obesity

Defining the ABC of gene essentiality in streptococci

Background Utilising next generation sequencing to interrogate saturated bacterial mutant libraries provides unprecedented information for the assignment of genome-wide gene essentiality. Exposure of saturated mutant libraries to specific conditions and subsequent sequencing can be exploited to uncover gene essentiality relevant to the condition. Here we present a barcoded transposon directed insertion-site sequencing (TraDIS) system to define an essential gene list for Streptococcus equi subsp. equi, the causative agent of strangles in horses, for the first time. The gene essentiality data for this group C Streptococcus was compared to that of group A and B streptococci. Results Six barcoded variants of pGh9:ISS1 were designed and used to generate mutant libraries containing between 33,000-66,000 unique mutants. TraDIS was performed on DNA extracted from each library and data were analysed separately and as a combined master pool. Gene essentiality determined that 19.5% of the S. equi genome was essential. Gene essentialities were compared to those of group A and group B streptococci, identifying concordances of 90.2% and 89.4%, respectively and an overall concordance of 83.7% between the three species. Conclusions The use of barcoded pGh9:ISS1 to generate mutant libraries provides a highly useful tool for the assignment of gene function in S. equi and other streptococci. The shared essential gene set of group A, B and C streptococci provides further evidence of the close genetic relationships between these important pathogenic bacteria. Therefore, the ABC of gene essentiality reported here provides a solid foundation towards reporting the functional genome of streptococci

Conservative treatment of a comminuted cervical fracture in a racehorse

The 'classical' or 'Hangman' neck fracture involves the odontoid peg (process) of the second cervical vertebra (C2), and is described as an axial, dens or odontoid peg fracture in both the veterinary and human literature. Possible surgical treatment in both foals and adult horses requires a technique that allows decompression, anatomical alignment and stabilisation of the odontoid fracture. A limited number of surgical cases in foals have been reported in literature, but never in an adult horse. A mature Irish Thoroughbred racehorse was diagnosed with a type 2a odontoid peg fracture. Clinical signs included reluctance to move the head and neck, a left hind limb lameness and a neurological status of grade 2. The horse was treated conservatively and raced successfully five months after the diagnosed injury

Two-stage directed self-assembly of a cyclic [3]catenane.

Interlocked molecules possess properties and functions that depend upon their intricate connectivity. In addition to the topologically trivial rotaxanes, whose structures may be captured by a planar graph, the topologically non-trivial knots and catenanes represent some of chemistry's most challenging synthetic targets because of the three-dimensional assembly necessary for their construction. Here we report the synthesis of a cyclic [3]catenane, which consists of three mutually interpenetrating rings, via an unusual synthetic route. Five distinct building blocks self-assemble into a heteroleptic triangular framework composed of two joined Fe(II)3L3 circular helicates. Subcomponent exchange then enables specific points in the framework to be linked together to generate the cyclic [3]catenane product. Our method represents an advance both in the intricacy of the metal-templated self-assembly procedure and in the use of selective imine exchange to generate a topologically complex product.This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) and a Marie Curie fellowship for J.J.H. (ITN-2010–264645). The authors thank the Diamond Light Source (UK) for synchrotron beamtime on I19 (MT7984 and MT8464).This is the author accepted manuscript. The final version is available from NPG via http://dx.doi.org/10.1038/nchem.220

A closer look at neuron interaction with track-etched microporous membranes

Microporous membranes support the growth of neurites into and through micro-channels, providing a different type of neural growth platform to conventional dish cultures. Microporous membranes are used to support various types of culture, however, the role of pore diameter in relation to neurite growth through the membrane has not been well characterised. In this study, the human cell line (SH-SY5Y) was differentiated into neuron-like cells and cultured on track-etched microporous membranes with pore and channel diameters selected to accommodate neurite width (0.8 µm to 5 µm). Whilst neurites extended through all pore diameters, the extent of neurite coverage on the non-seeded side of the membranes after 5 days in culture was found to be directly proportional to channel diameter. Neurite growth through membrane pores reduced significantly when neural cultures were non-confluent. Scanning electron microscopy revealed that neurites bridged pores and circumnavigated pore edges – such that the overall likelihood of a neurite entering a pore channel was decreased. These findings highlight the role of pore diameter, cell sheet confluence and contact guidance in directing neurite growth through pores and may be useful in applications that seek to use physical substrates to maintain separate neural populations whilst permitting neurite contact between cultures

Evaluating the Potential Effectiveness of Compensatory Mitigation Strategies for Marine Bycatch

Conservationists are continually seeking new strategies to reverse population declines and safeguard against species extinctions. Here we evaluate the potential efficacy of a recently proposed approach to offset a major anthropogenic threat to many marine vertebrates: incidental bycatch in commercial fisheries operations. This new approach, compensatory mitigation for marine bycatch (CMMB), is conceived as a way to replace or reduce mandated restrictions on fishing activities with compensatory activities (e.g., removal of introduced predators from islands) funded by levies placed on fishers. While efforts are underway to bring CMMB into policy discussions, to date there has not been a detailed evaluation of CMMB's potential as a conservation tool, and in particular, a list of necessary and sufficient criteria that CMMB must meet to be an effective conservation strategy. Here we present a list of criteria to assess CMMB that are tied to critical ecological aspects of the species targeted for conservation, the range of possible mitigation activities, and the multi-species impact of fisheries bycatch. We conclude that, overall, CMMB has little potential for benefit and a substantial potential for harm if implemented to solve most fisheries bycatch problems. In particular, CMMB is likely to be effective only when applied to short-lived and highly-fecund species (not the characteristics of most bycatch-impacted species) and to fisheries that take few non-target species, and especially few non-seabird species (not the characteristics of most fisheries). Thus, CMMB appears to have limited application and should only be implemented after rigorous appraisal on a case-specific basis; otherwise it has the potential to accelerate declines of marine species currently threatened by fisheries bycatch

Cortical recovery of swallowing function in wound botulism

<p>Abstract</p> <p>Background</p> <p>Botulism is a rare disease caused by intoxication leading to muscle weakness and rapidly progressive dysphagia. With adequate therapy signs of recovery can be observed within several days. In the last few years, brain imaging studies carried out in healthy subjects showed activation of the sensorimotor cortex and the insula during volitional swallowing. However, little is known about cortical changes and compensation mechanisms accompanying swallowing pathology.</p> <p>Methods</p> <p>In this study, we applied whole-head magnetoencephalography (MEG) in order to study changes in cortical activation in a 27-year-old patient suffering from wound botulism during recovery from dysphagia. An age-matched group of healthy subjects served as control group. A self-paced swallowing paradigm was performed and data were analyzed using synthetic aperture magnetometry (SAM).</p> <p>Results</p> <p>The first MEG measurement, carried out when the patient still demonstrated severe dysphagia, revealed strongly decreased activation of the somatosensory cortex but a strong activation of the right insula and marked recruitment of the left posterior parietal cortex (PPC). In the second measurement performed five days later after clinical recovery from dysphagia we found a decreased activation in these two areas and a bilateral cortical activation of the primary and secondary sensorimotor cortex comparable to the results seen in a healthy control group.</p> <p>Conclusion</p> <p>These findings indicate parallel development to normalization of swallowing related cortical activation and clinical recovery from dysphagia and highlight the importance of the insula and the PPC for the central coordination of swallowing. The results suggest that MEG examination of swallowing can reflect short-term changes in patients suffering from neurogenic dysphagia.</p