Novelty Search for Deep Reinforcement Learning Policy Network Weights by Action Sequence Edit Metric Distance

Reinforcement learning (RL) problems often feature deceptive local optima, and learning methods that optimize purely for reward signal often fail to learn strategies for overcoming them. Deep neuroevolution and novelty search have been proposed as effective alternatives to gradient-based methods for learning RL policies directly from pixels. In this paper, we introduce and evaluate the use of novelty search over agent action sequences by string edit metric distance as a means for promoting innovation. We also introduce a method for stagnation detection and population resampling inspired by recent developments in the RL community that uses the same mechanisms as novelty search to promote and develop innovative policies. Our methods extend a state-of-the-art method for deep neuroevolution using a simple-yet-effective genetic algorithm (GA) designed to efficiently learn deep RL policy network weights. Experiments using four games from the Atari 2600 benchmark were conducted. Results provide further evidence that GAs are competitive with gradient-based algorithms for deep RL. Results also demonstrate that novelty search over action sequences is an effective source of selection pressure that can be integrated into existing evolutionary algorithms for deep RL.Comment: Submitted to GECCO 201

Algebraic Neural Architecture Representation, Evolutionary Neural Architecture Search, and Novelty Search in Deep Reinforcement Learning

Evolutionary algorithms have recently re-emerged as powerful tools for machine learning and artificial intelligence, especially when combined with advances in deep learning developed over the last decade. In contrast to the use of fixed architectures and rigid learning algorithms, we leveraged the open-endedness of evolutionary algorithms to make both theoretical and methodological contributions to deep reinforcement learning. This thesis explores and develops two major areas at the intersection of evolutionary algorithms and deep reinforcement learning: generative network architectures and behaviour-based optimization. Over three distinct contributions, both theoretical and experimental methods were applied to deliver a novel mathematical framework and experimental method for generative, modular neural network architecture search for reinforcement learning, and a generalized formulation of a behaviour- based optimization framework for reinforcement learning called novelty search. Experimental results indicate that both alternative, behaviour-based optimization and neural architecture search can each be used to improve learning in the popular Atari 2600 benchmark compared to DQN — a popular gradient-based method. These results are in-line with related work demonstrating that strictly gradient-free methods are competitive with gradient-based reinforcement learning. These contributions, together with other successful combinations of evolutionary algorithms and deep learning, demonstrate that alternative architectures and learning algorithms to those conventionally used in deep learning should be seriously investigated in an effort to drive progress in artificial intelligence

Stochastic Pulse Switching in a Degenerate Resonant Optical Medium

Using the idealized integrable Maxwell-Bloch model, we describe random optical-pulse polarization switching along an active optical medium in the Lambda-configuration with disordered occupation numbers of its lower energy sub-level pair. The description combines complete integrability and stochastic dynamics. For the single-soliton pulse, we derive the statistics of the electric-field polarization ellipse at a given point along the medium in closed form. If the average initial population difference of the two lower sub-levels vanishes, we show that the pulse polarization will switch intermittently between the two circular polarizations as it travels along the medium. If this difference does not vanish, the pulse will eventually forever remain in the circular polarization determined by which sub-level is more occupied on average. We also derive the exact expressions for the statistics of the polarization-switching dynamics, such as the probability distribution of the distance between two consecutive switches and the percentage of the distance along the medium the pulse spends in the elliptical polarization of a given orientation in the case of vanishing average initial population difference. We find that the latter distribution is given in terms of the well-known arcsine law

Formation of the Isthmus of Panama

The formation of the Isthmus of Panama stands as one of the greatest natural events of the Cenozoic, driving profound biotic transformations on land and in the oceans. Some recent studies suggest that the Isthmus formed manymillions of years earlier than the widely recognized age of approximately 3 million years ago (Ma), a result that if true would revolutionize our understanding of environmental, ecological, and evolutionary change across the Americas. To bring clarity to the question of when the Isthmus of Panama formed, we provide an exhaustive review and reanalysis of geological, paleontological, and molecular records. These independent lines of evidence converge upon a cohesive narrative of gradually emerging land and constricting seaways,withformationof theIsthmus of Panama sensustricto around 2.8 Ma. The evidence used to support an older isthmus is inconclusive, and we caution against the uncritical acceptance of an isthmus before the Pliocene.Facultad de Ciencias Naturales y Muse

Rare coding variants and X-linked loci associated with age at menarche.

More than 100 loci have been identified for age at menarche by genome-wide association studies; however, collectively these explain only ∼3% of the trait variance. Here we test two overlooked sources of variation in 192,974 European ancestry women: low-frequency protein-coding variants and X-chromosome variants. Five missense/nonsense variants (in ALMS1/LAMB2/TNRC6A/TACR3/PRKAG1) are associated with age at menarche (minor allele frequencies 0.08-4.6%; effect sizes 0.08-1.25 years per allele; P<5 × 10(-8)). In addition, we identify common X-chromosome loci at IGSF1 (rs762080, P=9.4 × 10(-13)) and FAAH2 (rs5914101, P=4.9 × 10(-10)). Highlighted genes implicate cellular energy homeostasis, post-transcriptional gene silencing and fatty-acid amide signalling. A frequently reported mutation in TACR3 for idiopathic hypogonatrophic hypogonadism (p.W275X) is associated with 1.25-year-later menarche (P=2.8 × 10(-11)), illustrating the utility of population studies to estimate the penetrance of reportedly pathogenic mutations. Collectively, these novel variants explain ∼0.5% variance, indicating that these overlooked sources of variation do not substantially explain the 'missing heritability' of this complex trait.UK sponsors (see article for overseas ones): This work made use of data and samples generated by the 1958 Birth Cohort (NCDS). Access to these resources was enabled via the 58READIE Project funded by Wellcome Trust and Medical Research Council (grant numbers WT095219MA and G1001799). A full list of the financial, institutional and personal contributions to the development of the 1958 Birth Cohort Biomedical resource is available at http://www2.le.ac.uk/projects/birthcohort. Genotyping was undertaken as part of the Wellcome Trust Case-Control Consortium (WTCCC) under Wellcome Trust award 076113, and a full list of the investigators who contributed to the generation of the data is available at www.wtccc.org.uk ... The Fenland Study is funded by the Wellcome Trust and the Medical Research Council, as well as by the Support for Science Funding programme and CamStrad. ... SIBS - CRUK ref: C1287/A8459 SEARCH - CRUK ref: A490/A10124 EMBRACE is supported by Cancer Research UK Grants C1287/A10118, C1287/A16563 and C1287/A17523. Genotyping was supported by Cancer Research - UK grant C12292/A11174D and C8197/A16565. Gareth Evans and Fiona Lalloo are supported by an NIHR grant to the Biomedical Research Centre, Manchester. The Investigators at The Institute of Cancer Research and The Royal Marsden NHS Foundation Trust are supported by an NIHR grant to the Biomedical Research Centre at The Institute of Cancer Research and The Royal Marsden NHS Foundation Trust. Ros Eeles and Elizabeth Bancroft are supported by Cancer Research UK Grant C5047/A8385. ... Generation Scotland - Scottish Executive Health Department, Chief Scientist Office, grant number CZD/16/6. Exome array genotyping for GS:SFHS was funded by the Medical Research Council UK. 23andMe - This work was supported in part by NIH Award 2R44HG006981-02 from the National Human Genome Research Institute.This is the final version of the article. It first appeared from NPG via http://dx.doi.org/10.1038/ncomms875

Imputation of coding variants in African Americans: better performance using data from the exome sequencing project

Summary: Although the 1000 Genomes haplotypes are the most commonly used reference panel for imputation, medical sequencing projects are generating large alternate sets of sequenced samples. Imputation in African Americans using 3384 haplotypes from the Exome Sequencing Project, compared with 2184 haplotypes from 1000 Genomes Project, increased effective sample size by 8.3–11.4% for coding variants with minor allele frequency <1%. No loss of imputation quality was observed using a panel built from phenotypic extremes. We recommend using haplotypes from Exome Sequencing Project alone or concatenation of the two panels over quality score-based post-imputation selection or IMPUTE2’s two-panel combination

Evaluating the Effects of SARS-CoV-2 Spike Mutation D614G on Transmissibility and Pathogenicity.

Global dispersal and increasing frequency of the SARS-CoV-2 spike protein variant D614G are suggestive of a selective advantage but may also be due to a random founder effect. We investigate the hypothesis for positive selection of spike D614G in the United Kingdom using more than 25,000 whole genome SARS-CoV-2 sequences. Despite the availability of a large dataset, well represented by both spike 614 variants, not all approaches showed a conclusive signal of positive selection. Population genetic analysis indicates that 614G increases in frequency relative to 614D in a manner consistent with a selective advantage. We do not find any indication that patients infected with the spike 614G variant have higher COVID-19 mortality or clinical severity, but 614G is associated with higher viral load and younger age of patients. Significant differences in growth and size of 614G phylogenetic clusters indicate a need for continued study of this variant

Rare and low-frequency coding variants alter human adult height

Height is a highly heritable, classic polygenic trait with ~700 common associated variants identified so far through genome - wide association studies . Here , we report 83 height - associated coding variants with lower minor allele frequenc ies ( range of 0.1 - 4.8% ) and effects of up to 2 16 cm /allele ( e.g. in IHH , STC2 , AR and CRISPLD2 ) , >10 times the average effect of common variants . In functional follow - up studies, rare height - increasing alleles of STC2 (+1 - 2 cm/allele) compromise d proteolytic inhibition of PAPP - A and increased cleavage of IGFBP - 4 in vitro , resulting in higher bioavailability of insulin - like growth factors . The se 83 height - associated variants overlap genes mutated in monogenic growth disorders and highlight new biological candidates ( e.g. ADAMTS3, IL11RA, NOX4 ) and pathways ( e.g . proteoglycan/ glycosaminoglycan synthesis ) involved in growth . Our results demonstrate that sufficiently large sample sizes can uncover rare and low - frequency variants of moderate to large effect associated with polygenic human phenotypes , and that these variants implicate relevant genes and pathways