Exploring Evolution Strategies for Reinforcement Learning in the Obstacle Tower Environment

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced AnalyticsIn 2017 OpenAI demonstrated that it was possible to train an AI agent by using Evolution Strategies (ES), and that the results rivaled standard Reinforcement Learning (RL) techniques on modern benchmarks. Their research effectively showed that Evolution Strategies is a viable alternative to traditional Reinforcement Learning techniques, and that it bypasses many of Reinforcement Learning’s inconveniences, notably the use of backpropagation. The Obstacle Tower environment aims to set a new Reinforcement Learning benchmark by challenging Artificial Intelligence (AI) agents to traverse 3-Dimensional procedurally generated levels using a real-time 3-Dimensional physics system. The environment tests an agent’s ability to generalize by requiring it to optimize aspects that are common in many Reinforcement Learning environments, but rarely combined in the same environment: vision, planning, and control. In this research, the original implementation of OpenAI’s Evolution Strategies algorithm was applied for the first time to the Obstacle Tower environment to assess how well it performs in a more complex environment, where the agent’s generalization ability is critical. Additionally, in the interest of exploring Evolution Strategies in this environment, common Genetic Algorithm selection and mutation techniques were developed and applied to try and improve the performance of the original Evolution Strategies implementation. Crossover techniques were not explored during this research, as they are rarely applied in Evolution Strategies. The results show that although the basic implementation of Evolution Strategies does not perform well in the complex Obstacle Tower environment, it is possible to improve its performance by applying different evolution methods borrowed from Genetic Algorithm (GA), which are algorithms belonging to the same family as Evolution Strategies

The quantum theory of time: a calculus for q-numbers

In quantum theory, physical systems are usually assumed to evolve relative to a c-number time. This c-number time is unphysical and has turned out to be unnecessary for explaining dynamics: in the timeless approach to quantum theory developed by Page & Wootters (1983), subsystems of a stationary universe can instead evolve relative to a 'clock', which is a quantum system with a q-number time observable. Page & Wootters formulated their construction in the Schr\"odinger picture and left open the possibility that the c-number time still plays an explanatory role in the Heisenberg picture. I formulate their construction in the Heisenberg picture and demonstrate that c-number time is completely unnecessary in that picture, too. When the Page-Wootters construction is formulated in the Heisenberg picture, the descriptors of physical systems are functions of the clock's q-number time, and derivatives with respect to this q-number time can be defined in terms of the clock's algebra of observables, resulting in a calculus for q-numbers

Functional implications of demyelination and the molecular control of remyelination in the adult mouse.

Oligodendrocyte (OL) loss contributes to the functional deficits underlying diseases with a demyelinating component (Gajjar et al., 1997; Miller et al., 2002). Remyelination can restore these deficits (Duncan et al., 2009). Chapter 1 is an introduction to de- and re-myelination thus providing the necessary background information for consideration in subsequent chapters. Ethidium bromide (EB) has been extensively used in the rat and cat as a model of spinal cord demyelination. However, this lesion has not been addressed in the adult mouse. Therefore, Chapters 2 and 3 characterize a model of chronic EB-induced spinal cord demyelination in the mouse which can be assessed behaviorally and electrophysiologically. MicroRNAs (miRNAs) possess both cell type- and differentiation stagespecific expression patterns (Lau et al., 2008). In oligodendrocytes (OLs), miRNAs regulate development and differentiation in vivo and in vitro, respectively (Lin et al., 2009; Zhao et al., 2010 and Dugas et al., 2010). However, it is unclear whether miRNAs involved in remyelination are distinct from those observed during normal myelination / development. Treatment of adult mice with the copper chelator cuprizone demyelinates specific brain regions which remyelinate following cuprizone cessation (Torkildsen et al., 2008). Therefore, Chapter 4 focuses on the role of oligodendroglial-specific miRNAs in cuprizone-induced de- and re-myelination. Lastly, Chapter 5 summarizes all of the data provided herein with an emphasis on clinical significance and therapeutic potential

Dialoog Klimaat en de Stad A17 : tussenrapportage fase 1 en PvA fase 2

De gemiddelde temperaturen in stedelijke gebieden liggen hoger dan in de omringende landelijke gebieden tengevolge van het warmte eiland effect. Het project is gericht op implementatie van maatregelen in die sterk verstedelijkte regio’s in Nederland die naar verwachting door zullen groeien in de nabije toekoms

Dutch retail trade on the rise? Relation between competition, innovation and productivity

The Dutch retail trade demonstrated a relatively meagre performance in terms of productivity (growth) during the 1990s, especially seen from an international perspective. This study analyses the productivity performance of the Dutch retail trade in more detail, and focuses on competition and innovation as two main drivers of productivity growth. More precisely, it takes the mutual relationship between competition, innovation and productivity explicitly into account. Between 1993 and 2002 changes in competition varied substantially within the retail trade. However, on average competition slightly declined. Furthermore, only a few firms in the Dutch retail trade innovate. Regression analysis reveals that both competition and innovation enhance productivity growth directly. Further, fiercer competition induces more innovation, and consequently also raises productivity indirectly via innovation.

Multiple erythroid isoforms of human long-chain acyl-CoA synthetases are produced by switch of the fatty acid gate domains

BACKGROUND: The formation of acyl-CoA by the action of acyl-CoA synthetases plays a crucial role in membrane lipid turnover, including the plasma membrane of erythrocytes. In human, five Acyl-CoA Synthetase Long-chain (ACSL) genes have been identified with as many as 3 different transcript variants for each. RESULTS: Acyl-CoA Synthetase Long-chain member 6 (ACSL6) is responsible for activation of long-chain fatty acids in erythrocytes. Two additional transcript variants were also isolated from brain and testis. We report the expression in reticulocytes of two new variants and of the one isolated from brain. All three represented different spliced variants of a mutually exclusive exon pair. They encode a slightly different short motif which contains a conserved structural domain, the fatty acid Gate domain. The motifs differ in the presence of either the aromatic residue phenylalanine (Phe) or tyrosine (Tyr). Based on homology, two new isoforms for the closely related ACSL1 were predicted and characterized. One represented a switch of the Phe- to the Tyr-Gate domain motif, the other resulted from the exclusion of both. Swapping of this motif also appears to be common in all mammalian ACSL member 1 and 6 homologs. CONCLUSION: We propose that a Phe to Tyr substitution or deletion of the Gate domain, is the structural reason for the conserved alternative splicing that affects these motifs. Our findings support our hypothesis that this region is structurally important to define the activity of these enzymes

A Comparative Framing Analysis of Embedded and Behind-the-Lines Reporting on the 2003 Iraq War

Although a contested position, we believe that reporters and editors frame the news in a way that reflects their personal feelings and newsroom culture (Kuypers, 1997, 2002, 2005; Cooper, in press). Audiences usually receive their political news from only a few press sources; rarely do they read the original statements of those being reported upon