Automated Reinforcement Learning:An Overview

Reinforcement Learning and recently Deep Reinforcement Learning are popular methods for solving sequential decision making problems modeled as Markov Decision Processes. RL modeling of a problem and selecting algorithms and hyper-parameters require careful considerations as different configurations may entail completely different performances. These considerations are mainly the task of RL experts; however, RL is progressively becoming popular in other fields where the researchers and system designers are not RL experts. Besides, many modeling decisions, such as defining state and action space, size of batches and frequency of batch updating, and number of timesteps are typically made manually. For these reasons, automating different components of RL framework is of great importance and it has attracted much attention in recent years. Automated RL provides a framework in which different components of RL including MDP modeling, algorithm selection and hyper-parameter optimization are modeled and defined automatically. In this article, we explore the literature and present recent work that can be used in automated RL. Moreover, we discuss the challenges, open questions and research directions in AutoRL

Estimação de Parâmetros do Aprendizado por Reforço para o Problema de Planejamento de Rotas com Reabastecimento

Path planning is a important problem in mobile robotics. One of the aspects of this type of autonomous vehicles planning refers to observe the fuel-constraints. In this sense, the objective of this work is to estimate the Reinforcement Learning parameters for the path planning problem with refueling. The results indicate that the parameters estimated with the Response Surface Methodology reached the best solutions in most of the experiments. Resumo: O planejamento de rotas ´e um importante problema na rob´otica m´ovel. Uma das vertentes desse tipo de planejamento para ve´ıculos autˆonomos, refere-se a observar as restri¸c˜oes operacionais com combust´ıvel. Nesse sentido, o objetivo deste trabalho ´e estimar os parˆametros do Aprendizado por Refor¸co para o problema planejamento de rotas com reabastecimento. Os resultados apontam que os parˆametros estimados com a Metodologia de Superf´ıcie de Resposta alcan¸caram as melhores solu¸c˜oes na maioria dos experimentos

Analysis of Human Affect and Bug Patterns to Improve Software Quality and Security

The impact of software is ever increasing as more and more systems are being software operated. Despite the usefulness of software, many instances software failures have been causing tremendous losses in lives and dollars. Software failures take place because of bugs (i.e., faults) in the software systems. These bugs cause the program to malfunction or crash and expose security vulnerabilities exploitable by malicious hackers. Studies confirm that software defects and vulnerabilities appear in source code largely due to the human mistakes and errors of the developers. Human performance is impacted by the underlying development process and human affects, such as sentiment and emotion. This thesis examines these human affects of software developers, which have drawn recent interests in the community. For capturing developers’ sentimental and emotional states, we have developed several software tools (i.e., SentiStrength-SE, DEVA, and MarValous). These are novel tools facilitating automatic detection of sentiments and emotions from the software engineering textual artifacts. Using such an automated tool, the developers’ sentimental variations are studied with respect to the underlying development tasks (e.g., bug-fixing, bug-introducing), development periods (i.e., days and times), team sizes and project sizes. We expose opportunities for exploiting developers’ sentiments for higher productivity and improved software quality. While developers’ sentiments and emotions can be leveraged for proactive and active safeguard in identifying and minimizing software bugs, this dissertation also includes in-depth studies of the relationship among various bug patterns, such as software defects, security vulnerabilities, and code smells to find actionable insights in minimizing software bugs and improving software quality and security. Bug patterns are exposed through mining software repositories and bug databases. These bug patterns are crucial in localizing bugs and security vulnerabilities in software codebase for fixing them, predicting portions of software susceptible to failure or exploitation by hackers, devising techniques for automated program repair, and avoiding code constructs and coding idioms that are bug-prone. The software tools produced from this thesis are empirically evaluated using standard measurement metrics (e.g., precision, recall). The findings of all the studies are validated with appropriate tests for statistical significance. Finally, based on our experience and in-depth analysis of the present state of the art, we expose avenues for further research and development towards a holistic approach for developing improved and secure software systems

Excitatory and inhibitory ensembles of the dorsal medial prefrontal cortex during the strengthening and weakening of an appetitive association

Animals associate relevant signals or ‘cues’ to food in order to efficiently gather nutrients. Sparse sets of neurons called ‘neuronal ensembles’ in the dorsal medial prefrontal cortex (dmPFC) play a vital role in associative memory formation. However, how these neurons are recruited into an ensemble to establish appetitive ‘food-cue’ associations during conditioning remains unclear. The aims of this study were twofold: 1) examine the recruitment of both pyramidal cell and interneuron ensembles during the establishment and extinction of appetitive associations in the dmPFC and 2) investigate their function in appetitive conditioning. Here, we took advantage of a microprism-based 2-photon imaging procedure to longitudinally image the dmPFC of Fos-GFP x GAD-tdTomato transgenic mice in vivo. These mice express GFP in strongly activated (Fos expressing) neurons and tdTomato in interneurons, which allowed us to track recently activated pyramidal cells and interneurons over conditioning and extinction. During conditioning, in which behavioural responding became cue-selective, a stable, repeatedly activated neuronal ensemble was recruited from a pyramidal cell pool activated during early learning. Furthermore, repeatedly enhancing the excitability of the initial learning activated pool with chemogenetics throughout training disrupted appetitive learning. In contrast, during extinction, a stable inhibitory ensemble emerged from interneurons activated in early extinction learning. These novel findings reveal ensemble recruitment patterns occurring in the dmPFC during alterations in the strength of food-cue associations

Tuning of reinforcement learning parameters applied to SOP using the Scott–Knott method

In this paper, we present a technique to tune the reinforcement learning (RL) parameters applied to the sequential ordering problem (SOP) using the Scott–Knott method. The RL has been widely recognized as a powerful tool for combinatorial optimization problems, such as travelling salesman and multidimensional knapsack problems. It seems, however, that less attention has been paid to solve the SOP. Here, we have developed a RL structure to solve the SOP that can partially fill that gap. Two traditional RL algorithms, Q-learning and SARSA, have been employed. Three learning specifications have been adopted to analyze the performance of the RL: algorithm type, reinforcement learning function, and € parameter. A complete factorial experiment and the Scott–Knott method are used to find the best combination of factor levels, when the source of variation is statistically different in analysis of variance. The performance of the proposed RL has been tested using benchmarks from the TSPLIB library. In general, the selected parameters indicate that SARSA overwhelms the performance of Q-learning

European Higher Education Area: Challenges for a New Decade

This open access book presents the major outcomes of the fourth edition of the Future of Higher Education – Bologna Process Researchers Conference (FOHE-BPRC 4) which was held in January 2020 and which has already established itself as a landmark in the European higher education environment. The conference is part of the official calendar of the European Higher Education Area (EHEA) for events that promote and sustain the development of EHEA. The conference provides a unique forum for dialogue between researchers, experts and policy makers in the field of higher education, all of which is documented in this proceedings volume. The book focuses on the following five sub-themes: - Furthering the Internationalization of Higher Education: Particular - Challenges in the EHEA - Access and Success for Every Learner in Higher Education - Advancing Learning and Teaching in the EHEA: Innovation and Links With Research - The Future of the EHEA - Principles, Challenges and Ways Forward - Bologna Process in the Global Higher Education Arena. Going Digital? While acknowledging the efforts and achievements so far at EHEA level, the Paris Ministerial Communiqué highlights the need to intensify crossdisciplinary and cross-border cooperation. One of the ways to achieve this objective is to develop more efficient peer-learning activities, involving policymakers and other stakeholders from as many member states as possible for which this book provides a platform. It acknowledges the importance of a continued dialogue between researchers and decisionmakers and benefits from the experience already acquired, this way enabling the higher education community to bring its input into the 2020. European Higher Education Area (EHEA) priorities for 2020 onwards. European Higher Education Area: Challenges for a New Decade marks 21 years of Bologna Process and 10 years of EHEA and brings together an unique collection of contributions that not only reflect on all that has been achieved in these years, but more importantly, shape directions for the future. This book is published under an open access CC BY license

Aeronautical Engineering. A continuing bibliography with indexes, supplement 135, May 1981

This bibliography lists 536 reports, articles, and other documents introduced into the NASA scientific and technical information system in April 1981