Parallel reward and punishment control in humans and robots: Safe reinforcement learning using the MaxPain algorithm

An important issue in reinforcement learning systems for autonomous agents is whether it makes sense to have separate systems for predicting rewards and punishments. In robotics, learning and control are typically achieved by a single controller, with punishments coded as negative rewards. However in biological systems, some evidence suggests that the brain has a separate system for punishment. Although this may in part be due to biological constraints of implementing negative quantities, it raises the question as to whether there is any computational rationale for keeping reward and punishment prediction operationally distinct. Here we outline a basic argument supporting this idea, based on the proposition that learning best-case predictions (as in Q-learning) does not always achieve the safest behaviour. We introduce a modified RL scheme involving a new algorithm which we call 'MaxPain' - which back-ups worst-case predictions in parallel, and then scales the two predictions in a multi-attribute RL policy. i.e. independently learning 'what to do' as well as 'what not to do' and then combining this information. We show how this scheme can improve performance in benchmark RL environments, including a grid-world experiment and a delayed version of the mountain car experiment. In particular, we demonstrate how early exploration and learning are substantially improved, leading to much 'safer' behaviour. In conclusion, the results illustrate the importance of independent punishment prediction in RL, and provide a testable framework for better understanding punishment (such as pain) and avoidance in humans, in both health and disease

Современные представления о НСV−инфекции

Рассмотрены достижения в изучении этиологии, патогенеза и клиники НСV−инфекции. Описаны современные возможности лабораторной и инструментальной диагностики заболевания, основные принципы лечения в зависимости от тяжести течения и активности патологического процесса в печени.The achievements in the study of the etiology, pathogenesis and clinical manifestations of HCV infection are discussed. Contemporary capabilities of laboratory and instrumental diagnosis of the disease as well as main principles of treatment depending on the severity of the course and activity of the pathological process in the liver are described

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro\u27s TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10 x 10 board, using TD(lambda) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa(lambda) agent with SiLU and dSiLU hidden units

Electronic polymers in lipid membranes

Electrical interfaces between biological cells and man-made electrical devices exist in many forms, but it remains a challenge to bridge the different mechanical and chemical environments of electronic conductors (metals, semiconductors) and biosystems. Here we demonstrate soft electrical interfaces, by integrating the metallic polymer PEDOT-S into lipid membranes. By preparing complexes between alkyl-ammonium salts and PEDOT-S we were able to integrate PEDOT-S into both liposomes and in lipid bilayers on solid surfaces. This is a step towards efficient electronic conduction within lipid membranes. We also demonstrate that the PEDOT-S@alkyl-ammonium:lipid hybrid structures created in this work affect ion channels in the membrane of Xenopus oocytes, which shows the possibility to access and control cell membrane structures with conductive polyelectrolytes

Aerobic power and flight capacity in birds: a phylogenetic test of heart-size hypothesis

Flight capacity is one of the most important innovations in animal evolution; it only evolved in insects, birds, mammals and the extinct pterodactyls. Given that powered flight represents a demanding aerobic activity, an efficient cardiovascular system is essential for the continuous delivery of oxygen to the pectoral muscles during flight. It is well known that the limiting step in the circulation is stroke volume (the volume of blood pumped from the ventricle to the body during each beat), which is determined by the size of the ventricle. Thus, the fresh mass of the heart represents a simple and repeatable anatomical measure of the aerobic power of an animal. Although several authors have compared heart masses across bird species, a phylogenetic comparative analysis is still lacking. By compiling heart sizes for 915 species and applying several statistical procedures controlling for body size and/or testing for adaptive trends in the dataset (e.g. model selection approaches, phylogenetic generalized linear models), we found that (residuals of) heart size is consistently associated with four categories of flight capacity. In general, our results indicate that species exhibiting continuous hovering flight (i.e. hummingbirds) have substantially larger hearts than other groups, species that use flapping flight and gliding show intermediate values, and that species categorized as poor flyers show the smallest values. Our study reveals that on a broad scale, routine flight modes seem to have shaped the energetic requirements of birds sufficiently to be anatomically detected at the comparative level

Potential links between Baltic Sea submarine terraces and groundwater seeping

Submarine groundwater discharge (SGD) influences ocean chemistry, circulation, and the spreading of nutrients and pollutants; it also shapes sea floor morphology. In the Baltic Sea, SGD was linked to the development of terraces and semicircular depressions mapped in an area of the southern Stockholm archipelago, Sweden, in the 1990s. We mapped additional parts of the Stockholm archipelago, areas in Blekinge, southern Sweden, and southern Finland using high-resolution multibeam sonars and sub-bottom profilers to investigate if the sea floor morphological features discovered in the 1990s are widespread and to further address the hypothesis linking their formation to SGD. Sediment coring and sea floor photography conducted with a remotely operated vehicle (ROV) and divers add additional information to the geophysical mapping results. We find that terraces, with general bathymetric expressions of about 1 m and lateral extents of sometimes > 100 m, are widespread in the surveyed areas of the Baltic Sea and are consistently formed in glacial clay. Semicircular depressions, however, are only found in a limited part of a surveyed area east of the island of Asko, southern Stockholm archipelago. While submarine terraces can be produced by several processes, we interpret our results to be in support of the basic hypothesis of terrace formation initially proposed in the 1990s; i.e. groundwater flows through siltier, more permeable layers in glacial clay to discharge at the sea floor, leading to the formation of a sharp terrace when the clay layers above seepage zones are undermined enough to collapse. By linking the terraces to a specific geologic setting, our study further refines the formation hypothesis and thereby forms the foundation for a future assessment of SGD in the Baltic Sea that may use marine geological mapping as a starting point. We propose that SGD through the submarine sea floor terraces is plausible and could be intermittent and linked to periods of higher groundwater levels, implying that to quantify the contribution of freshwater to the Baltic Sea through this potential mechanism, more complex hydrogeological studies are required.Peer reviewe

Evidence for a bimodal distribution of Escherichia coli doubling times below a threshold initial cell concentration

Abstract Background In the process of developing a microplate-based growth assay, we discovered that our test organism, a native E. coli isolate, displayed very uniform doubling times (τ) only up to a certain threshold cell density. Below this cell concentration (≤ 100 -1,000 CFU mL-1 ; ≤ 27-270 CFU well-1) we observed an obvious increase in the τ scatter. Results Working with a food-borne E. coli isolate we found that τ values derived from two different microtiter platereader-based techniques (i.e., optical density with growth time {=OD[t]} fit to the sigmoidal Boltzmann equation or time to calculated 1/2-maximal OD {=tm} as a function of initial cell density {=tm[CI]}) were in excellent agreement with the same parameter acquired from total aerobic plate counting. Thus, using either Luria-Bertani (LB) or defined (MM) media at 37°C, τ ranged between 17-18 (LB) or 51-54 (MM) min. Making use of such OD[t] data we collected many observations of τ as a function of manifold initial or starting cell concentrations (CI). We noticed that τ appeared to be distributed in two populations (bimodal) at low CI. When CI ≤100 CFU mL-1 (stationary phase cells in LB), we found that about 48% of the observed τ values were normally distributed around a mean (μτ1) of 18 ± 0.68 min (± στ1) and 52% with μτ2 = 20 ± 2.5 min (n = 479). However, at higher starting cell densities (CI>100 CFU mL-1), the τ values were distributed unimodally (μτ = 18 ± 0.71 min; n = 174). Inclusion of a small amount of ethyl acetate to the LB caused a collapse of the bimodal to a unimodal form. Comparable bimodal τ distribution results were also observed using E. coli cells diluted from mid-log phase cultures. Similar results were also obtained when using either an E. coli O157:H7 or a Citrobacter strain. When sterile-filtered LB supernatants, which formerly contained relatively low concentrations of bacteria(1,000-10,000 CFU mL-1), were employed as a diluent, there was an evident shift of the two populations towards each other but the bimodal effect was still apparent using either stationary or log phase cells. Conclusion These data argue that there is a dependence of growth rate on starting cell density.</p

Modelling interactions of acid–base balance and respiratory status in the toxicity of metal mixtures in the American oyster Crassostrea virginica

Author Posting. © The Author(s), 2009. This is the author's version of the work. It is posted here by permission of Elsevier B.V. for personal use, not for redistribution. The definitive version was published in Comparative Biochemistry and Physiology - Part A: Molecular & Integrative Physiology 155 (2010): 341-349, doi:10.1016/j.cbpa.2009.11.019.Heavy metals, such as copper, zinc and cadmium, represent some of the most common and serious pollutants in coastal estuaries. In the present study, we used a combination of linear and artificial neural network (ANN) modelling to detect and explore interactions among low-dose mixtures of these heavy metals and their impacts on fundamental physiological processes in tissues of the Eastern oyster, Crassostrea virginica. Animals were exposed to Cd (0.001 – 0.400 μM), Zn (0.001 – 3.059 μM) or Cu (0.002 – 0.787 μM), either alone or in combination for 1 to 27 days. We measured indicators of acid-base balance (hemolymph pH and total CO2), gas exchange (Po2), immunocompetence (total hemocyte counts, numbers of invasive bacteria), antioxidant status (glutathione, GSH), oxidative damage (lipid peroxidation; LPx), and metal accumulation in the gill and the hepatopancreas. Linear analysis showed that oxidative membrane damage from tissue accumulation of environmental metals was correlated with impaired acid-base balance in oysters. ANN analysis revealed interactions of metals with hemolymph acid-base chemistry in predicting oxidative damage that were not evident from linear analyses. These results highlight the usefulness of machine learning approaches, such as ANNs, for improving our ability to recognize and understand the effects of sub-acute exposure to contaminant mixtures.This study was supported by NOAA’s Center of Excellence in Oceans and Human Health at HML and the National Science Foundation