Reinforcement Learning in Sparse-Reward Environments with Hindsight Policy Gradients

A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enabling sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this letter, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency

Hindsight policy gradients

A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.Comment: Accepted to ICLR 201

Effect of the cation structure on the properties of homobaric imidazolium ionic liquids

In this work we investigate the structure–property relationships in a series of alkylimidazolium ionic liquids with almost identical molecular weight. Using a combination of theoretical calculations and experimental measurements, we have shown that re-arranging the alkyl side chain or adding functional groups results in quite distinct features in the resultant ILs. The synthesised ILs, although structurally very similar, cover a wide spectrum of properties ranging from highly fluid, glass forming liquids to high melting point crystalline salts. Theoretical ab initio calculations provide insight on minimum energy orientations for the cations, which then are compared to experimental X-ray crystallography measurements to extract information on hydrogen bonding and to verify our understanding of the studied structures. Molecular dynamics simulations of the simplest (core) ionic liquids are used in order to help us interpret our experimental results and understand better why methylation of C2 position of the imidazolium ring results in ILs with such different properties compared to their non-methylated analogues

Local Fuel Starvation Degradation of an Automotive PEMFC Full Size Stack

Special Issue: 23rd EFCF “Low‐Temperature Fuel Cells, Electrolyzers, H2‐Processing Forum” (EFCF2019

Sustainable Electrochemical Depolymerization of Lignin in Reusable Ionic Liquids

Lignin’s aromatic building blocks provide a chemical resource that is, in theory, ideal for substitution of aromatic petrochemicals. Moreover, degradation and valorization of lignin has the potential to generate many high-value chemicals for technical applications. In this study, electrochemical degradation of alkali and Organosolv lignin was performed using the ionic liquids 1-ethyl-3-methylimidazolium trifluoromethanesulfonate and triethylammonium methanesulfonate. The extensive degradation of the investigated lignins with simultaneous almost full recovery of the electrolyte materials provided a sustainable alternative to more common lignin degradation processes. We demonstrate here that both the presence (and the absence) of water during electrolysis and proton transport reactions had significant impact on the degradation efficiency. Hydrogen peroxide radical formation promoted certain electrochemical mechanisms in electrolyte systems “contaminated” with water and increased yields of low molecular weight products significantly. The proposed mechanisms were tentatively confirmed by determining product distributions using a combination of liquid chromatography-mass spectrometry and gas-chromatography-mass spectrometry, allowing measurement of both polar versus non-polar as well as volatile versus non-volatile components in the mixtures

Abnormal processing of emotional prosody in Williams syndrome: an event-related potentials study

Williams syndrome (WS), a neurodevelopmental genetic disorder due to a microdeletion in chromosome 7, is described as displaying an intriguing socio-cognitive phenotype. Deficits in prosody production and comprehension have been consistently reported in behavioral studies. It remains, however, to be clarified the neurobiological processes underlying prosody processing in WS. This study aimed at characterizing the electrophysiological response to neutral, happy, and angry prosody in WS, and examining if this response was dependent on the semantic content of the utterance. A group of 12 participants (5 female and 7male), diagnosed with WS, with age range between 9 and 31 years, was compared with a group of typically developing participants, individually matched for chronological age, gender and laterality. After inspection of EEG artifacts, data from 9 participants with WS and 10 controls were included in ERP analyses. Participants were presented with neutral, positive and negative sentences, in two conditions: (1) with intelligible semantic and syntactic information; (2) with unintelligible semantic and syntactic information (‘pure prosody’ condition). They were asked to decide which emotion was underlying the auditory sentence. Atypical event-related potentials (ERP) components were related with prosodic processing (N100, P200, N300) in WS. In particular, reduced N100 was observed for prosody sentences with semantic content; more positive P200 for sentences with semantic content, in particular for happy and angry intonations; and reduced N300 for both types of sentence conditions. These findings suggest abnormalities in early auditory processing, indicating a bottomup contribution to the impairment in emotional prosody processing and comprehension. Also, at least for N100 and P200, they suggest the top-down contributions of semantic processes in the sensory processing of speech. This study showed, for the first time, that abnormalities in ERP measures of early auditory processing in WS are also present during the processing of emotional vocal information. This may represent a physiological signature of underlying impaired on-line language and socio-emotional processing.This work was supported by a Doctoral Grant (SFRH/BD/35882/2007) awarded to APP, as well as by the grant PIC/IC/83290/2007Fundação para a Ciência e a Tecnologia (FCT

Ultra-processed food consumption and risk of obesity: a prospective cohort study of UK Biobank

Objective: To examine the associations between ultra-processed food consumption and risk of obesity among UK adults. Methods: Participants aged 40-69 years at recruitment in the UK Biobank (2006-2019) with dietary intakes collected using 24-hour recall and repeated measures of adiposity - body mass index (BMI), waist circumference (WC) and percentage of body fat (% BF) - were included (N=22,659; median follow-up: 5 years). Ultra-processed foods were identified using the NOVA classification and their consumption was expressed as a percentage of total energy intake. Multivariable Cox proportional hazards regression models were used to estimate hazard ratios (HR) of several indicators of obesity according to ultra-processed food consumption. Models were adjusted for sociodemographic and lifestyle characteristics. Results: 947 incident cases of overall obesity (BMI≥30 kg/m2) and 1,900 incident cases of abdominal obesity (men: WC≥102cm, women: WC≥88cm) were identified during follow-up. Participants in the highest quartile of ultra-processed food consumption had significantly higher risk of developing overall obesity (HR: 1.79; 95%CI: 1.06─3.03) and abdominal obesity (HR: 1.30; 95%CI: 1.14─1.48). They had higher risk of experiencing a ≥5% increase in BMI (HR: 1.31; 95%CI: 1.20─1.43), WC (HR: 1.35; 95%CI: 1.25─1.45) and %BF (HR: 1.14; 95%CI: 1.03─1.25), than those in the lowest quartile of consumption. Conclusions: Our findings provide evidence that higher consumption of ultra-processed food is strongly associated with a higher risk of multiple indicators of obesity in the UK adult population. Policy makers should consider actions that promote consumption of fresh or minimally processed foods and reduce consumption of ultra-processed foods

Dynamics, cation conformation and rotamers in guanidinium ionic liquids with ether groups

Ionic liquids are modern materials with a broad range of applications, including electrochemical devices, the exploitation of sustainable resources and chemical processing. Expanding the chemical space to include novel ion classes allows for the elucidation of novel structure-property relationships and fine tuning for specific applications. We prepared a set of ionic liquids based on the sparsely investigated pentamethyl guanidinium cation with a 2-ethoxy-ethyl side chain in combination with a series of frequently used anions. The resulting properties are compared to a cation with a pentyl side chain lacking ether functionalization. We measured the thermal transitions and transport properties to estimate the performance and trends of this cation class. The samples with imide-type anions form liquids at ambient temperature, and show good transport properties, comparable to imidazolium or ammonium ionic liquids. Despite the dynamics being significantly accelerated, ether functionalization of the cation favors the formation of crystalline solids. Single crystal structure analysis, ab initio calculations and variable temperature nuclear magnetic resonance measurements (VT-NMR) revealed that cation conformations for the ether- and alkyl-chain-substituted are different in both the solid and liquid states. While ether containing cations adopt compact, curled structures, those with pentyl side chains are linear. The Eyring plot revealed that the curled conformation is accompanied by a higher activation energy for rotation around the carbon-nitrogen bonds, due to the coordination of the ether chain as observed by VT-NMR

O IMPACTO DAS ATIVIDADES ANTRÓPICAS NAS ÁREAS ÚMIDAS NA PLANÍCIE DE INUNDAÇÃO DO RIO GRAVATAÍ - RS

A bacia hidrográfica do rio Gravataí se localiza na região leste do estado do Rio Grande do Sul. A importância da preservação das áreas úmidas junto à planície de inundação do rio Gravataí é imprescindível para o equilíbrio ecológico e hidrodinâmico do rio. O objetivo deste trabalho é identificar as mudanças ocorridas nas áreas úmidas. Foram realizados registros fotográficos e análise de imagens satelitais, no período de 2006 a 2012, avaliando a degradação das áreas úmidas