Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Physical activity, smoking, and the incidence of clinically diagnosed insomnia

OBJECTIVE: This study was designed to examine the independent and combined associations of physical activity and smoking on the incidence of doctor-diagnosed insomnia using a nationally representative sample over seven years, taking into account other relevant covariates. METHODS: Participants aged 18 years or older in the 2005 Taiwan National Health Interview Survey (NHIS) with links to National Health Insurance (NHI) claim data between 2005 and 2012 and without diagnosed insomnia before 2005, were selected into this study (n = 12,728). Participants were classified as having insomnia with International Classification of Diseases, Ninth Revision (ICD-9) CM codes 307.41, 307.42, or 780.52. Self-reported smoking status and frequency, duration, and types of leisure-time and non−leisure-time physical activities were collected. Metabolic equivalent (MET) intensity levels for each activity were assigned, and weekly energy expenditure of each activity was calculated and summed. RESULTS: Inactive participants had a higher risk of incident insomnia [hazard ratio (HR) = 1.22, 95% confidence interval (CI) = 1.06–1.42, p = 0.007] than the active group, and ever-smokers were more likely to have incident insomnia than never smokers (HR = 1.45, 95% CI = 1.20–1.76, p < 0.001). Compared with the nonsmoker/active group, the ever-smoker/inactive group had a higher risk of incident insomnia (HR = 1.78, 95% CI = 1.41–2.25, p < 0.001). Sensitivity analyses excluding individuals diagnosed with other sleep disorders or mental disorders yielded similar results, with the ever-smoker/inactive group having the highest risk of insomnia. CONCLUSIONS: Inactive adults and smokers are at higher risk for incident insomnia, highlighting the importance of a healthy lifestyle and pointing to strategies such as encouraging smoking cessation and physical activity to avoid insomnia among adults

Prospective association between late-life physical activity and hospital care utilisation: a 7-year nationwide follow-up study

BACKGROUND: It is still equivocal whether there is a potential role of late-life physical activity in ameliorating the challenges of increasing healthcare expenditure due to the consequence of global population ageing. OBJECTIVE: this study aimed to examine the prospective association between physical activity and subsequent hospital care utilisation in older adults and to explore the optimal dose of physical activity required to reduce hospital care utilisation. DESIGN: this was a prospective cohort study based on the data from the Taiwan 2005 National Health Interview Survey, which were linked to the 2005-12 claims data from the National Health Insurance system. PARTICIPANTS: 1,760 older adults aged 65 or more. METHODS: the frequency, duration and intensity for physical activity were assessed, and total physical activity energy expenditure was estimated. The average annualised hospital care utilisation for the period 2006 through 2012, including number of hospitalisations, number of days in hospital and the costs of hospitalisation, were calculated. RESULTS: older adults engaging in at least moderate volume of physical activity (≥1,000 kcal/week) experienced fewer subsequent hospital admissions and fewer days in hospital than did sedentary individuals, after adjusting for covariates. Trends for reduced hospitalisation costs were also found. These associations persisted in sensitivity analyses, including tests of reverse causation. CONCLUSION: this study has provided evidence that older adults who are at least moderately active may minimise utilisation of hospital care services. The findings highlight the importance of maintaining a physically active lifestyle in later life

A computational study of astrocytic glutamate influence on post-synaptic neuronal excitability

<p><b>Postsynaptic activity due to synaptic and intrinsic currents</b>, triggered by (a) synaptic glutamate [Glu]_syn (b-d) simulation with [Glu]_ast,eq = 1.5mM, 5mM, and 10mM respectively, synaptic currents (I_syn) combined AMPA- and NMDA-mediated currents in response to synaptic glutamate, membrane potential (V_m) of postsynaptic neuron resulting from combination of I_syn and voltage-gated currents (Na⁺, K⁺ and leak). Prolonged time course of synaptic glutamate leads to enhanced synaptic currents (I_syn) and higher frequency postsynaptic firing response (V_m depolarisations) as [Glu]_ast,eq increases.</p

A Computational Study of Astrocytic GABA Release at the Glutamatergic Synapse: EAAT-2 and GAT-3 Coupled Dynamics

Neurotransmitter dynamics within neuronal synapses can be controlled by astrocytes and reflect key contributors to neuronal activity. In particular, Glutamate (Glu) released by activated neurons is predominantly removed from the synaptic space by perisynaptic astrocytic transporters EAAT-2 (GLT-1). In previous work, we showed that the time course of Glu transport is affected by ionic concentration gradients either side of the astrocytic membrane and has the propensity for influencing postsynaptic neuronal excitability. Experimental findings co-localize GABA transporters GAT-3 with EAAT-2 on the perisynaptic astrocytic membrane. While these transporters are unlikely to facilitate the uptake of synaptic GABA, this paper presents simulation results which demonstrate the coupling of EAAT-2 and GAT-3, giving rise to the ionic-dependent reversed transport of GAT-3. The resulting efflux of GABA from the astrocyte to the synaptic space reflects an important astrocytic mechanism for modulation of hyperexcitability. Key results also illustrate an astrocytic-mediated modulation of synaptic neuronal excitation by released GABA at the glutamatergic synapse

Most vital segment barriers

We study continuous analogues of "vitality" for discrete network flows/paths, and consider problems related to placing segment barriers that have highest impact on a flow/path in a polygonal domain. This extends the graph-theoretic notion of "most vital arcs" for flows/paths to geometric environments. We give hardness results and efficient algorithms for various versions of the problem, (almost) completely separating hard and polynomially-solvable cases

Determinants of Bone and Blood Lead Levels among Minorities Living in the Boston Area

We measured blood and bone lead levels among minority individuals who live in some of Boston’s neighborhoods with high minority representation. Compared with samples of predominantly white subjects we had studied before, the 84 volunteers in this study (33:67 male:female ratio; 31–72 years of age) had similar educational, occupational, and smoking profiles and mean blood, tibia, and patella lead levels (3 μg/dL, 11.9 μg/g, and 14.2 μg/g, respectively) that were also similar. The slopes of the univariate regressions of blood, tibia, and patella lead versus age were 0.10 μg/dL/year (p < 0.001), 0.45 μg/g/year (p < 0.001), and 0.73 μg/g/year (p < 0.001), respectively. Analyses of smoothing curves and regression lines for tibia and patella lead suggested an inflection point at 55 years of age, with slopes for subjects ≥ 55 years of age that were not only steeper than those of younger subjects but also substantially steeper than those observed for individuals > 55 years of age in studies of predominantly white participants. This apparent racial disparity at older ages may be related to differences in historic occupational and/or environmental exposures, or possibly the lower rates of bone turnover that are known to occur in postmenopausal black women. The higher levels of lead accumulation seen in this age group are of concern because such levels have been shown in other studies to predict elevated risks of chronic disease such as hypertension and cognitive dysfunction. Additional research on bone lead levels in minorities and their socioeconomic and racial determinants is needed

Identification and validation of oncologic miRNA biomarkers for Luminal A-like breast cancer

Introduction: Breast cancer is a common disease with distinct tumor subtypes phenotypically characterized by ER and HER2/neu receptor status. MiRNAs play regulatory roles in tumor initiation and progression, and altered miRNA expression has been demonstrated in a variety of cancer states presenting the potential for exploitation as cancer biomarkers. Blood provides an excellent medium for biomarker discovery. This study investigated systemic miRNAs differentially expressed in Luminal A-like (ER+PR+HER2/neu-) breast cancer and their effectiveness as oncologic biomarkers in the clinical setting. Methods: Blood samples were prospectively collected from patients with Luminal A-like breast cancer (n=54) and controls (n=56). RNA was extracted, reverse transcribed and subjected to microarray analysis (n=10 Luminal A-like; n=10 Control). Differentially expressed miRNAs were identified by artificial neural network (ANN) data-mining algorithms. Expression of specific miRNAs was validated by RQ-PCR (n=44 Luminal A; n=46 Control) and potential relationships between circulating miRNA levels and clinicopathological features of breast cancer were investigated. Results: Microarray analysis identified 76 differentially expressed miRNAs. ANN revealed 10 miRNAs for further analysis ( miR-19b, miR-29a, miR-93, miR-181a, miR-182, miR-223, miR-301a, miR-423-5p, miR-486-5 and miR-652 ). The biomarker potential of 4 miRNAs ( miR-29a, miR-181a , miR-223 and miR-652 ) was confirmed by RQ-PCR, with significantly reduced expression in blood of women with Luminal A-like breast tumors compared to healthy controls (p=0.001, 0.004, 0.009 and 0.004 respectively). Binary logistic regression confirmed that combination of 3 of these miRNAs ( miR-29a, miR-181a and miR-652 ) could reliably differentiate between cancers and controls with an AUC of 0.80. Conclusion: This study provides insight into the underlying molecular portrait of Luminal A-like breast cancer subtype. From an initial 76 miRNAs, 4 were validated with altered expression in the blood of women with Luminal A-like breast cancer. The expression profiles of these 3 miRNAs, in combination with mammography, has potential to facilitate accurate subtype- specific breast tumor detection

Electrically controlled long-distance spin transport through an antiferromagnetic insulator

Spintronics uses spins, the intrinsic angular momentum of electrons, as an alternative for the electron charge. Its long-term goal is in the development of beyond-Moore low dissipation technology devices. Recent progress demonstrated the long-distance transport of spin signals across ferromagnetic insulators. Antiferromagnetically ordered materials are however the most common class of magnetic materials with several crucial advantages over ferromagnetic systems. In contrast to the latter, antiferromagnets exhibit no net magnetic moment, which renders them stable and impervious to external fields. In addition, they can be operated at THz frequencies. While fundamentally their properties bode well for spin transport, previous indirect observations indicate that spin transmission through antiferromagnets is limited to short distances of a few nanometers. Here we demonstrate the long-distance, over tens of micrometers, propagation of spin currents through hematite (\alpha-Fe2O3), the most common antiferromagnetic iron oxide, exploiting the spin Hall effect for spin injection. We control the spin current flow by the interfacial spin-bias and by tuning the antiferromagnetic resonance frequency with an external magnetic field. This simple antiferromagnetic insulator is shown to convey spin information parallel to the compensated moment (N\'eel order) over distances exceeding tens of micrometers. This newly-discovered mechanism transports spin as efficiently as the net magnetic moments in the best-suited complex ferromagnets. Our results pave the way to ultra-fast, low-power antiferromagnet-insulator-based spin-logic devices that operate at room temperature and in the absence of magnetic fields