Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Biofilm Origin of Clay-Coated Sand Grains

The presence of clay-sized particles and clay minerals in modern sands and ancient sandstones has long presented an interesting problem, because primary depositional processes tend to lead to physical separation of fine- and coarse-grained materials. Numerous processes have been invoked to explain the common presence of clay minerals in sandstones, including infiltration, the codeposition of flocculated muds, and bioturbation-induced sediment mixing. How and why clay minerals form as grain coats at the site of deposition remains uncertain, despite clay-coated sand grains being of paramount importance for subsequent diagenetic sandstone properties. We have identified a new biofilm mechanism that explains clay material attachment to sand grain surfaces that leads to the production of detrital clay coats. This study focuses on a modern estuary using a combination of field work, scanning electron microscopy, petrography, biomarker analysis, and Raman spectroscopy to provide evidence of the pivotal role that biofilms play in the formation of clay-coated sand grains. This study shows that within modern marginal marine systems, clay coats primarily result from adhesive biofilms. This bio-mineral interaction potentially revolutionizes the understanding of clay-coated sand grains and offers a first step to enhanced reservoir quality prediction in ancient and deeply buried sandstones

Major Modes of Variability

This chapter focused on major modes of variability which serve the key role in controlling the regional climate. In terms of tropospheric variability, it defined and discussed ENSO (El Niño Southern Oscillation), NAO (North Atlantic Oscillation), AO and AAO (Arctic Oscillation, Antarctic Oscillation), Indian Monsoon, Indian Ocean Dipole (IOD), PDO (Pacific Decadal Oscillation) and AMO (Atlantic Multidecadal Oscillation). Later it attended stratosphere variability; this constitutes QBO (quasi-biennial oscillation) and SSW (stratospheric sudden warming). Main characteristic features of each of these modes were elaborately discussed

Exercise-based cardiac rehabilitation for coronary heart disease

PublishedReviewThis review is published as a Cochrane Review in the Cochrane Database of Systematic Reviews 2016, Issue 1. Cochrane Reviews are regularly updated as new evidence emerges and in response to comments and criticisms, and the Cochrane Database of Systematic Reviews should be consulted for the most recent version of the Review.Background Coronary heart disease (CHD) is the single most common cause of death globally. However, with falling CHD mortality rates, an increasing number of people live with CHD and may need support to manage their symptoms and prognosis. Exercise-based cardiac rehabilitation (CR) aims to improve the health and outcomes of people with CHD. This is an update of a Cochrane systematic review previously published in 2011. Objectives To assess the effectiveness and cost-effectiveness of exercise-based CR (exercise training alone or in combination with psychosocial or educational interventions) compared with usual care on mortality, morbidity and HRQL in patients with CHD. To explore the potential study level predictors of the effectiveness of exercise-based CR in patients with CHD. Search methods We updated searches from the previous Cochrane review, by searching Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library, Issue 6, 2014) from December 2009 to July 2014. We also searched MEDLINE (Ovid), EMBASE (Ovid), CINAHL (EBSCO) and Science Citation Index Expanded (December 2009 to July 2014). Selection criteria We included randomised controlled trials (RCTs) of exercise-based interventions with at least six months’ follow-up, compared with a no exercise control. The study population comprised men and women of all ages who have had a myocardial infarction (MI), coronary artery bypass graft (CABG) or percutaneous coronary intervention (PCI), or who have angina pectoris, or coronary artery disease. We included RCTs that reported at least one of the following outcomes: mortality, MI, revascularisations, hospitalisations, health-related quality of life (HRQL), or costs. Data collection and analysis Two review authors independently screened all identified references for inclusion based on the above inclusion and exclusion criteria. One author extracted data from the included trials and assessed their risk of bias; a second review author checked data. We stratified meta-analysis by the duration of follow up of trials, i.e. short-term: 6 to 12 months, medium-term: 13 to 36 months, and long-term: > 3 years. Main results This review included 63 trials which randomised 14,486 people with CHD. This latest update identified 16 new trials (3872 participants). The population included predominantly post-MI and post-revascularisation patients and the mean age of patients within the trials ranged from 47.5 to 71.0 years.Women accounted for fewer than 15% of the patients recruited. Overall trial reporting was poor, although there was evidence of an improvement in quality of reporting in more recent trials. As we found no significant difference in the impact of exercise-based CR on clinical outcomes across follow-up, we focused on reporting findings pooled across all trials at their longest follow-up (median 12 months). Exercise-based CR reduced cardiovascular mortality compared with no exercise control (27 trials; risk ratio (RR) 0.74, 95% CI 0.64 to 0.86). There was no reduction in total mortality with CR (47 trials, RR 0.96, 95% CI 0.88 to 1.04). The overall risk of hospital admissions was reduced with CR (15 trials; RR 0.82, 95% CI 0.70 to 0.96) but there was no significant impact on the risk of MI (36 trials; RR 0.90, 95% CI 0.79 to 1.04), CABG (29 trials; RR 0.96, 95% CI 0.80 to 1.16) or PCI (18 trials; RR 0.85, 95% CI 0.70 to 1.04). There was little evidence of statistical heterogeneity across trials for all event outcomes, and there was evidence of small study bias for MI and hospitalisation, but no other outcome. Predictors of clinical outcomes were examined across the longest follow-up of studies using univariate meta-regression. Results show that benefits in outcomes were independent of participants’ CHD case mix (proportion of patients with MI), type of CR (exercise only vs comprehensive rehabilitation) dose of exercise, length of follow-up, trial publication date, setting (centre vs home-based), study location (continent), sample size or risk of bias. Given the heterogeneity in outcome measures and reporting methods, meta-analysis was not undertaken for HRQL. In five out of 20 trials reporting HRQL using validated measures, there was evidence of significant improvement in most or all of the sub-scales with exercise-based CR compared to control at follow-up. Four trial-based economic evaluation studies indicated exercise-based CR to be a potentially cost-effective use of resources in terms of gain in quality-adjusted life years. The quality of the evidence for outcomes reported in the review was rated using the GRADE method. The quality of the evidence varied widely by outcome and ranged from low to moderate. Authors’ conclusions This updated Cochrane review supports the conclusions of the previous version of this review that, compared with no exercise control, exercise-based CR reduces the risk of cardiovascular mortality but not total mortality. We saw a significant reduction in the risk of hospitalisation with CR but not in the risk of MI or revascularisation. We identified further evidence supporting improved HRQL with exercise-based CR. More recent trials were more likely to be well reported and include older and female patients. However, the population studied in this review still consists predominantly of lower risk individuals following MI or revascularisation. Further well conducted RCTs are needed to assess the impact of exercise-based CR in higher risk CHD groups and also those presenting with stable angina. These trials should include validated HRQL outcome measures, explicitly report clinical event outcomes including mortality and hospital admissions, and assess costs and cost-effectiveness

Identification of a new member of the phage shock protein response in Escherichia coli, the phage shock protein g (PspG)

The phage shock protein operon (pspABCDE) of Escherichia coli is strongly up-regulated in response to overexpression of the filamentous phage secretin protein IV (pIV) and by many other stress conditions including defects in protein export. PspA has an established role in maintenance of the proton-motive force of the cell under stress conditions. Here we present evidence for a new member of the phage shock response in E. coli. Using transcriptional profiling, we show that the synthesis of pIV in E. coli leads to a highly restricted response limited to the up-regulation of the psp operon genes and yjbO. The psp operon and yjbO are also up-regulated in response to pIV in Salmonella enterica serovar Typhimurium. yjbO is a highly conserved gene found exclusively in bacteria that contain a psp operon but is physically unlinked to the psp operon. yjbO encodes a putative inner membrane protein that is co-controlled with the psp operon genes and is predicted to be an effector of the psp response in E. coli. We present evidence that yjbO expression is driven by sigma(54)-RNA polymerase, activated by PspF and integration host factor, and negatively regulated by PspA. PspF specifically regulates only members of the PspF regulon: pspABCDE and yjbO. We found that increased expression of YjbO results in decreased motility of bacteria. Because yjbO is co-conserved and co-regulated with the psp operon and is a member of the phage shock protein F regulon, we propose that yjbO be renamed pspG

Discrete physiological effects of beetroot juice and potassium nitrate supplementation following 4-wk sprint interval training

This is the author accepted manuscript. The final version is available from the American Physiological Society via the DOI in this record.The physiological and exercise performance adaptations to sprint interval training (SIT) may be modified by dietary nitrate (NO3) supplementation. However, it is possible that different types of NO3 supplementation evoke divergent physiological and performance adaptations to SIT. The purpose of this study was to compare the effects of 4-wk SIT with and without concurrent dietary NO3 supplementation administered as either NO3-rich beetroot juice (BR) or potassium NO3 (KNO3). Thirty recreationally active subjects completed a battery of exercise tests before and after a 4-wk intervention in which they were allocated to one of three groups: 1) SIT undertaken without dietary NO3 supplementation (SIT); 2) SIT accompanied by concurrent BR supplementation (SIT BR); or 3) SIT accompanied by concurrent KNO3 supplementation (SIT KNO3). During severe-intensity exercise, V O2peak and time to task failure were improved to a greater extent with SIT +BR than SIT and SIT KNO3 (P 0.05). There was also a greater reduction in the accumulation of muscle lactate at 3 min of severe-intensity exercise in SIT BR compared with SIT KNO3 (P <0.05). Plasma NO2 concentration fell to a greater extent during severe-intensity exercise in SIT BR compared with SIT and SIT KNO3 (P <0.05). There were no differences between groups in the reduction in the muscle phosphocreatine recovery time constant from pre- to postintervention (P <0.05). These findings indicate that 4-wk SIT with concurrent BR supplementation results in greater exercise capacity adaptations compared with SIT alone and SIT with concurrent KNO3 supplementation. This may be the result of greater NO-mediated signaling in SIT +BR compared with SIT+ KNO3. NEW & NOTEWORTHY We compared the influence of different forms of dietary nitrate supplementation on the physiological and performance adaptations to sprint interval training (SIT). Compared with SIT alone, supplementation with nitrate-rich beetroot juice, but not potassium NO3, enhanced some physiological adaptations to training