Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Marriage, religion and human flourishing: how sustainable is the classic Durkheim thesis in contemporary Europe?

This paper draws on the three waves of the European Values Survey across five countries (Great Britain, Italy, the Netherlands, Northern Ireland, Spain and Sweden) to investigate the relationship between indicators of positive psychology (conceptualised as feelings of happiness and satisfaction with life), religiosity (conceptualised as self-assigned religious affiliation and self-reported religious attendance) and marital status. The results demonstrate that religiosity is, in general, positively correlated with both indicators of positive psychology. Further, across all waves and all countries, the pattern emerges that those respondents who are married are likely to report higher levels of happiness and greater satisfaction in life. These data provide contemporary support for the classic Durkheim thesis linking the two institutions of marriage and religion with human flourishing

Examining links between anxiety, reinvestment and walking when talking by older adults during adaptive gait

Falls by older adults often result in reduced quality of life and debilitating fear of further falls. Stopping walking when talking (SWWT) is a significant predictor of future falls by older adults and is thought to reflect age-related increases in attentional demands of walking. We examine whether SWWT is associated with use of explicit movement cues during locomotion, and evaluate if conscious control (i.e., movement specific reinvestment) is causally linked to falls-related anxiety during a complex walking task. We observed whether twenty-four older adults stopped walking when talking when asked a question during an adaptive gait task. After certain trials, participants completed a visual-spatial recall task regarding walkway features, or answered questions about their movements during the walk. In a subsequent experimental condition, participants completed the walking task under conditions of raised postural threat. Compared to a control group, participants who SWWT reported higher scores for aspects of reinvestment relating to conscious motor processing but not movement self-consciousness. The higher scores for conscious motor processing were preserved when scores representing cognitive function were included as a covariate. There were no group differences in measures of general cognitive function, visual spatial working memory or balance confidence. However, the SWWT group reported higher scores on a test of external awareness when walking, indicating allocation of attention away from task-relevant environmental features. Under conditions of increased threat, participants self-reported significantly greater state anxiety and reinvestment and displayed more accurate responses about their movements during the task. SWWT is not associated solely with age-related cognitive decline or generic increases in age-related attentional demands of walking. SWWT may be caused by competition for phonological resources of working memory associated with consciously processing motor actions and appears to be causally linked with fall-related anxiety and increased vigilance.This research was supported by The Royal Society (IE131576) and British Academy (SG132820)

Foot health education for people with rheumatoid arthritis : the practitioner's perspective

Background: Patient education is considered to be a key role for podiatrists in the management of patients with rheumatoid arthritis (RA). Patient education has undoubtedly led to improved clinical outcomes, however no attempts have been made to optimise its content or delivery to maximise benefits within the context of the foot affected by rheumatoid arthritis. The aim of this study was to identify the nature and content of podiatrists' foot health education for people with RA. Any potential barriers to its provision were also explored. Methods: A focus group was conducted. The audio dialogue was recorded digitally, transcribed verbatim and analysed using a structured, thematic approach. The full transcription was verified by the focus group as an accurate account of what was said. The thematic analysis framework was verified by members of the research team to ensure validity of the data. Results: Twelve members (all female) of the north west Podiatry Clinical Effectiveness Group for Rheumatology participated. Six overarching themes emerged: (i) the essence of patient education; (ii) the content; (iii) patient-centred approach to content and timing; (iv) barriers to provision; (v) the therapeutic relationship; and (vi) tools of the trade. Conclusion: The study identified aspects of patient education that this group of podiatrists consider most important in relation to its: content, timing, delivery and barriers to its provision. General disease and foot health information in relation to RA together with a potential prognosis for foot health, the role of the podiatrist in management of foot health, and appropriate self-management strategies were considered to be key aspects of content, delivered according to the needs of the individual. Barriers to foot health education provision, including financial constraints and difficulties in establishing effective therapeutic relationships, were viewed as factors that strongly influenced foot health education provision. These data will contribute to the development of a patient-centred, negotiated approach to the provision of foot health education for people with RA

Surface Chemistry of Perfluoropolyethers and Hydrogenated Analogs: Are Studies of Model Compounds Useful?

We have studied adsorption, desorption, and decomposition of ethers on Ru(001), an atomically-smooth metal surface. We have compared diethers with monoethers, and fluorinated ethers with hydrogenated ethers. The number of ether linkages does not strongly influence adsorption bond strength, nor the extent of decomposition. Fluorination does weaken the adsorption bond strength and prevents decomposition. These studies suggest that the surface properties of monomeric ethers can be used to predict properties of oligomeric, and perhaps even polymeric, ethers

Bilateral ischemic optic neuropathy after transurethral prostatic resection: a case report

BACKGROUND: Nonarteritic ischemic optic neuropathy affects the anterior portion of the optic nerve and is characterized by sudden, painless visual loss. The affected eye has a relative afferent pupillary defect. The typical funduscopic appearance includes optic disc edema, with associated nerve fiber layer hemorrhage. Risk factors include advanced age, systemic hypertension, nocturnal hypotension, diabetes mellitus, and a small cup-to-disc ratio. Bilateral presentation is rare. Postoperative optic neuropathy has been associated with nonocular surgery; risk factors include a combination of prolonged surgical times, acute systemic hypotension, anemia due to blood loss, or prone positioning. We report for the first time a patient with bilateral, simultaneous anterior ischemic optic neuropathy after elective transurethral prostatic resection. CASE PRESENTATION: A 66-year old man underwent surgery for benign prostatic hyperplasia. The preoperative blood pressure was 140/85 mmHg, hemoglobin 15.9 g/dL, and hematocrit 48.6%. Two hours postoperatively, the blood pressure, hemoglobin, and hematocrit dropped dramatically. One day later, transient horizontal diplopia developed. Funduscopy showed a congenitally small cup-to-disc ratio without papillary edema. Other ocular findings were unremarkable. By 4 days postoperatively, sudden and painless amaurosis bilaterally developed when the patient awoke with nausea and vomiting. Visual acuity was no light perception bilaterally. The optic discs were swollen with small hemorrhages. Scans of the head and orbits and electrolyte levels were normal. There were no responses on visual evoked potentials bilaterally. The blood pressure was 90/50 mm Hg, the hemoglobin 7.0 g/dL, and the hematocrit 22.9%, necessitating infusion of three units of packed red blood cells. The blood pressure, hematocrit, and hemoglobin increased to normal levels. Three months later the visual acuity remained no light perception. The pupils were unreactive and there was marked optic disc atrophy bilaterally. CONCLUSION: Bilateral and simultaneous acute ischemic optic neuropathy may be a rare but devastating surgical complication. The combination of anemia and hypotension may increase the risk of anterior ischemic optic neuropathy postoperatively after transurethral prostatic resection

The direction of postural threat alters balance control when standing at virtual elevation

This is the author accepted manuscript. The final version is available from Springer via the DOI in this recordAvailability of Data and Material: All data are available here: https://github.com/keithlohse/Gait_VR/tree/master/standing_balanceCode availability: All analysis code are available here: https://github.com/keithlohse/Gait_VR/tree/master/standing_balanceAnxiogenic settings lead to reduced postural sway while standing, but anxiety-related balance may be influenced by the location of postural threat in the environment. We predicted that the direction of threat would elicit a parallel controlled manifold relative to the standing surface, and an orthogonal uncontrolled manifold during standing. Altogether, 14 healthy participants (8 women, mean age = 27.5 yrs, SD = 8.2) wore a virtual reality (VR) headset and stood on a matched real-world walkway (2m x 40cm x 2cm) for 30s at ground level and simulated heights (elevated 15m) in two positions: (1) parallel to walkway, lateral threat; and (2) perpendicular to walkway, anteroposterior threat. Inertial sensors measured postural sway acceleration (e.g., 95% ellipse, root mean square (RMS) of acceleration), and a wrist-worn monitor measured heart rate coefficient of variation (HR CV). Fully factorial linear-mixed effect regressions (LMER) determined the effects of height and position. HR CV moderately increased from low to high height (p = 0.050, g = 0.397). The Height x Position interaction approached significance for sway area (95% ellipse; ß = -0.018, p = 0.062) and was significant for RMS (ß = -0.022, p = 0.007). Post-hoc analyses revealed that frontal plane sway accelerations and RMS increased from low to high elevation in parallel standing, but decreased when facing the threat during perpendicular standing. Postural response to threat varies depending on the direction of threat, suggesting that the control strategies used during standing are sensitive to the direction of threat.National Institutes of Health (NIH

Zebrafish Reproduction: Revisiting In Vitro Fertilization to Increase Sperm Cryopreservation Success

Although conventional cryopreservation is a proven method for long-term, safe storage of genetic material, protocols used by the zebrafish community are not standardized and yield inconsistent results, thereby putting the security of many genotypes in individual laboratories and stock centers at risk. An important challenge for a successful zebrafish sperm cryopreservation program is the large variability in the post-thaw in vitro fertilization success (0 to 80%). But how much of this variability was due to the reproductive traits of the in vitro fertilization process, and not due to the cryopreservation process? These experiments only assessed the in vitro process with fresh sperm, but yielded the basic metrics needed for successful in vitro fertilization using cryopreserved sperm, as well. We analyzed the reproductive traits for zebrafish males with a strict body condition range. It did not correlate with sperm volume, or motility (P>0.05), but it did correlate with sperm concentration. Younger males produced more concentrated sperm (P<0.05). To minimize the wastage of sperm during the in vitro fertilization process, 106 cells/ml was the minimum sperm concentration needed to achieve an in vitro fertilization success of ≥ 70%. During the in vitro process, pooling sperm did not reduce fertilization success (P>0.05), but pooling eggs reduced it by approximately 30 to 50% (P<0.05). This reduction in fertilization success was due not to the pooling of the females' eggs, but to the type of tools used to handle the eggs. Recommendations to enhance the in vitro process for zebrafish include: 1) using males of a body condition closer to 1.5 for maximal sperm concentration; 2) minimizing sperm wastage by using a working sperm concentration of 106 motile cells/ml for in vitro fertilization; and 3) never using metal or sharp-edged tools to handle eggs prior to fertilization

The feasibility of using virtual reality to induce mobility-related anxiety during turning

This is the author accepted manuscript. The final version is available from Elsevier via the DOI in this recordThe fear of falling, or mobility-related anxiety, profoundly affects gait, but is challenging to study without risk to participants. Purpose: To determine the efficacy of using virtual reality (VR) to manipulate illusions of height and consequently, elevated mobility-related anxiety when turning. Moreover, we examined if mobility-related anxiety effects decline across time in VR environments as participants habituate. Methods: Altogether, 10 healthy participants (five women, mean (standard deviation) age = 28.5 (8.5) years) turned at self-selected and fast speeds on a 2.2 m walkway under two simulated environments: (1) ground elevation; and (2) high elevation (15 m above ground). Peak turning velocity was recorded using inertial sensors and participants rated their cognitive (i.e., worry) and somatic (i.e., tension) anxiety, confidence, and mental effort. Results: A significant Height × Speed × Trial interaction (p =  0.013) was detected for peak turning velocity. On average, the virtual height illusion decreased peak turning velocity, especially at fast speeds. At low elevation, participants decreased speed across trials, but not significantly (p =  0.381), but at high elevation, they significantly increased speed across trials (p =  0.001). At self-selected speeds, no effects were revealed (all p >  0.188) and only effects for Height were observed for fast speeds (p <  0.001). After turning at high elevation, participants reported greater cognitive (p =  0.008) and somatic anxiety (p =  0.007), reduced confidence (p = 0.021), and greater mental effort (p <  0.001) compared to the low elevation. Conclusion: VR can safely induce mobility-related anxiety during dynamic motor tasks, and habituation effects from repeated exposure should be carefully considered in experimental designs and analysis