Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

UK Chiari 1 Study: protocol for a prospective, observational, multicentre study

INTRODUCTION: Chiari 1 malformation (CM1) is a structural abnormality of the hindbrain characterised by the descent of the cerebellar tonsils through the foramen magnum. The management of patients with CM1 remains contentious since there are currently no UK or international guidelines for clinicians. We therefore propose a collaborative, prospective, multicentre study on the investigation, management and outcome of CM1 in the UK: the UK Chiari 1 Study (UKC1S). Our primary objective is to determine the health-related quality of life (HRQoL) in patients with a new diagnosis of CM1 managed either conservatively or surgically at 12 months of follow-up. We also aim to: (A) determine HRQoL 12 months following surgery; (B) measure complications 12 months following surgery; (C) determine the natural history of patients with CM1 treated conservatively without surgery; (D) determine the radiological correlates of presenting symptoms, signs and outcomes; and (E) determine the scope and variation within UK practice in referral patterns, patient pathways, investigations and surgical decisions. METHODS AND ANALYSIS: The UKC1S will be a prospective, multicentre and observational study that will follow the British Neurosurgical Trainee Research Collaborative model of collaborative research. Patients will be recruited after attending their first neurosurgical outpatient clinic appointment. Follow-up data will be collected from all patients at 12 months from baseline regardless of whether they are treated surgically or not. A further 12-month postoperative follow-up timepoint will be added for patients treated with decompressive surgery. The study is expected to last three years. ETHICS AND DISSEMINATION: The UKC1S received a favourable ethical opinion from the East Midlands Leicester South Research Ethics Committee (REC reference: 20/EM/0053; IRAS 269739) and the Health Research Authority. The results of the study will be published in peer-reviewed medical journals, presented at scientific conferences, shared with collaborating sites and shared with participant patients if they so wish

Pre-launch calibration of the HIRDLS instrument

Report about the aims for the calibration of the HIRDLS instrument

Health Advice from Internet Discussion Forums: How Bad Is Dangerous?

Background: Concerns over online health information–seeking behavior point to the potential harm incorrect, incomplete, or biased information may cause. However, systematic reviews of health information have found few examples of documented harm that can be directly attributed to poor quality information found online. Objective: The aim of this study was to improve our understanding of the quality and quality characteristics of information found in online discussion forum websites so that their likely value as a peer-to-peer health information–sharing platform could be assessed. Methods: A total of 25 health discussion threads were selected across 3 websites (Reddit, Mumsnet, and Patient) covering 3 health conditions (human immunodeficiency virus [HIV], diabetes, and chickenpox). Assessors were asked to rate information found in the discussion threads according to 5 criteria: accuracy, completeness, how sensible the replies were, how they thought the questioner would act, and how useful they thought the questioner would find the replies. Results: In all, 78 fully completed assessments were returned by 17 individuals (8 were qualified medical doctors, 9 were not). When the ratings awarded in the assessments were analyzed, 25 of the assessments placed the discussion threads in the highest possible score band rating them between 5 and 10 overall, 38 rated them between 11 and 15, 12 rated them between 16 and 20, and 3 placed the discussion thread they assessed in the lowest rating band (21-25). This suggests that health threads on Internet discussion forum websites are more likely than not (by a factor of 4:1) to contain information of high or reasonably high quality. Extremely poor information is rare; the lowest available assessment rating was awarded only 11 times out of a possible 353, whereas the highest was awarded 54 times. Only 3 of 78 fully completed assessments rated a discussion thread in the lowest possible overall band of 21 to 25, whereas 25 of 78 rated it in the highest of 5 to 10. Quality assessments differed depending on the health condition (chickenpox appeared 17 times in the 20 lowest-rated threads, HIV twice, and diabetes once). Although assessors tended to agree on which discussion threads contained good quality information, what constituted poor quality information appeared to be more subjective. Conclusions: Most of the information assessed in this study was considered by qualified medical doctors and nonmedically qualified respondents to be of reasonably good quality. Although a small amount of information was assessed as poor, not all respondents agreed that the original questioner would have been led to act inappropriately based on the information presented. This suggests that discussion forum websites may be a useful platform through which people can ask health-related questions and receive answers of acceptable quality

A sense of embodiment is reflected in people's signature size

BACKGROUND: The size of a person's signature may reveal implicit information about how the self is perceived although this has not been closely examined. METHODS/RESULTS: We conducted three experiments to test whether increases in signature size can be induced. Specifically, the aim of these experiments was to test whether changes in signature size reflect a person's current implicit sense of embodiment. Experiment 1 showed that an implicit affect task (positive subliminal evaluative conditioning) led to increases in signature size relative to an affectively neutral task, showing that implicit affective cues alter signature size. Experiments 2 and 3 demonstrated increases in signature size following experiential self-focus on sensory and affective stimuli relative to both conceptual self-focus and external (non-self-focus) in both healthy participants and patients with anorexia nervosa, a disorder associated with self-evaluation and a sense of disembodiment. In all three experiments, increases in signature size were unrelated to changes in self-reported mood and larger than manipulation unrelated variations. CONCLUSIONS: Together, these findings suggest that a person's sense of embodiment is reflected in their signature size

Effectiveness of guided self-help in decreasing expressed emotion in family caregivers of people diagnosed with depression in Thailand: a randomised controlled trial

Background: High expressed emotion (EE) can extend the duration of illness and precipitate relapse; however, little evidence-based information is available to assist family caregivers of individuals with depression. In the present exploratory study, we examined the effectiveness of a cognitive behaviour therapy (CBT) based guided self-help (GSH) manual in decreasing EE in caregivers of people with depression, in Thailand. Method: A parallel group randomised controlled trial was conducted, following CONSORT guidelines, with 54 caregivers who were allocated equally to GSH or control group (standard outpatient department support). In addition, both groups were contacted weekly by telephone. EE was assessed, using the Family Questionnaire (FQ), at baseline, post-test (Week 8) and follow-up (Week 12). Results: FQ scores at baseline indicated that both groups had similar, though moderately high level of EE. However, between baseline and post-test EE scores decreased markedly in the intervention group, but in contrast, they increased slightly in the control group. Between post-test and follow-up, little change took place in the EE scores of either group. Overall, the intervention group recipients of GSH showed a significant decrease in EE whereas the control group recipients of standard outpatient department support reported a slight increase in EE. Conclusion: These findings provide preliminary evidence that GSH is beneficial in reducing EE in caregivers, which is advantageous to family members with depression and caregivers. The approach may be used as an adjunct to the limited outpatient department support given to caregivers by mental health professionals and, perhaps, to caregivers who do not attend these departments

Minimally invasive strabismus surgery (MISS) for inferior obliquus recession

PURPOSE: To present a novel, minimally invasive strabismus surgery (MISS) technique for inferior obliquus recessions. METHODS: Graded MISS inferior obliquus recessions were performed in 20 eyes of 15 patients by applying two small conjunctival cuts, one at the insertion of inferior obliquus and another where the scleral anchoring of the muscle occurred. RESULTS: The amount of recession was 12.2 +/- 2.3 mm (range 6 to 14 mm). The vertical deviation, which was measured in 25 degrees of adduction, decreased from preoperatively 12.8 degrees +/- 5.6 degrees to 2.7 degrees +/- 2.2 degrees (p 0.1). In one eye (2.5%) the two cuts had to be joined because of excessive bleeding. Binocular vision improved in eight patients, remained unchanged in six patients, and decreased from 30 to 60 arcsec in one patient (p > 0.1). Conjunctival and lid swelling were hardly visible on the first postoperative day in primary gaze position in 10/20 (50%) of eyes. Five of the eyes (25%) had mild and five (25%) moderate visibility of surgery. One patient out of 15 (7%) needed repeat surgery because of insufficient reduction of the sursoadduction within the first 6 months. The dose-effect relationship 6 months postoperatively for an accommodative near target at 25 degrees adduction was 0.83 degrees +/- 0.43 degrees per mm of recession. CONCLUSIONS: This study demonstrates that small-incision, minimal dissection inferior obliquus graded recessions are feasible and effective to improve ocular alignment in patients with strabismus sursoadductorius

Extracellular Hsp72 concentration relates to a minimum endogenous criteria during acute exercise-heat exposure

Extracellular heat-shock protein 72 (eHsp72) concentration increases during exercise-heat stress when conditions elicit physiological strain. Differences in severity of environmental and exercise stimuli have elicited varied response to stress. The present study aimed to quantify the extent of increased eHsp72 with increased exogenous heat stress, and determine related endogenous markers of strain in an exercise-heat model. Ten males cycled for 90 min at 50% O2peak in three conditions (TEMP, 20°C/63% RH; HOT, 30.2°C/51%RH; VHOT, 40.0°C/37%RH). Plasma was analysed for eHsp72 pre, immediately post and 24-h post each trial utilising a commercially available ELISA. Increased eHsp72 concentration was observed post VHOT trial (+172.4%) (P<0.05), but not TEMP (-1.9%) or HOT (+25.7%) conditions. eHsp72 returned to baseline values within 24hrs in all conditions. Changes were observed in rectal temperature (Trec), rate of Trec increase, area under the curve for Trec of 38.5°C and 39.0°C, duration Trec ≥ 38.5°C and ≥ 39.0°C, and change in muscle temperature, between VHOT, and TEMP and HOT, but not between TEMP and HOT. Each condition also elicited significantly increasing physiological strain, described by sweat rate, heart rate, physiological strain index, rating of perceived exertion and thermal sensation. Stepwise multiple regression reported rate of Trec increase and change in Trec to be predictors of increased eHsp72 concentration. Data suggests eHsp72 concentration increases once systemic temperature and sympathetic activity exceeds a minimum endogenous criteria elicited during VHOT conditions and is likely to be modulated by large, rapid changes in core temperature

The Role of Repetitive Negative Thoughts in the Vulnerability for Emotional Problems in Non-Clinical Children

The current study examined the role of repetitive negative thoughts in the vulnerability for emotional problems in non-clinical children aged 8–13 years (N = 158). Children completed self-report questionnaires for assessing (1) neuroticism and behavioral inhibition as indicators of general vulnerability (2) worry and rumination which are two important manifestations of repetitive negative thoughts, and (3) emotional problems (i.e., anxiety, depression, and sleep difficulties). Results demonstrated that there were positive correlations between measures of general vulnerability, repetitive negative thoughts, and emotional problems. Further, support was found for a model in which worry and rumination acted as partial mediators in the relation between neuroticism and symptoms of anxiety and depression. In the case of sleep difficulties, no evidence was obtained for such a mediation model. In fact, data suggested that sleeping difficulties are better conceived as an epiphenomenon of high symptom levels of anxiety and depression or as a risk factor for the development of other types of psychopathology. Finally, besides neuroticism, the temperamental trait of behavioral inhibition appeared to play a unique direct role in the model predicting anxiety symptoms but not in the models predicting depressive symptoms or sleep difficulties. To conclude, the current findings seem to indicate that worry and rumination contribute to children’s vulnerability for anxiety and depression