Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Surface transfer doped diamond diodes with metal oxide passivation and field-plate

Surface transfer-doping, involving hydrogen terminated diamond surfaces, has been an effective method for producing diamond devices for some years but suffered from poor device longevity and reproducibility. The emergence of metal oxides as an encapsulant has begun to change this situation. Here, HfO2 encapsulated surface transfer doped diamond Schottky diodes with stable device characteristics have been demonstrated. Ideality factor and Schottky barrier heights of the devices did not vary considerably across extended periods of use (up to 39 days). The devices showed excellent blocking capabilities, demonstrating no catastrophic breakdown under the maximum field applied and only a slight increase in leakage current at the reverse bias and field strength of 200 V and 0.167 MV cm−1 , respectively. Indeed, a large rectification ratio of up to 108 and a very low leakage current of ≈10−9 A cm−1 were maintained at this reverse bias (200 V). Furthermore, multiple devices were compared across a single substrate, something rarely reported previously for surface transfer doped diamond diodes. Leakage currents and rectification ratios were similar for all of the devices. The authors are grateful to the UKs Engineering and Physical Sciences Research Council (EPSRC) and BAE Systems Marine Ltd. for the award of a “Cooperative Awards in Science and Engineering (CASE)” Ph.D. Studentship for R.J.W. and to EPSRC for the award of related research funding (No. EP/H020055/1). A.C.P.-T. and R.B.J. also acknowledge invaluable assistance, both financial and in the form of international collaborations, from the European Commission Horizon 2020 Project “GREENDIAMOND” (H2020 Large Project under Grant No. SEP-210184415). Lambda Photometrics Ltd. and Everbeing International Corporation are gratefully acknowledged for use of a Everbeing EB-6 DC probe station. Finally, the LCN Cleanroom is acknowledged for the invaluable assistance of technicians and for the use of the ALD, evaporation, and photo-lithography tools

Pseudorehearsal in value function approximation

Catastrophic forgetting is of special importance in reinforcement learning, as the data distribution is generally non-stationary over time. We study and compare several pseudorehearsal approaches for Q-learning with function approximation in a pole balancing task. We have found that pseudorehearsal seems to assist learning even in such very simple problems, given proper initialization of the rehearsal parameters

UK Chiari 1 Study: protocol for a prospective, observational, multicentre study

INTRODUCTION: Chiari 1 malformation (CM1) is a structural abnormality of the hindbrain characterised by the descent of the cerebellar tonsils through the foramen magnum. The management of patients with CM1 remains contentious since there are currently no UK or international guidelines for clinicians. We therefore propose a collaborative, prospective, multicentre study on the investigation, management and outcome of CM1 in the UK: the UK Chiari 1 Study (UKC1S). Our primary objective is to determine the health-related quality of life (HRQoL) in patients with a new diagnosis of CM1 managed either conservatively or surgically at 12 months of follow-up. We also aim to: (A) determine HRQoL 12 months following surgery; (B) measure complications 12 months following surgery; (C) determine the natural history of patients with CM1 treated conservatively without surgery; (D) determine the radiological correlates of presenting symptoms, signs and outcomes; and (E) determine the scope and variation within UK practice in referral patterns, patient pathways, investigations and surgical decisions. METHODS AND ANALYSIS: The UKC1S will be a prospective, multicentre and observational study that will follow the British Neurosurgical Trainee Research Collaborative model of collaborative research. Patients will be recruited after attending their first neurosurgical outpatient clinic appointment. Follow-up data will be collected from all patients at 12 months from baseline regardless of whether they are treated surgically or not. A further 12-month postoperative follow-up timepoint will be added for patients treated with decompressive surgery. The study is expected to last three years. ETHICS AND DISSEMINATION: The UKC1S received a favourable ethical opinion from the East Midlands Leicester South Research Ethics Committee (REC reference: 20/EM/0053; IRAS 269739) and the Health Research Authority. The results of the study will be published in peer-reviewed medical journals, presented at scientific conferences, shared with collaborating sites and shared with participant patients if they so wish

Adherence and persistence to direct oral anticoagulants in atrial fibrillation: a population-based study

Background Despite simpler regimens than vitamin K antagonists (VKAs) for stroke prevention in atrial fibrillation (AF), adherence (taking drugs as prescribed) and persistence (continuation of drugs) to direct oral anticoagulants are suboptimal, yet understudied in electronic health records (EHRs). Objective We investigated (1) time trends at individual and system levels, and (2) the risk factors for and associations between adherence and persistence. Methods In UK primary care EHR (The Health Information Network 2011–2016), we investigated adherence and persistence at 1 year for oral anticoagulants (OACs) in adults with incident AF. Baseline characteristics were analysed by OAC and adherence/persistence status. Risk factors for non-adherence and non-persistence were assessed using Cox and logistic regression. Patterns of adherence and persistence were analysed. Results Among 36 652 individuals with incident AF, cardiovascular comorbidities (median CHA2DS2VASc[Congestive heart failure, Hypertension, Age≥75 years, Diabetes mellitus, Stroke, Vascular disease, Age 65-74 years, Sex category] 3) and polypharmacy (median number of drugs 6) were common. Adherence was 55.2% (95% CI 54.6 to 55.7), 51.2% (95% CI 50.6 to 51.8), 66.5% (95% CI 63.7 to 69.2), 63.1% (95% CI 61.8 to 64.4) and 64.7% (95% CI 63.2 to 66.1) for all OACs, VKA, dabigatran, rivaroxaban and apixaban. One-year persistence was 65.9% (95% CI 65.4 to 66.5), 63.4% (95% CI 62.8 to 64.0), 61.4% (95% CI 58.3 to 64.2), 72.3% (95% CI 70.9 to 73.7) and 78.7% (95% CI 77.1 to 80.1) for all OACs, VKA, dabigatran, rivaroxaban and apixaban. Risk of non-adherence and non-persistence increased over time at individual and system levels. Increasing comorbidity was associated with reduced risk of non-adherence and non-persistence across all OACs. Overall rates of ‘primary non-adherence’ (stopping after first prescription), ‘non-adherent non-persistence’ and ‘persistent adherence’ were 3.5%, 26.5% and 40.2%, differing across OACs. Conclusions Adherence and persistence to OACs are low at 1 year with heterogeneity across drugs and over time at individual and system levels. Better understanding of contributory factors will inform interventions to improve adherence and persistence across OACs in individuals and populations

Deep forecasting of translational impact in medical research.

The value of biomedical research-a $1.7 trillion annual investment-is ultimately determined by its downstream, real-world impact, whose predictability from simple citation metrics remains unquantified. Here we sought to determine the comparative predictability of future real-world translation-as indexed by inclusion in patents, guidelines, or policy documents-from complex models of title/abstract-level content versus citations and metadata alone. We quantify predictive performance out of sample, ahead of time, across major domains, using the entire corpus of biomedical research captured by Microsoft Academic Graph from 1990-2019, encompassing 43.3 million papers. We show that citations are only moderately predictive of translational impact. In contrast, high-dimensional models of titles, abstracts, and metadata exhibit high fidelity (area under the receiver operating curve [AUROC] > 0.9), generalize across time and domain, and transfer to recognizing papers of Nobel laureates. We argue that content-based impact models are superior to conventional, citation-based measures and sustain a stronger evidence-based claim to the objective measurement of translational potential

What factors indicate prognosis for adults with depression in primary care? A protocol for meta-analyses of individual patient data using the Dep-GP database [version 2; peer review: 2 approved]

BACKGROUND: Pre-treatment severity is a key indicator of prognosis for those with depression. Knowledge is limited on how best to encompass severity of disorders. A number of non-severity related factors such as social support and life events are also indicators of prognosis. It is not clear whether this holds true after adjusting for pre-treatment severity as a) a depressive symptom scale score, and b) a broader construct encompassing symptom severity and related indicators: “disorder severity”. In order to investigate this, data from the individual participants of clinical trials which have measured a breadth of “disorder severity” related factors are needed. AIMS: 1) To assess the association between outcomes for adults seeking treatment for depression and the severity of depression pre-treatment, considered both as i) depressive symptom severity only and ii) “disorder severity” which includes depressive symptom severity and comorbid anxiety, chronicity, history of depression, history of previous treatment, functional impairment and health-related quality of life. 2) To determine whether i) social support, ii) life events, iii) alcohol misuse, and iv) demographic factors (sex, age, ethnicity, marital status, employment status, level of educational attainment, and financial wellbeing) are prognostic indicators of outcomes, independent of baseline “disorder severity” and the type of treatment received. METHODS: Databases were searched for randomised clinical trials (RCTs) that recruited adults seeking treatment for depression from their general practitioners and used the same diagnostic and screening instrument to measure severity at baseline – the Revised Clinical Interview Schedule; outcome measures could differ between studies. Chief investigators of all studies meeting inclusion criteria were contacted and individual patient data (IPD) were requested. CONCLUSIONS: In total 15 RCTs met inclusion criteria. The Dep-GP database will include the 6271 participants from the 13 studies that provided IPD. This protocol outlines how these data will be analysed. REGISTRATION: PROSPERO CRD42019129512 (01/04/2019

Complex exon-intron marking by histone modifications is not determined solely by nucleosome distribution

It has recently been shown that nucleosome distribution, histone modifications and RNA polymerase II (Pol II) occupancy show preferential association with exons (“exon-intron marking”), linking chromatin structure and function to co-transcriptional splicing in a variety of eukaryotes. Previous ChIP-sequencing studies suggested that these marking patterns reflect the nucleosomal landscape. By analyzing ChIP-chip datasets across the human genome in three cell types, we have found that this marking system is far more complex than previously observed. We show here that a range of histone modifications and Pol II are preferentially associated with exons. However, there is noticeable cell-type specificity in the degree of exon marking by histone modifications and, surprisingly, this is also reflected in some histone modifications patterns showing biases towards introns. Exon-intron marking is laid down in the absence of transcription on silent genes, with some marking biases changing or becoming reversed for genes expressed at different levels. Furthermore, the relationship of this marking system with splicing is not simple, with only some histone modifications reflecting exon usage/inclusion, while others mirror patterns of exon exclusion. By examining nucleosomal distributions in all three cell types, we demonstrate that these histone modification patterns cannot solely be accounted for by differences in nucleosome levels between exons and introns. In addition, because of inherent differences between ChIP-chip array and ChIP-sequencing approaches, these platforms report different nucleosome distribution patterns across the human genome. Our findings confound existing views and point to active cellular mechanisms which dynamically regulate histone modification levels and account for exon-intron marking. We believe that these histone modification patterns provide links between chromatin accessibility, Pol II movement and co-transcriptional splicing

A sense of embodiment is reflected in people's signature size

BACKGROUND: The size of a person's signature may reveal implicit information about how the self is perceived although this has not been closely examined. METHODS/RESULTS: We conducted three experiments to test whether increases in signature size can be induced. Specifically, the aim of these experiments was to test whether changes in signature size reflect a person's current implicit sense of embodiment. Experiment 1 showed that an implicit affect task (positive subliminal evaluative conditioning) led to increases in signature size relative to an affectively neutral task, showing that implicit affective cues alter signature size. Experiments 2 and 3 demonstrated increases in signature size following experiential self-focus on sensory and affective stimuli relative to both conceptual self-focus and external (non-self-focus) in both healthy participants and patients with anorexia nervosa, a disorder associated with self-evaluation and a sense of disembodiment. In all three experiments, increases in signature size were unrelated to changes in self-reported mood and larger than manipulation unrelated variations. CONCLUSIONS: Together, these findings suggest that a person's sense of embodiment is reflected in their signature size

Effectiveness of guided self-help in decreasing expressed emotion in family caregivers of people diagnosed with depression in Thailand: a randomised controlled trial

Background: High expressed emotion (EE) can extend the duration of illness and precipitate relapse; however, little evidence-based information is available to assist family caregivers of individuals with depression. In the present exploratory study, we examined the effectiveness of a cognitive behaviour therapy (CBT) based guided self-help (GSH) manual in decreasing EE in caregivers of people with depression, in Thailand. Method: A parallel group randomised controlled trial was conducted, following CONSORT guidelines, with 54 caregivers who were allocated equally to GSH or control group (standard outpatient department support). In addition, both groups were contacted weekly by telephone. EE was assessed, using the Family Questionnaire (FQ), at baseline, post-test (Week 8) and follow-up (Week 12). Results: FQ scores at baseline indicated that both groups had similar, though moderately high level of EE. However, between baseline and post-test EE scores decreased markedly in the intervention group, but in contrast, they increased slightly in the control group. Between post-test and follow-up, little change took place in the EE scores of either group. Overall, the intervention group recipients of GSH showed a significant decrease in EE whereas the control group recipients of standard outpatient department support reported a slight increase in EE. Conclusion: These findings provide preliminary evidence that GSH is beneficial in reducing EE in caregivers, which is advantageous to family members with depression and caregivers. The approach may be used as an adjunct to the limited outpatient department support given to caregivers by mental health professionals and, perhaps, to caregivers who do not attend these departments