Influencing Exploration in Actor-Critic Reinforcement Learning Algorithms

Reinforcement Learning (RL) is a subset of machine learning primarily concerned with goal-directed learning and optimal decision making. RL agents learn based on a reward signal discovered from trial and error in complex, uncertain environments with the goal of maximizing positive reward signals. RL approaches need to scale up as they are applied to more complex environments with extremely large state spaces. Inefficient exploration methods cannot sufficiently explore complex environments in a reasonable amount of time, and optimal policies will be unrealized resulting in RL agents failing to solve an environment. This thesis proposes a novel variant of the Actor-Advantage Critic (A2C) algorithm. The variant is validated against two state-of-the-art RL algorithms, Deep Q-Network (DQN) and A2C, across six Atari 2600 games of varying difficulty. The experimental results are competitive with state-of-the-art and achieve lower variance and quicker learning speed. Additionally, the thesis introduces a metric to objectively quantify the difficulty of any Markovian environment with respect to the exploratory capacity of RL agents

Assessing direct contributions of morphological awareness and prosodic sensitivity to children’s word reading and reading comprehension

We examined the independent contributions of prosodic sensitivity and morphological awareness to word reading, text reading accuracy, and reading comprehension. We did so in a longitudinal study of English-speaking children (N = 70). At 5 to 7 years of age, children completed the metalinguistic measures along with control measures of phonological awareness and vocabulary. Children completed the reading measures two years later. Morphological awareness, but not prosodic sensitivity made a significant independent contribution to word reading, text reading accuracy and reading comprehension. The effects of morphological awareness on reading comprehension remained after controls for word reading. These results suggest that morphological awareness needs to be considered seriously in models of reading development and that prosodic sensitivity might have primarily indirect relations to reading outcomes. Keywords: Morphological Awareness; Prosody; Word Reading; Reading Comprehension

Risk factors for genital infections in people initiating SGLT2 inhibitors and their impact on discontinuation

Introduction: To identify risk factors, absolute risk, and impact on treatment discontinuation of genital infections with sodium-glucose co-transporter-2 inhibitors (SGLT2i). Research design and methods: We assessed the relationship between baseline characteristics and genital infection in 21 004 people with type 2 diabetes initiating SGLT2i and 55 471 controls initiating dipeptidyl peptidase-4 inhibitors (DPP4i) in a UK primary care database. We assessed absolute risk of infection in those with key risk factors and the association between early genital infection and treatment discontinuation. Results: Genital infection was substantially more common in those treated with SGLT2i (8.1% within 1 year) than DPP4i (1.8%). Key predictors of infection with SGLT2i were female sex (HR 3.64; 95% CI 3.23 to 4.11) and history of genital infection; <1 year before initiation (HR 4.38; 3.73 to 5.13), 1–5 years (HR 3.04; 2.64 to 3.51), and >5 years (HR 1.79; 1.55 to 2.07). Baseline HbA1c was not associated with infection risk for SGLT2i, in contrast to DPP4i where risk increased with higher HbA1c. One-year absolute risk of genital infection with SGLT2i was highest for those with a history of prior infection (females 23.7%, males 12.1%), compared with those without (females 10.8%, males 2.7%). Early genital infection was associated with a similar discontinuation risk for SGLT2i (HR 1.48; 1.21–1.80) and DPP4i (HR 1.58; 1.21–2.07). Conclusions: Female sex and history of prior infection are simple features that can identify subgroups at greatly increased risk of genital infections with SGLT2i therapy. These data can be used to risk-stratify patients. High HbA1c is not a risk factor for genital infections with SGLT2i

Transcriptional and immunohistological assessment of immune infiltration in pancreatic cancer.

Pancreatic adenocarcinoma is characterized by a complex tumor environment with a wide diversity of infiltrating stromal and immune cell types that impact the tumor response to conventional treatments. However, even in this poorly responsive tumor the extent of T cell infiltration as determined by quantitative immunohistology is a candidate prognostic factor for patient outcome. As such, even more comprehensive immunophenotyping of the tumor environment, such as immune cell type deconvolution via inference models based on gene expression profiling, holds significant promise. We hypothesized that RNA-Seq can provide a comprehensive alternative to quantitative immunohistology for immunophenotyping pancreatic cancer. We performed RNA-Seq on a prospective cohort of pancreatic tumor specimens and compared multiple approaches for gene expression-based immunophenotyping analysis compared to quantitative immunohistology. Our analyses demonstrated that while gene expression analyses provide additional information on the complexity of the tumor immune environment, they are limited in sensitivity by the low overall immune infiltrate in pancreatic cancer. As an alternative approach, we identified a set of genes that were enriched in highly T cell infiltrated pancreatic tumors, and demonstrate that these can identify patients with improved outcome in a reference population. These data demonstrate that the poor immune infiltrate in pancreatic cancer can present problems for analyses that use gene expression-based tools; however, there remains enormous potential in using these approaches to understand the relationships between diverse patterns of infiltrating cells and their impact on patient treatment outcomes

The impact of antibiotics on growth in children in low and middle income countries: systematic review and meta-analysis of randomised controlled trials

Objectives To determine whether antibiotic treatment leads to improvements in growth in prepubertal children in low and middle income countries, to determine the magnitude of improvements in growth, and to identify moderators of this treatment effect.Design Systematic review and meta-analysis.Data sources Medline, Embase, Scopus, the Cochrane central register of controlled trials, and Web of Science.Study selection Randomised controlled trials conducted in low or middle income countries in which an orally administered antibacterial agent was allocated by randomisation or minimisation and growth was measured as an outcome. Participants aged 1 month to 12 years were included. Control was placebo or non-antimicrobial intervention.Results Data were pooled from 10 randomised controlled trials representing 4316 children, across a variety of antibiotics, indications for treatment, treatment regimens, and countries. in random effects models, antibiotic use increased height by 0.04 cm/month (95% confidence interval 0.00 to 0.07) and weight by 23.8 g/month (95% confidence interval 4.3 to 43.3). After adjusting for age, effects on height were larger in younger populations and effects on weight were larger in African studies compared with other regions.Conclusion Antibiotics have a growth promoting effect in prepubertal children in low and middle income countries. This effect was more pronounced for ponderal than for linear growth. the antibiotic growth promoting effect may be mediated by treatment of clinical or subclinical infections or possibly by modulation of the intestinal microbiota. Better definition of the mechanisms underlying this effect will be important to inform optimal and safe approaches to achieving healthy growth in vulnerable populations.Vanier Canada Graduate ScholarshipMcGill Univ, Dept Epidemiol Biostat & Occupat Hlth, Montreal, PQ, CanadaZvitambo Inst Maternal Child Hlth Res, Harare, ZimbabweQueen Mary Univ London, Blizard Inst, Ctr Paediat, London, EnglandMRC, Clin Trials Unit, London, EnglandJohns Hopkins Bloomberg Sch Publ Hlth, Dept Int Hlth, Baltimore, MD USACornell Univ, Div Nutr Sci, Program Int Nutr, Ithaca, NY 14853 USAWashington Univ, Sch Med, Dept Pediat, St Louis, MO 63110 USAUniv Malawi, Blantyre, MalawiUniv Cambridge, Dept Archaeol & Anthropol, Div Biol Anthropol, Cambridge CB2 1TN, EnglandUniversidade Federal de São Paulo, Escola Paulista Med, São Paulo, BrazilUniv British Columbia, Sch Populat & Publ Hlth, Vancouver, BC V6T 1Z3, CanadaUniversidade Federal de São Paulo, Escola Paulista Med, São Paulo, BrazilWeb of Scienc

The labels and models used to describe problematic substance use impact discrete elements of stigma: A Registered Report

Objectives: Problematic substance use is one of the most stigmatised health conditions leading research to examine how the labels and models used to describe it influence public stigma. Two recent studies examine whether beliefs in a disease model of addiction influence public stigma but result in equivocal findings – in line with the mixed-blessings model, Kelly et al. (2021) found that whilst the label ‘chronically relapsing brain disease’ reduced blame attribution, it decreased prognostic optimism and increased perceived danger and need for continued care; however, Rundle et al. (2021) conclude absence of evidence. This study isolates the different factors used in these two studies to assess whether health condition (drug use vs. health concern), aetiological label (brain disease vs. problem), and attributional judgement (low vs. high treatment stability) influence public stigma towards problematic substance use. Methods: 1613 participants were assigned randomly to one of eight vignette conditions that manipulated these factors. They completed self-report measures of discrete and general public stigma and an indirect measure of discrimination. Results: Greater social distance, danger, and public stigma but lower blame were ascribed to drug use relative to a health concern. Greater (genetic) blame was reported when drug use was labelled as a ‘chronically relapsing brain disease’ relative to a ‘problem’. Findings for attributional judgement were either inconclusive or statistically equivalent. Discussion: The labels used to describe problematic substance use appear to impact discrete elements of stigma. We suggest that addiction is a functional attribution, which may explain the mixed literature on the impact of aetiological labels on stigma to date

Expression of arginase I in myeloid cells limits control of residual disease after radiation therapy of tumors in mice

An accumulating body of evidence demonstrates that radiation therapy can generate adaptive immune responses that contribute to tumor control. However, in the absence of additional immune therapy, the adaptive immune response is insufficient to prevent tumor recurrence or affect distant disease. It has been shown in multiple models that tumor-infiltrating myeloid cells exhibit alternative activation phenotypes and are able to suppress adaptive immune responses, and recent data suggests that the myeloid response in tumors treated with cytotoxic therapy limits tumor control. We hypothesized that tumor myeloid cells inhibit the adaptive immune response after radiation therapy through expression of the enzyme arginase I. Using a myeloid cell-specific deletion of arginase I in mice, we demonstrate an improved tumor control after radiation therapy. However, tumors still recurred despite the conditional knockdown of arginase I. Since multiple alternative factors may combine to inhibit adaptive immunity, we propose that targeting macrophage differentiation may be a more effective strategy than targeting individual suppressive pathways

Pharmacogenomics of GLP-1 Receptor Agonists:a genome-wide analysis of observational data and large randomised controlled trials

Background: In the treatment of type 2 diabetes, GLP-1 receptor agonists lower blood glucose concentrations, body weight, and have cardiovascular benefits. The efficacy and side effects of GLP-1 receptor agonists vary between people. Human pharmacogenomic studies of this inter-individual variation can provide both biological insight into drug action and provide biomarkers to inform clinical decision making. We therefore aimed to identify genetic variants associated with glycaemic response to GLP-1 receptor agonist treatment. Methods: In this genome-wide analysis we included adults (aged >= 18 years) with type 2 diabetes treated with GLP-1 receptor agonists with baseline HbA1c of 7% or more (53 mmol/mol) from four prospective observational cohorts (DIRECT, PRIBA, PROMASTER, and GoDARTS) and two randomised clinical trials (HARMONY phase 3 and AWARD). The primary endpoint was HbA1c reduction at 6 months after starting GLP-1 receptor agonists. We evaluated variants in GLP1R, then did a genome-wide association study and gene-based burden tests. Findings: 4571 adults were included in our analysis, of these, 3339 (73%) were White European, 449 (10%) Hispanic, 312 (7%) American Indian or Alaskan Native, and 471 (10%) were other, and around 2140 (47%) of the participants were women. Variation in HbA1c reduction with GLP-1 receptor agonists treatment was associated with rs6923761G -> A (Gly168Ser) in the GLP1R (0.08% [95% CI 0.04-0.12] or 0.9 mmol/mol lower reduction in HbA1c per serine, p=6.0 x 10(-5)) and low frequency variants in ARRB1 (optimal sequence kernel association test p=6.7 x 10(-8)), largely driven by rs140226575G -> A (Thr370Met; 0.25% [SE 0.06] or 2.7 mmol/mol [SE 0.7] greater HbA1c reduction per methionine, p=5.2 x 10(-6)). A similar effect size for the ARRB1 Thr370Met was seen in Hispanic and American Indian or Alaska Native populations who have a higher frequency of this variant (6-11%) than in White European populations. Combining these two genes identified 4% of the population who had a 30% greater reduction in HbA1c than the 9% of the population with the worse response. Interpretation: This genome-wide pharmacogenomic study of GLP-1 receptor agonists provides novel biological and clinical insights. Clinically, when genotype is routinely available at the point of prescribing, individuals with ARRB1 variants might benefit from earlier initiation of GLP-1 receptor agonists

Gravimetric and density profiling using the combination of surface acoustic waves and neutron reflectivity

A new approach is described herein, where neutron reflectivity measurements that probe changes in the density profile of thin films as they absorb material from the gas phase have been combined with a Love wave based gravimetric assay that measures the mass of absorbed material. This combination of techniques not only determines the spatial distribution of absorbed molecules, but also reveals the amount of void space within the thin film (a quantity that can be difficult to assess using neutron reflectivity measurements alone). The uptake of organic solvent vapours into spun cast films of polystyrene has been used as a model system with a view to this method having the potential for extension to the study of other systems. These could include, for example, humidity sensors, hydrogel swelling, biomolecule adsorption or transformations of electroactive and chemically reactive thin films. This is the first ever demonstration of combined neutron reflectivity and Love wave-based gravimetry and the experimental caveats, limitations and scope of the method are explored and discussed in detail