Deep Ordinal Reinforcement Learning

Reinforcement learning usually makes use of numerical rewards, which have nice properties but also come with drawbacks and difficulties. Using rewards on an ordinal scale (ordinal rewards) is an alternative to numerical rewards that has received more attention in recent years. In this paper, a general approach to adapting reinforcement learning problems to the use of ordinal rewards is presented and motivated. We show how to convert common reinforcement learning algorithms to an ordinal variation by the example of Q-learning and introduce Ordinal Deep Q-Networks, which adapt deep reinforcement learning to ordinal rewards. Additionally, we run evaluations on problems provided by the OpenAI Gym framework, showing that our ordinal variants exhibit a performance that is comparable to the numerical variations for a number of problems. We also give first evidence that our ordinal variant is able to produce better results for problems with less engineered and simpler-to-design reward signals.Comment: replaced figures for better visibility, added github repository, more details about source of experimental results, updated target value calculation for standard and ordinal Deep Q-Networ

National, regional, and global estimates of anaemia by severity in women and children for 2000-19: a pooled analysis of population-representative data

BACKGROUND: Anaemia causes health and economic harms. The prevalence of anaemia in women aged 15-49 years, by pregnancy status, is indicator 2.2.3 of the UN Sustainable Development Goals, and the aim of halving the anaemia prevalence in women of reproductive age by 2030 is an extension of the 2025 global nutrition targets endorsed by the World Health Assembly (WHA). We aimed to estimate the prevalence of anaemia by severity for children aged 6-59 months, non-pregnant women aged 15-49 years, and pregnant women aged 15-49 years in 197 countries and territories and globally for the period 2000-19. METHODS: For this pooled analysis of population-representative data, we collated 489 data sources on haemoglobin distribution in children and women from 133 countries, including 4·5 million haemoglobin measurements. Our data sources comprised health examination, nutrition, and household surveys, accessed as anonymised individual records or as summary statistics such as mean haemoglobin and anaemia prevalence. We used a Bayesian hierarchical mixture model to estimate haemoglobin distributions in each population and country-year. This model allowed for coherent estimation of mean haemoglobin and prevalence of anaemia by severity. FINDINGS: Globally, in 2019, 40% (95% uncertainty interval [UI] 36-44) of children aged 6-59 months were anaemic, compared to 48% (45-51) in 2000. Globally, the prevalence of anaemia in non-pregnant women aged 15-49 years changed little between 2000 and 2019, from 31% (95% UI 28-34) to 30% (27-33), while in pregnant women aged 15-49 years it decreased from 41% (39-43) to 36% (34-39). In 2019, the prevalence of anaemia in children aged 6-59 months exceeded 70% in 11 countries and exceeded 50% in all women aged 15-49 years in ten countries. Globally in all populations and in most countries and regions, the prevalence of mild anaemia changed little, while moderate and severe anaemia declined in most populations and geographical locations, indicating a shift towards mild anaemia. INTERPRETATION: Globally, regionally, and in nearly all countries, progress on anaemia in women aged 15-49 years is insufficient to meet the WHA global nutrition target to halve anaemia prevalence by 2030, and the prevalence of anaemia in children also remains high. A better understanding of the context-specific causes of anaemia and quality implementation of effective multisectoral actions to address these causes are needed. FUNDING: USAID, US Centers for Disease Control and Prevention, and Bill & Melinda Gates Foundation

On the unification of dwarf and giant elliptical galaxies

The near orthogonal distributions of dwarf elliptical (dE) and giant elliptical (E) galaxies in the mu_e-Mag and mu_e-log(R_e) diagrams have been interpreted as evidence for two distinct galaxy formation processes. However, continuous, linear relationships across the alleged dE/E boundary at M_B = -18 mag - such as those between central surface brightness (mu_0) and (i) galaxy magnitude and (ii) light-profile shape (n) - suggest a similar, governing formation mechanism. Here we explain how these latter two linear trends necessitate a different behavior for dE and E galaxies, exactly as observed, in diagrams involving mu_e (and also _e). A natural consequence is that the distribution of dEs and Es in Fundamental Plane type analyses that use the associated intensity I_e, or _e, are expected to appear different. Together with other linear trends across the alleged dE/E boundary, such as those between luminosity and color, metallicity, and velocity dispersion, it appears that the dEs form a continuous extension to the E galaxies. The presence of partially depleted cores in luminous (M_B < -20.5 mag) Es does however signify the action of a different physical process at the centers (< ~300 pc) of these galaxies.Comment: 5 pages from the proceedings of the 2004 conference "Penetrating bars through masks of cosmic dust: the Hubble tuning fork strikes a new note". Edited by D. L. Block, I. Puerari, K. C. Freeman, R. Groess, and E. K. Bloc

Intra- and inter-individual genetic differences in gene expression

Genetic variation is known to influence the amount of mRNA produced by a gene. Given that the molecular machines control mRNA levels of multiple genes, we expect genetic variation in the components of these machines would influence multiple genes in a similar fashion. In this study we show that this assumption is correct by using correlation of mRNA levels measured independently in the brain, kidney or liver of multiple, genetically typed, mice strains to detect shared genetic influences. These correlating groups of genes (CGG) have collective properties that account for 40-90% of the variability of their constituent genes and in some cases, but not all, contain genes encoding functionally related proteins. Critically, we show that the genetic influences are essentially tissue specific and consequently the same genetic variations in the one animal may up-regulate a CGG in one tissue but down-regulate the same CGG in a second tissue. We further show similarly paradoxical behaviour of CGGs within the same tissues of different individuals. The implication of this study is that this class of genetic variation can result in complex inter- and intra-individual and tissue differences and that this will create substantial challenges to the investigation of phenotypic outcomes, particularly in humans where multiple tissues are not readily available.&#xd;&#xa;&#xd;&#xa

The host metabolite D-serine contributes to bacterial niche specificity through gene selection

Escherichia coli comprise a diverse array of both commensals and niche-specific pathotypes. The ability to cause disease results from both carriage of specific virulence factors and regulatory control of these via environmental stimuli. Moreover, host metabolites further refine the response of bacteria to their environment and can dramatically affect the outcome of the host–pathogen interaction. Here, we demonstrate that the host metabolite, D-serine, selectively affects gene expression in E. coli O157:H7. Transcriptomic profiling showed exposure to D-serine results in activation of the SOS response and suppresses expression of the Type 3 Secretion System (T3SS) used to attach to host cells. We also show that concurrent carriage of both the D-serine tolerance locus (dsdCXA) and the locus of enterocyte effacement pathogenicity island encoding a T3SS is extremely rare, a genotype that we attribute to an ‘evolutionary incompatibility’ between the two loci. This study demonstrates the importance of co-operation between both core and pathogenic genetic elements in defining niche specificity

Differential Regulation of Horizontally Acquired and Core Genome Genes by the Bacterial Modulator H-NS

Horizontal acquisition of DNA by bacteria dramatically increases genetic diversity and hence successful bacterial colonization of several niches, including the human host. A relevant issue is how this newly acquired DNA interacts and integrates in the regulatory networks of the bacterial cell. The global modulator H-NS targets both core genome and HGT genes and silences gene expression in response to external stimuli such as osmolarity and temperature. Here we provide evidence that H-NS discriminates and differentially modulates core and HGT DNA. As an example of this, plasmid R27-encoded H-NS protein has evolved to selectively silence HGT genes and does not interfere with core genome regulation. In turn, differential regulation of both gene lineages by resident chromosomal H-NS requires a helper protein: the Hha protein. Tight silencing of HGT DNA is accomplished by H-NS-Hha complexes. In contrast, core genes are modulated by H-NS homoligomers. Remarkably, the presence of Hha-like proteins is restricted to the Enterobacteriaceae. In addition, conjugative plasmids encoding H-NS variants have hitherto been isolated only from members of the family. Thus, the H-NS system in enteric bacteria presents unique evolutionary features. The capacity to selectively discriminate between core and HGT DNA may help to maintain horizontally transmitted DNA in silent form and may give these bacteria a competitive advantage in adapting to new environments, including host colonization

Diagnosing collaboration in practice-based learning: Equality and intra-individual variability of physical interactivity

Collaborative problem solving (CPS), as a teaching and learning approach, is considered to have the potential to improve some of the most important skills to prepare students for their future. CPS often differs in its nature, practice, and learning outcomes from other kinds of peer learning approaches, including peer tutoring and cooperation; and it is important to establish what identifies collaboration in problem-solving situations. The identification of indicators of collaboration is a challenging task. However, students physical interactivity can hold clues of such indicators. In this paper, we investigate two non-verbal indexes of student physical interactivity to interpret collaboration in practice-based learning environments: equality and intra-individual variability. Our data was generated from twelve groups of three Engineering students working on open-ended tasks using a learning analytics system. The results show that high collaboration groups have member students who present high and equal amounts of physical interactivity and low and equal amounts of intra-individual variability

Application of Autologous Bone Marrow Derived Mesenchymal Stem Cells to an Ovine Model of Growth Plate Cartilage Injury

Injury to growth plate cartilage in children can lead to bone bridge formation and result in bone growth deformities, a significant clinical problem currently lacking biological treatment. Mesenchymal stem/stromal cells (MSC) offer a promising therapeutic option for regeneration of damaged cartilage, due to their self renewing and multi-lineage differentiation attributes. Although some small animal model studies highlight the therapeutic potential of MSC for growth plate repair, translational research in large animal models, which more closely resemble the human condition, are lacking. Our laboratory has recently characterised MSCs derived from ovine bone marrow, and demonstrated these cells form cartilage-like tissue when transplanted within the gelatin sponge, Gelfoam, in vivo. In the current study, autologous bone marrow MSC were seeded into Gelfoam scaffold containing TGF-β1, and transplanted into a surgically created defect of the proximal ovine tibial growth plate. Examination of implants at 5 week post-operatively revealed transplanted autologous MSC failed to form new cartilage structure at the defect site, but contributed to an increase in formation of a dense fibrous tissue. Importantly, the extent of osteogenesis was diminished, and bone bridge formation was not accelerated due to transplantation of MSCs or the gelatin scaffold. The current study represents the first work that has utilised this ovine large animal model to investigate whether autologous bone marrow derived MSC can be used to initiate regeneration at the injured growth plate

Genetic and expression studies of SMN2 gene in Russian patients with spinal muscular atrophy type II and III

<p>Abstract</p> <p>Background</p> <p>Spinal muscular atrophy (SMA type I, II and III) is an autosomal recessive neuromuscular disorder caused by mutations in the survival motor neuron gene (<it>SMN1</it>). <it>SMN2 </it>is a centromeric copy gene that has been characterized as a major modifier of SMA severity. SMA type I patients have one or two <it>SMN2 </it>copies while most SMA type II patients carry three <it>SMN2 </it>copies and SMA III patients have three or four <it>SMN2 </it>copies. The <it>SMN1 </it>gene produces a full-length transcript (FL-SMN) while <it>SMN2 </it>is only able to produce a small portion of the FL-SMN because of a splice mutation which results in the production of abnormal SMNΔ7 mRNA.</p> <p>Methods</p> <p>In this study we performed quantification of the <it>SMN2 </it>gene copy number in Russian patients affected by SMA type II and III (42 and 19 patients, respectively) by means of real-time PCR. Moreover, we present two families consisting of asymptomatic carriers of a homozygous absence of the <it>SMN1 </it>gene. We also developed a novel RT-qPCR-based assay to determine the FL-SMN/SMNΔ7 mRNA ratio as SMA biomarker.</p> <p>Results</p> <p>Comparison of the <it>SMN2 </it>copy number and clinical features revealed a significant correlation between mild clinical phenotype (SMA type III) and presence of four copies of the <it>SMN2 </it>gene. In both asymptomatic cases we found an increased number of <it>SMN2 </it>copies in the healthy carriers and a biallelic <it>SMN1 </it>absence. Furthermore, the novel assay revealed a difference between SMA patients and healthy controls.</p> <p>Conclusions</p> <p>We suggest that the <it>SMN2 </it>gene copy quantification in SMA patients could be used as a prognostic tool for discrimination between the SMA type II and SMA type III diagnoses, whereas the FL-SMN/SMNΔ7 mRNA ratio could be a useful biomarker for detecting changes during SMA pharmacotherapy.</p