Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

The use of metabonomics to uncover differences between the small molecule profiles of eggs from cage and barn housing systems

Metabonomic techniques have been used to observe differences in the small molecule profiles of chicken eggs, to work towards the detection, and thus prevention, of fraud regarding the misrepresentation of egg farming systems. High Performance Liquid Chromatography-Quadrupole-Time-of-Flight-Mass Spectrometry (HPLC-Q-ToF-MS) was used to obtain profiles of the small molecules found in the yolks of eggs that were laid by hens in enriched cage systems, and in barn systems. Statistical analysis of these small molecule profiles, including the use of XCMS Online and multivariate statistics, was able to uncover differences between the yolks of cage and barn eggs. Several unidentified compounds were found to be present in significantly different abundances between cage and barn egg yolks and one of these compounds was tentatively identified, through the use of METLIN, as 1,2-dipalmitoyl-glycero-3-phosphocholine

The effects of climatic fluctuations and extreme events on running water ecosystems

Most research on the effects of environmental change in freshwaters has focused on incremental changes in average conditions, rather than fluctuations or extreme events such as heatwaves, cold snaps, droughts, floods or wildfires, which may have even more profound consequences. Such events are commonly predicted to increase in frequency, intensity and duration with global climate change, with many systems being exposed to conditions with no recent historical precedent. We propose a mechanistic framework for predicting potential impacts of environmental fluctuations on running water ecosystems by scaling up effects of fluctuations from individuals to entire ecosystems. This framework requires integration of four key components: effects of the environment on individual metabolism, metabolic and biomechanical constraints on fluctuating species interactions, assembly dynamics of local food webs and mapping the dynamics of the meta-community onto ecosystem function. We illustrate the framework by developing a mathematical model of environmental fluctuations on dynamically assembling food webs. We highlight (currently limited) empirical evidence for emerging insights and theoretical predictions. For example, widely supported predictions about the effects of environmental fluctuations are: high vulnerability of species with high per capita metabolic demands such as large-bodied ones at the top of food webs; simplification of food web network structure and impaired energetic transfer efficiency; reduced resilience and top-down relative to bottom-up regulation of food web and ecosystem processes. We conclude by identifying key questions and challenges that need to be addressed to develop more accurate and predictive bio-assessments of the effects of fluctuations, and implications of fluctuations for management practices in an increasingly uncertain world

The experiential impact of hospitalisation:parents' accounts of caring for young people with early psychosis

This research examines the experiential impact of hospitalisation on the parents of young people with early psychosis. In-depth interviews were conducted with a small sample of parents, and the resulting transcripts were analysed using Interpretative Phenomenological Analysis. Five themes emerged from the data: Accepting and blaming, Feeling out of control, Hospitalisation as temporary containment, Feeling let down by services and Stigma. Aspects of the hospitalisation process were characterised by parents as generally negative, but a number of positive affirmations were also offered regarding the containing, supportive and crucial role of services. Parents' perceptions of hospitalisation as a difficult, and sometimes distressing, experience are exacerbated by the complexity of being the carer of a young person. Negotiating services and boundaries within the context of this relationship contributes to feelings of exclusion and disregard by professionals and services. The implications of this study resonate with the current government mental health strategy with regard to how services can engage and include carers in the mental health system, and equip and enable them to support their relatives with early psychosis

Structuring of Bacterioplankton Diversity in a Large Tropical Bay

Structuring of bacterioplanktonic populations and factors that determine the structuring of specific niche partitions have been demonstrated only for a limited number of colder water environments. In order to better understand the physical chemical and biological parameters that may influence bacterioplankton diversity and abundance, we examined their productivity, abundance and diversity in the second largest Brazilian tropical bay (Guanabara Bay, GB), as well as seawater physical chemical and biological parameters of GB. The inner bay location with higher nutrient input favored higher microbial (including vibrio) growth. Metagenomic analysis revealed a predominance of Gammaproteobacteria in this location, while GB locations with lower nutrient concentration favored Alphaproteobacteria and Flavobacteria. According to the subsystems (SEED) functional analysis, GB has a distinctive metabolic signature, comprising a higher number of sequences in the metabolism of phosphorus and aromatic compounds and a lower number of sequences in the photosynthesis subsystem. The apparent phosphorus limitation appears to influence the GB metagenomic signature of the three locations. Phosphorus is also one of the main factors determining changes in the abundance of planktonic vibrios, suggesting that nutrient limitation can be observed at community (metagenomic) and population levels (total prokaryote and vibrio counts)

Molecular evidence of Rickettsia felis infection in dogs from northern territory, Australia

The prevalence of spotted fever group rickettsial infection in dogs from a remote indigenous community in the Northern Territory (NT) was determined using molecular tools. Blood samples collected from 130 dogs in the community of Maningrida were subjected to a spotted fever group (SFG)-specific PCR targeting the ompB gene followed by a Rickettsia felis-specific PCR targeting the gltA gene of R. felis. Rickettsia felis ompB and gltA genes were amplified from the blood of 3 dogs. This study is the first report of R. felis infection in indigenous community dogs in NT

W-Curve Alignments for HIV-1 Genomic Comparisons

The W-curve was originally developed as a graphical visualization technique for viewing DNA and RNA sequences. Its ability to render features of DNA also makes it suitable for computational studies. Its main advantage in this area is utilizing a single-pass algorithm for comparing the sequences. Avoiding recursion during sequence alignments offers advantages for speed and in-process resources. The graphical technique also allows for multiple models of comparison to be used depending on the nucleotide patterns embedded in similar whole genomic sequences. The W-curve approach allows us to compare large numbers of samples quickly.We are currently tuning the algorithm to accommodate quirks specific to HIV-1 genomic sequences so that it can be used to aid in diagnostic and vaccine efforts. Tracking the molecular evolution of the virus has been greatly hampered by gap associated problems predominantly embedded within the envelope gene of the virus. Gaps and hypermutation of the virus slow conventional string based alignments of the whole genome. This paper describes the W-curve algorithm itself, and how we have adapted it for comparison of similar HIV-1 genomes. A treebuilding method is developed with the W-curve that utilizes a novel Cylindrical Coordinate distance method and gap analysis method. HIV-1 C2-V5 env sequence regions from a Mother/Infant cohort study are used in the comparison.The output distance matrix and neighbor results produced by the W-curve are functionally equivalent to those from Clustal for C2-V5 sequences in the mother/infant pairs infected with CRF01_AE.Significant potential exists for utilizing this method in place of conventional string based alignment of HIV-1 genomes, such as Clustal X. With W-curve heuristic alignment, it may be possible to obtain clinically useful results in a short time-short enough to affect clinical choices for acute treatment. A description of the W-curve generation process, including a comparison technique of aligning extremes of the curves to effectively phase-shift them past the HIV-1 gap problem, is presented. Besides yielding similar neighbor-joining phenogram topologies, most Mother and Infant C2-V5 sequences in the cohort pairs geometrically map closest to each other, indicating that W-curve heuristics overcame any gap problem

TRY plant trait database - enhanced coverage and open access

Plant traits-the morphological, anatomical, physiological, biochemical and phenological characteristics of plants-determine how plants respond to environmental factors, affect other trophic levels, and influence ecosystem properties and their benefits and detriments to people. Plant trait data thus represent the basis for a vast area of research spanning from evolutionary biology, community and functional ecology, to biodiversity conservation, ecosystem and landscape management, restoration, biogeography and earth system modelling. Since its foundation in 2007, the TRY database of plant traits has grown continuously. It now provides unprecedented data coverage under an open access data policy and is the main plant trait database used by the research community worldwide. Increasingly, the TRY database also supports new frontiers of trait-based plant research, including the identification of data gaps and the subsequent mobilization or measurement of new data. To support this development, in this article we evaluate the extent of the trait data compiled in TRY and analyse emerging patterns of data coverage and representativeness. Best species coverage is achieved for categorical traits-almost complete coverage for 'plant growth form'. However, most traits relevant for ecology and vegetation modelling are characterized by continuous intraspecific variation and trait-environmental relationships. These traits have to be measured on individual plants in their respective environment. Despite unprecedented data coverage, we observe a humbling lack of completeness and representativeness of these continuous traits in many aspects. We, therefore, conclude that reducing data gaps and biases in the TRY database remains a key challenge and requires a coordinated approach to data mobilization and trait measurements. This can only be achieved in collaboration with other initiatives