Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

The luminosity function of field galaxies

Schmidt's method for construction of luminosity function of galaxies is generalized by taking into account the dependence of density of galaxies from the distance in the near Universe. The logarithmical luminosity function (LLF) of field galaxies depending on morphological type is constructed. We show that the LLF for all galaxies, and also separately for elliptical and lenticular galaxies can be presented by Schechter function in narrow area of absolute magnitudes. The LLF of spiral galaxies was presented by Schechter function for enough wide area of absolute magnitudes: . Spiral galaxies differ slightly by parameter . At transition from early spirals to the late spirals parameter in Schechter function is reduced. The reduction of mean luminosity of galaxies is observed at transition from elliptical galaxies to lenticular galaxies, to early spiral galaxies, and further, to late spiral galaxies, in a bright end, . The completeness and the average density of samples of galaxies of different morphological types are estimated. In the range the mean number density of all galaxies is equal 0.127 Mpc-3.Comment: 14 page, 8 figures, to appear in Astrophysic

Can oral infection be a risk factor for Alzheimer’s disease?

Alzheimer’s disease (AD) is a scourge of longevity that will drain enormous resources from public health budgets in the future. Currently, there is no diagnostic biomarker and/or treatment for this most common form of dementia in humans. AD can be of early familial-onset or sporadic with a late-onset. Apart from the two main hallmarks, amyloid-beta and neurofibrillary tangles, inflammation is a characteristic feature of AD neuropathology. Inflammation may be caused by a local central nervous system insult and/or by peripheral infections. Numerous microorganisms are suspected in AD brains ranging from bacteria (mainly oral and non-oral Treponema species), viruses (Herpes simplex type I) and yeasts (Candida species). A causal relationship between periodontal pathogens/non-oral Treponema species of bacteria has been proposed via the amyloid-beta and inflammatory links. Periodontitis constitutes a peripheral oral infection that can provide the brain with intact bacteria and virulence factors and inflammatory mediators due to daily, transient bacteraemias. If and when genetic risk factors meet environmental risk factors in the brain, disease is expressed, in which neurocognition may be impacted, leading to the development of dementia. To achieve the goal of finding a diagnostic biomarker and possible prophylactic treatment for AD, there is an initial need to solve the etiological puzzle contributing to its pathogenesis. This review therefore addresses oral infection as the plausible aetiology of late onset AD (LOAD)

Search algorithms as a framework for the optimization of drug combinations

Combination therapies are often needed for effective clinical outcomes in the management of complex diseases, but presently they are generally based on empirical clinical experience. Here we suggest a novel application of search algorithms, originally developed for digital communication, modified to optimize combinations of therapeutic interventions. In biological experiments measuring the restoration of the decline with age in heart function and exercise capacity in Drosophila melanogaster, we found that search algorithms correctly identified optimal combinations of four drugs with only one third of the tests performed in a fully factorial search. In experiments identifying combinations of three doses of up to six drugs for selective killing of human cancer cells, search algorithms resulted in a highly significant enrichment of selective combinations compared with random searches. In simulations using a network model of cell death, we found that the search algorithms identified the optimal combinations of 6-9 interventions in 80-90% of tests, compared with 15-30% for an equivalent random search. These findings suggest that modified search algorithms from information theory have the potential to enhance the discovery of novel therapeutic drug combinations. This report also helps to frame a biomedical problem that will benefit from an interdisciplinary effort and suggests a general strategy for its solution.Comment: 36 pages, 10 figures, revised versio

Topoisomer Differentiation of Molecular Knots by FTICR MS: Lessons from Class II Lasso Peptides

Lasso peptides constitute a class of bioactive peptides sharing a knotted structure where the C-terminal tail of the peptide is threaded through and trapped within an N-terminalmacrolactamring. The structural characterization of lasso structures and differentiation from their unthreaded topoisomers is not trivial and generally requires the use of complementary biochemical and spectroscopic methods. Here we investigated two antimicrobial peptides belonging to the class II lasso peptide family and their corresponding unthreaded topoisomers: microcin J25 (MccJ25), which is known to yield two-peptide product ions specific of the lasso structure under collisioninduced dissociation (CID), and capistruin, for which CID does not permit to unambiguously assign the lasso structure. The two pairs of topoisomers were analyzed by electrospray ionization Fourier transform ion cyclotron resonance mass spectrometry (ESI-FTICR MS) upon CID, infrared multiple photon dissociation (IRMPD), and electron capture dissociation (ECD). CID and ECDspectra clearly permitted to differentiate MccJ25 from its non-lasso topoisomer MccJ25-Icm, while for capistruin, only ECD was informative and showed different extent of hydrogen migration (formation of c\bullet/z from c/z\bullet) for the threaded and unthreaded topoisomers. The ECD spectra of the triply-charged MccJ25 and MccJ25-lcm showed a series of radical b-type product ions {\eth}b0In{\TH}. We proposed that these ions are specific of cyclic-branched peptides and result from a dual c/z\bullet and y/b dissociation, in the ring and in the tail, respectively. This work shows the potentiality of ECD for structural characterization of peptide topoisomers, as well as the effect of conformation on hydrogen migration subsequent to electron capture

Structure of hadron resonances with a nearby zero of the amplitude

We discuss the relation between the analytic structure of the scattering amplitude and the origin of an eigenstate represented by a pole of the amplitude.If the eigenstate is not dynamically generated by the interaction in the channel of interest, the residue of the pole vanishes in the zero coupling limit. Based on the topological nature of the phase of the scattering amplitude, we show that the pole must encounter with the Castillejo-Dalitz-Dyson (CDD) zero in this limit. It is concluded that the dynamical component of the eigenstate is small if a CDD zero exists near the eigenstate pole. We show that the line shape of the resonance is distorted from the Breit-Wigner form as an observable consequence of the nearby CDD zero. Finally, studying the positions of poles and CDD zeros of the KbarN-piSigma amplitude, we discuss the origin of the eigenstates in the Lambda(1405) region.Comment: 7 pages, 3 figures, v2: published versio

Projected WIMP sensitivity of the LUX-ZEPLIN dark matter experiment

LUX-ZEPLIN (LZ) is a next-generation dark matter direct detection experiment that will operate 4850 feet underground at the Sanford Underground Research Facility (SURF) in Lead, South Dakota, USA. Using a two-phase xenon detector with an active mass of 7 tonnes, LZ will search primarily for low-energy interactions with weakly interacting massive particles (WIMPs), which are hypothesized to make up the dark matter in our galactic halo. In this paper, the projected WIMP sensitivity of LZ is presented based on the latest background estimates and simulations of the detector. For a 1000 live day run using a 5.6-tonne fiducial mass, LZ is projected to exclude at 90% confidence level spin-independent WIMP-nucleon cross sections above 1.4×10-48 cm2 for a 40 GeV/c2 mass WIMP. Additionally, a 5σ discovery potential is projected, reaching cross sections below the exclusion limits of recent experiments. For spin-dependent WIMP-neutron(-proton) scattering, a sensitivity of 2.3×10-43 cm2 (7.1×10-42 cm2) for a 40 GeV/c2 mass WIMP is expected. With underground installation well underway, LZ is on track for commissioning at SURF in 2020

Planet Populations as a Function of Stellar Properties

Exoplanets around different types of stars provide a window into the diverse environments in which planets form. This chapter describes the observed relations between exoplanet populations and stellar properties and how they connect to planet formation in protoplanetary disks. Giant planets occur more frequently around more metal-rich and more massive stars. These findings support the core accretion theory of planet formation, in which the cores of giant planets form more rapidly in more metal-rich and more massive protoplanetary disks. Smaller planets, those with sizes roughly between Earth and Neptune, exhibit different scaling relations with stellar properties. These planets are found around stars with a wide range of metallicities and occur more frequently around lower mass stars. This indicates that planet formation takes place in a wide range of environments, yet it is not clear why planets form more efficiently around low mass stars. Going forward, exoplanet surveys targeting M dwarfs will characterize the exoplanet population around the lowest mass stars. In combination with ongoing stellar characterization, this will help us understand the formation of planets in a large range of environments.Comment: Accepted for Publication in the Handbook of Exoplanet

The function of the alula in avian flight

The alula is a small structure located at the joint between the hand-wing and arm-wing of birds and is known to be used in slow flight with high angles of attack such as landing. It is assumed to function similarly to a leading-edge slat that increases lift and delays stall. However, in spite of its universal presence in flying birds and the wide acceptance of stall delay as its main function, how the alula delays the stall and aids the flight of birds remains unclear. Here, we investigated the function of alula on the aerodynamic performance of avian wings based on data from flight tasks and wind-tunnel experiments. With the alula, the birds performed steeper descending flights with greater changes in body orientation. Force measurements revealed that the alula increases the lift and often delays the stall. Digital particle image velocimetry showed that these effects are caused by the streamwise vortex, formed at the tip of the alula, that induces strong downwash and suppresses the flow separation over the wing surface. This is the first experimental evidence that the alula functions as a vortex generator that increases the lift force and enhances manoeuvrability in flights at high angles of attack.open1

A review of physical supply and EROI of fossil fuels in China

This paper reviews China’s future fossil fuel supply from the perspectives of physical output and net energy output. Comprehensive analyses of physical output of fossil fuels suggest that China’s total oil production will likely reach its peak, at about 230 Mt/year (or 9.6 EJ/year), in 2018; its total gas production will peak at around 350 Bcm/year (or 13.6 EJ/year) in 2040, while coal production will peak at about 4400 Mt/year (or 91.9 EJ/year) around 2020 or so. In terms of the forecast production of these fuels, there are significant differences among current studies. These differences can be mainly explained by different ultimately recoverable resources assumptions, the nature of the models used, and differences in the historical production data. Due to the future constraints on fossil fuels production, a large gap is projected to grow between domestic supply and demand, which will need to be met by increasing imports. Net energy analyses show that both coal and oil and gas production show a steady declining trend of EROI (energy return on investment) due to the depletion of shallow-buried coal resources and conventional oil and gas resources, which is generally consistent with the approaching peaks of physical production of fossil fuels. The peaks of fossil fuels production, coupled with the decline in EROI ratios, are likely to challenge the sustainable development of Chinese society unless new abundant energy resources with high EROI values can be found