Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Can oral infection be a risk factor for Alzheimer’s disease?

Alzheimer’s disease (AD) is a scourge of longevity that will drain enormous resources from public health budgets in the future. Currently, there is no diagnostic biomarker and/or treatment for this most common form of dementia in humans. AD can be of early familial-onset or sporadic with a late-onset. Apart from the two main hallmarks, amyloid-beta and neurofibrillary tangles, inflammation is a characteristic feature of AD neuropathology. Inflammation may be caused by a local central nervous system insult and/or by peripheral infections. Numerous microorganisms are suspected in AD brains ranging from bacteria (mainly oral and non-oral Treponema species), viruses (Herpes simplex type I) and yeasts (Candida species). A causal relationship between periodontal pathogens/non-oral Treponema species of bacteria has been proposed via the amyloid-beta and inflammatory links. Periodontitis constitutes a peripheral oral infection that can provide the brain with intact bacteria and virulence factors and inflammatory mediators due to daily, transient bacteraemias. If and when genetic risk factors meet environmental risk factors in the brain, disease is expressed, in which neurocognition may be impacted, leading to the development of dementia. To achieve the goal of finding a diagnostic biomarker and possible prophylactic treatment for AD, there is an initial need to solve the etiological puzzle contributing to its pathogenesis. This review therefore addresses oral infection as the plausible aetiology of late onset AD (LOAD)

Roles of TRPV1 and neuropeptidergic receptors in dorsal root reflex-mediated neurogenic inflammation induced by intradermal injection of capsaicin

<p>Abstract</p> <p>Background</p> <p>Acute cutaneous neurogenic inflammation initiated by activation of transient receptor potential vanilloid-1 (TRPV₁) receptors following intradermal injection of capsaicin is mediated mainly by dorsal root reflexes (DRRs). Inflammatory neuropeptides are suggested to be released from primary afferent nociceptors participating in inflammation. However, no direct evidence demonstrates that the release of inflammatory substances is due to the triggering of DRRs and how activation of TRPV₁receptors initiates neurogenic inflammation via triggering DRRs.</p> <p>Results</p> <p>Here we used pharmacological manipulations to analyze the roles of TRPV₁and neuropeptidergic receptors in the DRR-mediated neurogenic inflammation induced by intradermal injection of capsaicin. The degree of cutaneous inflammation in the hindpaw that followed capsaicin injection was assessed by measurements of local blood flow (vasodilation) and paw-thickness (edema) of the foot skin in anesthetized rats. Local injection of capsaicin, calcitonin gene-related peptide (CGRP) or substance P (SP) resulted in cutaneous vasodilation and edema. Removal of DRRs by either spinal dorsal rhizotomy or intrathecal administration of the GABA_Areceptor antagonist, bicuculline, reduced dramatically the capsaicin-induced vasodilation and edema. In contrast, CGRP- or SP-induced inflammation was not significantly affected after DRR removal. Dose-response analysis of the antagonistic effect of the TRPV₁receptor antagonist, capsazepine administered peripherally, shows that the capsaicin-evoked inflammation was inhibited in a dose-dependent manner, and nearly completely abolished by capsazepine at doses between 30–150 μg. In contrast, pretreatment of the periphery with different doses of CGRP_8–37(a CGRP receptor antagonist) or spantide I (a neurokinin 1 receptor antagonist) only reduced the inflammation. If both CGRP and NK₁receptors were blocked by co-administration of CGRP_8–37and spantide I, a stronger reduction in the capsaicin-initiated inflammation was produced.</p> <p>Conclusion</p> <p>Our data suggest that 1) the generation of DRRs is critical for driving the release of neuropeptides antidromically from primary afferent nociceptors; 2) activation of TRPV₁receptors in primary afferent nociceptors following intradermal capsaicin injection initiates this process; 3) the released CGRP and SP participate in neurogenic inflammation.</p

Topoisomer Differentiation of Molecular Knots by FTICR MS: Lessons from Class II Lasso Peptides

Lasso peptides constitute a class of bioactive peptides sharing a knotted structure where the C-terminal tail of the peptide is threaded through and trapped within an N-terminalmacrolactamring. The structural characterization of lasso structures and differentiation from their unthreaded topoisomers is not trivial and generally requires the use of complementary biochemical and spectroscopic methods. Here we investigated two antimicrobial peptides belonging to the class II lasso peptide family and their corresponding unthreaded topoisomers: microcin J25 (MccJ25), which is known to yield two-peptide product ions specific of the lasso structure under collisioninduced dissociation (CID), and capistruin, for which CID does not permit to unambiguously assign the lasso structure. The two pairs of topoisomers were analyzed by electrospray ionization Fourier transform ion cyclotron resonance mass spectrometry (ESI-FTICR MS) upon CID, infrared multiple photon dissociation (IRMPD), and electron capture dissociation (ECD). CID and ECDspectra clearly permitted to differentiate MccJ25 from its non-lasso topoisomer MccJ25-Icm, while for capistruin, only ECD was informative and showed different extent of hydrogen migration (formation of c\bullet/z from c/z\bullet) for the threaded and unthreaded topoisomers. The ECD spectra of the triply-charged MccJ25 and MccJ25-lcm showed a series of radical b-type product ions {\eth}b0In{\TH}. We proposed that these ions are specific of cyclic-branched peptides and result from a dual c/z\bullet and y/b dissociation, in the ring and in the tail, respectively. This work shows the potentiality of ECD for structural characterization of peptide topoisomers, as well as the effect of conformation on hydrogen migration subsequent to electron capture

PageRank without hyperlinks: Reranking with PubMed related article networks for biomedical text retrieval

Graph analysis algorithms such as PageRank and HITS have been successful in Web environments because they are able to extract important inter-document relationships from manually-created hyperlinks. We consider the application of these algorithms to related document networks comprised of automatically-generated content-similarity links. Specifically, this work tackles the problem of document retrieval in the biomedical domain, in the context of the PubMed search engine. A series of reranking experiments demonstrate that incorporating evidence extracted from link structure yields significant improvements in terms of standard ranked retrieval metrics. These results extend the applicability of link analysis algorithms to different environments