Multiagent Deep Reinforcement Learning: Challenges and Directions Towards Human-Like Approaches

This paper surveys the field of multiagent deep reinforcement learning. The combination of deep neural networks with reinforcement learning has gained increased traction in recent years and is slowly shifting the focus from single-agent to multiagent environments. Dealing with multiple agents is inherently more complex as (a) the future rewards depend on the joint actions of multiple players and (b) the computational complexity of functions increases. We present the most common multiagent problem representations and their main challenges, and identify five research areas that address one or more of these challenges: centralised training and decentralised execution, opponent modelling, communication, efficient coordination, and reward shaping. We find that many computational studies rely on unrealistic assumptions or are not generalisable to other settings; they struggle to overcome the curse of dimensionality or nonstationarity. Approaches from psychology and sociology capture promising relevant behaviours such as communication and coordination. We suggest that, for multiagent reinforcement learning to be successful, future research addresses these challenges with an interdisciplinary approach to open up new possibilities for more human-oriented solutions in multiagent reinforcement learning.Comment: 37 pages, 6 figure

Microbicidal Effects of α- and θ-Defensins Against Antibiotic-Resistant \u3cem\u3eStaphylococcus aureus\u3c/em\u3e and \u3cem\u3ePseudomonas aeruginosa\u3c/em\u3e

Antibiotic-resistant bacterial pathogens threaten public health. Because many antibiotics target specific bacterial enzymes or reactions, corresponding genes may mutate under selection and lead to antibiotic resistance. Accordingly, antimicrobials that selectively target overall microbial cell integrity may offer alternative approaches to therapeutic design. Naturally occurring mammalian α- and θ-defensins are potent, non-toxic microbicides that may be useful for treating infections by antibiotic-resistant pathogens because certain defensin peptides disrupt bacterial, but not mammalian, cell membranes. To test this concept, clinical isolates of methicillin-resistant Staphylococcus aureus (MRSA), including vancomycin heteroresistant strains, and ciprofloxacin-resistant Pseudomonas aeruginosa (CipR-PA) were tested for sensitivity to α-defensins Crp-4, RMAD-4 and HNPs 1-3, and to RTD-1, macaque θ-defensin-1. In vitro, 3 μM Crp-4, RMAD-4 and RTD-1 reduced MRSA cell survival by 99%, regardless of vancomycin susceptibility. For PA clinical isolates that differ in fluoroquinolone resistance and virulence phenotype, peptide efficacy was independent of strain ciprofloxacin resistance, site of isolation or virulence factor expression. Thus, Crp-4, RMAD-4 and RTD-1 are effective in vitro antimicrobials against clinical isolates of MRSA and CipR-PA, perhaps providing templates for development of α- and θ-defensin-based microbicides against antibiotic resistant or virulent infectious agents

The state of the Martian climate

60°N was +2.0°C, relative to the 1981–2010 average value (Fig. 5.1). This marks a new high for the record. The average annual surface air temperature (SAT) anomaly for 2016 for land stations north of starting in 1900, and is a significant increase over the previous highest value of +1.2°C, which was observed in 2007, 2011, and 2015. Average global annual temperatures also showed record values in 2015 and 2016. Currently, the Arctic is warming at more than twice the rate of lower latitudes

Argo data 1999-2019: two million temperature-salinity profiles and subsurface velocity observations from a global array of profiling floats.

© The Author(s), 2020. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Wong, A. P. S., Wijffels, S. E., Riser, S. C., Pouliquen, S., Hosoda, S., Roemmich, D., Gilson, J., Johnson, G. C., Martini, K., Murphy, D. J., Scanderbeg, M., Bhaskar, T. V. S. U., Buck, J. J. H., Merceur, F., Carval, T., Maze, G., Cabanes, C., Andre, X., Poffa, N., Yashayaev, I., Barker, P. M., Guinehut, S., Belbeoch, M., Ignaszewski, M., Baringer, M. O., Schmid, C., Lyman, J. M., McTaggart, K. E., Purkey, S. G., Zilberman, N., Alkire, M. B., Swift, D., Owens, W. B., Jayne, S. R., Hersh, C., Robbins, P., West-Mack, D., Bahr, F., Yoshida, S., Sutton, P. J. H., Cancouet, R., Coatanoan, C., Dobbler, D., Juan, A. G., Gourrion, J., Kolodziejczyk, N., Bernard, V., Bourles, B., Claustre, H., D'Ortenzio, F., Le Reste, S., Le Traon, P., Rannou, J., Saout-Grit, C., Speich, S., Thierry, V., Verbrugge, N., Angel-Benavides, I. M., Klein, B., Notarstefano, G., Poulain, P., Velez-Belchi, P., Suga, T., Ando, K., Iwasaska, N., Kobayashi, T., Masuda, S., Oka, E., Sato, K., Nakamura, T., Sato, K., Takatsuki, Y., Yoshida, T., Cowley, R., Lovell, J. L., Oke, P. R., van Wijk, E. M., Carse, F., Donnelly, M., Gould, W. J., Gowers, K., King, B. A., Loch, S. G., Mowat, M., Turton, J., Rama Rao, E. P., Ravichandran, M., Freeland, H. J., Gaboury, I., Gilbert, D., Greenan, B. J. W., Ouellet, M., Ross, T., Tran, A., Dong, M., Liu, Z., Xu, J., Kang, K., Jo, H., Kim, S., & Park, H. Argo data 1999-2019: two million temperature-salinity profiles and subsurface velocity observations from a global array of profiling floats. Frontiers in Marine Science, 7, (2020): 700, doi:10.3389/fmars.2020.00700.In the past two decades, the Argo Program has collected, processed, and distributed over two million vertical profiles of temperature and salinity from the upper two kilometers of the global ocean. A similar number of subsurface velocity observations near 1,000 dbar have also been collected. This paper recounts the history of the global Argo Program, from its aspiration arising out of the World Ocean Circulation Experiment, to the development and implementation of its instrumentation and telecommunication systems, and the various technical problems encountered. We describe the Argo data system and its quality control procedures, and the gradual changes in the vertical resolution and spatial coverage of Argo data from 1999 to 2019. The accuracies of the float data have been assessed by comparison with high-quality shipboard measurements, and are concluded to be 0.002°C for temperature, 2.4 dbar for pressure, and 0.01 PSS-78 for salinity, after delayed-mode adjustments. Finally, the challenges faced by the vision of an expanding Argo Program beyond 2020 are discussed.AW, SR, and other scientists at the University of Washington (UW) were supported by the US Argo Program through the NOAA Grant NA15OAR4320063 to the Joint Institute for the Study of the Atmosphere and Ocean (JISAO) at the UW. SW and other scientists at the Woods Hole Oceanographic Institution (WHOI) were supported by the US Argo Program through the NOAA Grant NA19OAR4320074 (CINAR/WHOI Argo). The Scripps Institution of Oceanography's role in Argo was supported by the US Argo Program through the NOAA Grant NA15OAR4320071 (CIMEC). Euro-Argo scientists were supported by the Monitoring the Oceans and Climate Change with Argo (MOCCA) project, under the Grant Agreement EASME/EMFF/2015/1.2.1.1/SI2.709624 for the European Commission

Basic science232. Certolizumab pegol prevents pro-inflammatory alterations in endothelial cell function

Background: Cardiovascular disease is a major comorbidity of rheumatoid arthritis (RA) and a leading cause of death. Chronic systemic inflammation involving tumour necrosis factor alpha (TNF) could contribute to endothelial activation and atherogenesis. A number of anti-TNF therapies are in current use for the treatment of RA, including certolizumab pegol (CZP), (Cimzia ®; UCB, Belgium). Anti-TNF therapy has been associated with reduced clinical cardiovascular disease risk and ameliorated vascular function in RA patients. However, the specific effects of TNF inhibitors on endothelial cell function are largely unknown. Our aim was to investigate the mechanisms underpinning CZP effects on TNF-activated human endothelial cells. Methods: Human aortic endothelial cells (HAoECs) were cultured in vitro and exposed to a) TNF alone, b) TNF plus CZP, or c) neither agent. Microarray analysis was used to examine the transcriptional profile of cells treated for 6 hrs and quantitative polymerase chain reaction (qPCR) analysed gene expression at 1, 3, 6 and 24 hrs. NF-κB localization and IκB degradation were investigated using immunocytochemistry, high content analysis and western blotting. Flow cytometry was conducted to detect microparticle release from HAoECs. Results: Transcriptional profiling revealed that while TNF alone had strong effects on endothelial gene expression, TNF and CZP in combination produced a global gene expression pattern similar to untreated control. The two most highly up-regulated genes in response to TNF treatment were adhesion molecules E-selectin and VCAM-1 (q 0.2 compared to control; p > 0.05 compared to TNF alone). The NF-κB pathway was confirmed as a downstream target of TNF-induced HAoEC activation, via nuclear translocation of NF-κB and degradation of IκB, effects which were abolished by treatment with CZP. In addition, flow cytometry detected an increased production of endothelial microparticles in TNF-activated HAoECs, which was prevented by treatment with CZP. Conclusions: We have found at a cellular level that a clinically available TNF inhibitor, CZP reduces the expression of adhesion molecule expression, and prevents TNF-induced activation of the NF-κB pathway. Furthermore, CZP prevents the production of microparticles by activated endothelial cells. This could be central to the prevention of inflammatory environments underlying these conditions and measurement of microparticles has potential as a novel prognostic marker for future cardiovascular events in this patient group. Disclosure statement: Y.A. received a research grant from UCB. I.B. received a research grant from UCB. S.H. received a research grant from UCB. All other authors have declared no conflicts of interes

New genetic loci link adipose and insulin biology to body fat distribution.

Body fat distribution is a heritable trait and a well-established predictor of adverse metabolic outcomes, independent of overall adiposity. To increase our understanding of the genetic basis of body fat distribution and its molecular links to cardiometabolic traits, here we conduct genome-wide association meta-analyses of traits related to waist and hip circumferences in up to 224,459 individuals. We identify 49 loci (33 new) associated with waist-to-hip ratio adjusted for body mass index (BMI), and an additional 19 loci newly associated with related waist and hip circumference measures (P < 5 × 10(-8)). In total, 20 of the 49 waist-to-hip ratio adjusted for BMI loci show significant sexual dimorphism, 19 of which display a stronger effect in women. The identified loci were enriched for genes expressed in adipose tissue and for putative regulatory elements in adipocytes. Pathway analyses implicated adipogenesis, angiogenesis, transcriptional regulation and insulin resistance as processes affecting fat distribution, providing insight into potential pathophysiological mechanisms

Using high-resolution contact networks to evaluate SARS-CoV-2 transmission and control in large-scale multi-day events

The emergence of highly transmissible SARS-CoV-2 variants has created a need to reassess the risk posed by increasing social contacts as countries resume pre-pandemic activities, particularly in the context of resuming large-scale events over multiple days. To examine how social contacts formed in different activity settings influences interventions required to control Delta variant outbreaks, we collected high-resolution data on contacts among passengers and crew on cruise ships and combined the data with network transmission models. We found passengers had a median of 20 (IQR 10–36) unique close contacts per day, and over 60% of their contact episodes were made in dining or sports areas where mask wearing is typically limited. In simulated outbreaks, we found that vaccination coverage and rapid antigen tests had a larger effect than mask mandates alone, indicating the importance of combined interventions against Delta to reduce event risk in the vaccine era

Impact of opioid-free analgesia on pain severity and patient satisfaction after discharge from surgery: multispecialty, prospective cohort study in 25 countries

Background: Balancing opioid stewardship and the need for adequate analgesia following discharge after surgery is challenging. This study aimed to compare the outcomes for patients discharged with opioid versus opioid-free analgesia after common surgical procedures.Methods: This international, multicentre, prospective cohort study collected data from patients undergoing common acute and elective general surgical, urological, gynaecological, and orthopaedic procedures. The primary outcomes were patient-reported time in severe pain measured on a numerical analogue scale from 0 to 100% and patient-reported satisfaction with pain relief during the first week following discharge. Data were collected by in-hospital chart review and patient telephone interview 1 week after discharge.Results: The study recruited 4273 patients from 144 centres in 25 countries; 1311 patients (30.7%) were prescribed opioid analgesia at discharge. Patients reported being in severe pain for 10 (i.q.r. 1-30)% of the first week after discharge and rated satisfaction with analgesia as 90 (i.q.r. 80-100) of 100. After adjustment for confounders, opioid analgesia on discharge was independently associated with increased pain severity (risk ratio 1.52, 95% c.i. 1.31 to 1.76; P &lt; 0.001) and re-presentation to healthcare providers owing to side-effects of medication (OR 2.38, 95% c.i. 1.36 to 4.17; P = 0.004), but not with satisfaction with analgesia (beta coefficient 0.92, 95% c.i. -1.52 to 3.36; P = 0.468) compared with opioid-free analgesia. Although opioid prescribing varied greatly between high-income and low- and middle-income countries, patient-reported outcomes did not.Conclusion: Opioid analgesia prescription on surgical discharge is associated with a higher risk of re-presentation owing to side-effects of medication and increased patient-reported pain, but not with changes in patient-reported satisfaction. Opioid-free discharge analgesia should be adopted routinely

Finishing the euchromatic sequence of the human genome

The sequence of the human genome encodes the genetic instructions for human physiology, as well as rich information about human evolution. In 2001, the International Human Genome Sequencing Consortium reported a draft sequence of the euchromatic portion of the human genome. Since then, the international collaboration has worked to convert this draft into a genome sequence with high accuracy and nearly complete coverage. Here, we report the result of this finishing process. The current genome sequence (Build 35) contains 2.85 billion nucleotides interrupted by only 341 gaps. It covers ∼99% of the euchromatic genome and is accurate to an error rate of ∼1 event per 100,000 bases. Many of the remaining euchromatic gaps are associated with segmental duplications and will require focused work with new methods. The near-complete sequence, the first for a vertebrate, greatly improves the precision of biological analyses of the human genome including studies of gene number, birth and death. Notably, the human enome seems to encode only 20,000-25,000 protein-coding genes. The genome sequence reported here should serve as a firm foundation for biomedical research in the decades ahead

Incident type 2 diabetes attributable to suboptimal diet in 184 countries

The global burden of diet-attributable type 2 diabetes (T2D) is not well established. This risk assessment model estimated T2D incidence among adults attributable to direct and body weight-mediated effects of 11 dietary factors in 184 countries in 1990 and 2018. In 2018, suboptimal intake of these dietary factors was estimated to be attributable to 14.1 million (95% uncertainty interval (UI), 13.8–14.4 million) incident T2D cases, representing 70.3% (68.8–71.8%) of new cases globally. Largest T2D burdens were attributable to insufficient whole-grain intake (26.1% (25.0–27.1%)), excess refined rice and wheat intake (24.6% (22.3–27.2%)) and excess processed meat intake (20.3% (18.3–23.5%)). Across regions, highest proportional burdens were in central and eastern Europe and central Asia (85.6% (83.4–87.7%)) and Latin America and the Caribbean (81.8% (80.1–83.4%)); and lowest proportional burdens were in South Asia (55.4% (52.1–60.7%)). Proportions of diet-attributable T2D were generally larger in men than in women and were inversely correlated with age. Diet-attributable T2D was generally larger among urban versus rural residents and higher versus lower educated individuals, except in high-income countries, central and eastern Europe and central Asia, where burdens were larger in rural residents and in lower educated individuals. Compared with 1990, global diet-attributable T2D increased by 2.6 absolute percentage points (8.6 million more cases) in 2018, with variation in these trends by world region and dietary factor. These findings inform nutritional priorities and clinical and public health planning to improve dietary quality and reduce T2D globally