A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning

A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other simultaneously learning agents. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural non-stationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accounts for the non-stationary policy dynamics inherent to multiagent learning settings. This is achieved by modeling our gradient updates to consider both an agent's own non-stationary policy dynamics and the non-stationary policy dynamics of other agents in the environment. We show that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently comprises all key aspects of previous state of the art approaches on this topic. We test our method on a diverse suite of multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than baseline methods across the full spectrum of mixed incentive, competitive, and cooperative domains.Comment: Accepted to ICML 2021. Code at https://github.com/dkkim93/meta-mapg and Videos at https://sites.google.com/view/meta-mapg/hom

Left-Right Symmetric Heterotic-String Derived Models

Recently it was demonstrated that free fermionic heterotic-strings can produce models with solely the Minimal Supersymmetric Standard Model states in the low energy spectrum. This unprecedented result provides further strong evidence for the possibility that the true string vacuum shares some of the properties of the free fermionic models. Past free fermionic models have focused on several possible unbroken observable SO(10) subgroups at the string scale, which include the flipped SU(5) (FSU5), the Pati-Salam (PS) string models, and the string Standard-like Models (SLM). We extend this study to include the case in which the SO(10) symmetry is broken to the Left-Right Symmetric (LRS) gauge group, SO(10) -> SU(3)_C X U(1)_{B-L} X SU(2)_L X SU(2)_R. We present several models of this type and discuss their phenomenological features. The most striking new outcome of the LRS string models, in contrast to the case of the FSU5, the PS, and the SLM string models, is that they can produce effective field theories that are free of Abelian anomalies. We discuss the distinction between the two types of free fermionic models which result in the presence, or absence, of an anomalous U(1). As a counter example we also present a LRS model that does contain an anomalous U(1). Additionally, we discuss how in string models the Standard Model spectrum may arise from the three \mbf 16 representations of SO(10), while the weak-hypercharge does not have the canonical SO(10) embedding.Comment: 39 pages. Standard Latex. Version to appear in PR

Increased levels of macrophage inflammatory proteins result in resistance to R5-tropic HIV-1 in a subset of elite controllers

Elite controllers (ECs) are a rare group of HIV seropositive individuals who are able to control viral replication without antiretroviral therapy. The mechanisms responsible for this phenotype, however, have not been fully elucidated. In this study, we examined CD4+ T cell resistance to HIV in a cohort of elite controllers and explored transcriptional signatures associated with cellular resistance. We demonstrate that a subgroup of elite controllers possess CD4+ T cells that are specifically resistant to R5-tropic HIV while remaining fully susceptible to X4-tropic and vesicular stomatitis virus G (VSV-G)-pseudotyped viruses. Transcriptome analysis revealed 17 genes that were differentially regulated in resistant elite controllers relative to healthy controls. Notably, the genes encoding macrophage inflammatory protein 1α (MIP-1α), CCL3 and CCL3L1, were found to be upregulated. The MIP-1α, MIP-1β, and RANTES chemokines are natural ligands of CCR5 and are known to interfere with HIV replication. For three elite controllers, we observed increased production of MIP-1α and/or MIP-1β at the protein level. The supernatant from resistant EC cells contained MIP-1α and MIP-1β and was sufficient to confer R5-tropic resistance to susceptible CD4+ T cells. Additionally, this effect was reversed by using inhibitory anti-MIP antibodies. These results suggest that the T cells of these particular elite controllers may be naturally resistant to HIV infection by blocking R5-tropic viral entr

Frequency and methylation status of selected retrotransposition competent L1 loci in amyotrophic lateral sclerosis

Long interspersed element-1 (LINE-1/L1) is the only autonomous transposable element in the human genome that currently mobilises in both germline and somatic tissues. Recent studies have identified correlations between altered retrotransposon expression and the fatal neurodegenerative disease amyotrophic lateral sclerosis (ALS) in a subset of patients. The risk of an individual developing ALS is dependent on an interaction of genetic variants and subsequent modifiers during life. These modifiers could include environmental factors, which can lead to epigenetic and genomic changes, such as somatic mutations, occurring in the neuronal cells that degenerate as the disease develops. There are more than 1 million L1 copies in the human genome today, but only 80–100 L1 loci in the reference genome are considered to be retrotransposition-competent (RC) and an even smaller number of these RC-L1s loci are highly active. We hypothesise that RC-L1s could affect normal cellular function through their mutagenic potential conferred by their ability to retrotranspose in neuronal cells and through DNA damage caused by the endonuclease activity of the L1-encoded ORF2 protein. To investigate whether either an increase in the genomic burden of RC-L1s or epigenetic changes to RC-L1s altering their expression, could play a role in disease development, we chose a set of seven well characterised genomic RC-L1 loci that were reported earlier to be highly active in a cellular L1 retrotransposition reporter assay or serve as major source elements for germline and/or somatic retrotransposition events. Analysis of the insertion allele frequency of five polymorphic RC-L1s, out of the set of seven, for their presence or absence, did not identify an increased number individually or when combined in individuals with the disease. However, we did identify reduced levels of methylation of RC-L1s in the motor cortex of those individuals with both familial and sporadic ALS compared to control brains. The changes to the regulation of the loci encompassing these RC-L1s demonstrated tissue specificity and could be related to the disease process

Clinically Actionable Hypercholesterolemia and Hypertriglyceridemia in Children with Nonalcoholic Fatty Liver Disease

OBJECTIVE: To determine the percentage of children with nonalcoholic fatty liver disease (NAFLD) in whom intervention for low-density lipoprotein cholesterol or triglycerides was indicated based on National Heart, Lung, and Blood Institute guidelines. STUDY DESIGN: This multicenter, longitudinal cohort study included children with NAFLD enrolled in the National Institute of Diabetes and Digestive and Kidney Diseases Nonalcoholic Steatohepatitis Clinical Research Network. Fasting lipid profiles were obtained at diagnosis. Standardized dietary recommendations were provided. After 1 year, lipid profiles were repeated and interpreted according to National Heart, Lung, and Blood Institute Expert Panel on Integrated Guidelines for Cardiovascular Health and Risk Reduction. Main outcomes were meeting criteria for clinically actionable dyslipidemia at baseline, and either achieving lipid goal at follow-up or meeting criteria for ongoing intervention. RESULTS: There were 585 participants, with a mean age of 12.8 years. The prevalence of children warranting intervention for low-density lipoprotein cholesterol at baseline was 14%. After 1 year of recommended dietary changes, 51% achieved goal low-density lipoprotein cholesterol, 27% qualified for enhanced dietary and lifestyle modifications, and 22% met criteria for pharmacologic intervention. Elevated triglycerides were more prevalent, with 51% meeting criteria for intervention. At 1 year, 25% achieved goal triglycerides with diet and lifestyle changes, 38% met criteria for advanced dietary modifications, and 37% qualified for antihyperlipidemic medications. CONCLUSIONS: More than one-half of children with NAFLD met intervention thresholds for dyslipidemia. Based on the burden of clinically relevant dyslipidemia, lipid screening in children with NAFLD is warranted. Clinicians caring for children with NAFLD should be familiar with lipid management