A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning

A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other simultaneously learning agents. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural non-stationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accounts for the non-stationary policy dynamics inherent to multiagent learning settings. This is achieved by modeling our gradient updates to consider both an agent's own non-stationary policy dynamics and the non-stationary policy dynamics of other agents in the environment. We show that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently comprises all key aspects of previous state of the art approaches on this topic. We test our method on a diverse suite of multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than baseline methods across the full spectrum of mixed incentive, competitive, and cooperative domains.Comment: Accepted to ICML 2021. Code at https://github.com/dkkim93/meta-mapg and Videos at https://sites.google.com/view/meta-mapg/hom

Left-Right Symmetric Heterotic-String Derived Models

Recently it was demonstrated that free fermionic heterotic-strings can produce models with solely the Minimal Supersymmetric Standard Model states in the low energy spectrum. This unprecedented result provides further strong evidence for the possibility that the true string vacuum shares some of the properties of the free fermionic models. Past free fermionic models have focused on several possible unbroken observable SO(10) subgroups at the string scale, which include the flipped SU(5) (FSU5), the Pati-Salam (PS) string models, and the string Standard-like Models (SLM). We extend this study to include the case in which the SO(10) symmetry is broken to the Left-Right Symmetric (LRS) gauge group, SO(10) -> SU(3)_C X U(1)_{B-L} X SU(2)_L X SU(2)_R. We present several models of this type and discuss their phenomenological features. The most striking new outcome of the LRS string models, in contrast to the case of the FSU5, the PS, and the SLM string models, is that they can produce effective field theories that are free of Abelian anomalies. We discuss the distinction between the two types of free fermionic models which result in the presence, or absence, of an anomalous U(1). As a counter example we also present a LRS model that does contain an anomalous U(1). Additionally, we discuss how in string models the Standard Model spectrum may arise from the three \mbf 16 representations of SO(10), while the weak-hypercharge does not have the canonical SO(10) embedding.Comment: 39 pages. Standard Latex. Version to appear in PR

Increased levels of macrophage inflammatory proteins result in resistance to R5-tropic HIV-1 in a subset of elite controllers

Elite controllers (ECs) are a rare group of HIV seropositive individuals who are able to control viral replication without antiretroviral therapy. The mechanisms responsible for this phenotype, however, have not been fully elucidated. In this study, we examined CD4+ T cell resistance to HIV in a cohort of elite controllers and explored transcriptional signatures associated with cellular resistance. We demonstrate that a subgroup of elite controllers possess CD4+ T cells that are specifically resistant to R5-tropic HIV while remaining fully susceptible to X4-tropic and vesicular stomatitis virus G (VSV-G)-pseudotyped viruses. Transcriptome analysis revealed 17 genes that were differentially regulated in resistant elite controllers relative to healthy controls. Notably, the genes encoding macrophage inflammatory protein 1α (MIP-1α), CCL3 and CCL3L1, were found to be upregulated. The MIP-1α, MIP-1β, and RANTES chemokines are natural ligands of CCR5 and are known to interfere with HIV replication. For three elite controllers, we observed increased production of MIP-1α and/or MIP-1β at the protein level. The supernatant from resistant EC cells contained MIP-1α and MIP-1β and was sufficient to confer R5-tropic resistance to susceptible CD4+ T cells. Additionally, this effect was reversed by using inhibitory anti-MIP antibodies. These results suggest that the T cells of these particular elite controllers may be naturally resistant to HIV infection by blocking R5-tropic viral entr

Frequency and methylation status of selected retrotransposition competent L1 loci in amyotrophic lateral sclerosis

Long interspersed element-1 (LINE-1/L1) is the only autonomous transposable element in the human genome that currently mobilises in both germline and somatic tissues. Recent studies have identified correlations between altered retrotransposon expression and the fatal neurodegenerative disease amyotrophic lateral sclerosis (ALS) in a subset of patients. The risk of an individual developing ALS is dependent on an interaction of genetic variants and subsequent modifiers during life. These modifiers could include environmental factors, which can lead to epigenetic and genomic changes, such as somatic mutations, occurring in the neuronal cells that degenerate as the disease develops. There are more than 1 million L1 copies in the human genome today, but only 80–100 L1 loci in the reference genome are considered to be retrotransposition-competent (RC) and an even smaller number of these RC-L1s loci are highly active. We hypothesise that RC-L1s could affect normal cellular function through their mutagenic potential conferred by their ability to retrotranspose in neuronal cells and through DNA damage caused by the endonuclease activity of the L1-encoded ORF2 protein. To investigate whether either an increase in the genomic burden of RC-L1s or epigenetic changes to RC-L1s altering their expression, could play a role in disease development, we chose a set of seven well characterised genomic RC-L1 loci that were reported earlier to be highly active in a cellular L1 retrotransposition reporter assay or serve as major source elements for germline and/or somatic retrotransposition events. Analysis of the insertion allele frequency of five polymorphic RC-L1s, out of the set of seven, for their presence or absence, did not identify an increased number individually or when combined in individuals with the disease. However, we did identify reduced levels of methylation of RC-L1s in the motor cortex of those individuals with both familial and sporadic ALS compared to control brains. The changes to the regulation of the loci encompassing these RC-L1s demonstrated tissue specificity and could be related to the disease process

Different modes of interaction by TIAR and HuR with target RNA and DNA

TIAR and HuR are mRNA-binding proteins that play important roles in the regulation of translation. They both possess three RNA recognition motifs (RRMs) and bind to AU-rich elements (AREs), with seemingly overlapping specificity. Here we show using SPR that TIAR and HuR bind to both U-rich and AU-rich RNA in the nanomolar range, with higher overall affinity for U-rich RNA. However, the higher affinity for U–rich sequences is mainly due to faster association with U-rich RNA, which we propose is a reflection of the higher probability of association. Differences between TIAR and HuR are observed in their modes of binding to RNA. TIAR is able to bind deoxy-oligonucleotides with nanomolar affinity, whereas HuR affinity is reduced to a micromolar level. Studies with U-rich DNA reveal that TIAR binding depends less on the 2′-hydroxyl group of RNA than HuR binding. Finally we show that SAXS data, recorded for the first two domains of TIAR in complex with RNA, are more consistent with a flexible, elongated shape and not the compact shape that the first two domains of Hu proteins adopt upon binding to RNA. We thus propose that these triple-RRM proteins, which compete for the same binding sites in cells, interact with their targets in fundamentally different ways