Strangeness-driven Exploration in Multi-Agent Reinforcement Learning

Efficient exploration strategy is one of essential issues in cooperative multi-agent reinforcement learning (MARL) algorithms requiring complex coordination. In this study, we introduce a new exploration method with the strangeness that can be easily incorporated into any centralized training and decentralized execution (CTDE)-based MARL algorithms. The strangeness refers to the degree of unfamiliarity of the observations that an agent visits. In order to give the observation strangeness a global perspective, it is also augmented with the the degree of unfamiliarity of the visited entire state. The exploration bonus is obtained from the strangeness and the proposed exploration method is not much affected by stochastic transitions commonly observed in MARL tasks. To prevent a high exploration bonus from making the MARL training insensitive to extrinsic rewards, we also propose a separate action-value function trained by both extrinsic reward and exploration bonus, on which a behavioral policy to generate transitions is designed based. It makes the CTDE-based MARL algorithms more stable when they are used with an exploration method. Through a comparative evaluation in didactic examples and the StarCraft Multi-Agent Challenge, we show that the proposed exploration method achieves significant performance improvement in the CTDE-based MARL algorithms.Comment: 9 pages, 7 figure

Induction of IL-10-producing CD4(+)CD25(+ )T cells in animal model of collagen-induced arthritis by oral administration of type II collagen

Induction of oral tolerance has long been considered a promising approach to the treatment of chronic autoimmune diseases, including rheumatoid arthritis (RA). Oral administration of type II collagen (CII) has been proven to improve signs and symptoms in RA patients without troublesome toxicity. To investigate the mechanism of immune suppression mediated by orally administered antigen, we examined changes in serum IgG subtypes and T-cell proliferative responses to CII, and generation of IL-10-producing CD4(+)CD25(+ )T-cell subsets in an animal model of collagen-induced arthritis (CIA). We found that joint inflammation in CIA mice peaked at 5 weeks after primary immunization with CII, which was significantly less in mice tolerized by repeated oral feeding of CII before CIA induction. Mice that had been fed with CII also exhibited increased serum IgG(1 )and decreased serum IgG(2a )as compared with nontolerized CIA animals. The T-cell proliferative response to CII was suppressed in lymph nodes of tolerized mice also. Production of IL-10 and of transforming growth factor-β from mononuclear lymphocytes was increased in the tolerized animals, and CD4(+ )T cells isolated from tolerized mice did not respond with induction of IFN-γ when stimulated in vitro with CII. We also observed greater induction of IL-10-producing CD4(+)CD25(+ )subsets among CII-stimulated splenic T cells from tolerized mice. These data suggest that when these IL-10-producing CD4(+)CD25(+ )T cells encounter CII antigen in affected joints they become activated to exert an anti-inflammatory effect

IL-17 induces production of IL-6 and IL-8 in rheumatoid arthritis synovial fibroblasts via NF-κB- and PI3-kinase/Akt-dependent pathways

Recent studies of the pathogenesis of rheumatoid arthritis (RA) have revealed that both synovial fibroblasts and T cells participate in the perpetuation of joint inflammation as dynamic partners in a mutual activation feedback, via secretion of cytokines and chemokines that stimulate each other. In this study, we investigated the role of IL-17, a major Th1 cytokine produced by activated T cells, in the activation of RA synovial fibroblasts. Transcripts of IL-17R (IL-17 receptor) and IL-17RB (IL-17 receptor B) were present in fibroblast-like synoviocytes (FLS) of RA patients. IL-17R responded with increased expression upon in vitro stimulation with IL-17, while the level of IL-17RB did not change. IL-17 enhanced the production of IL-6 and IL-8 in FLS, as previously shown, but did not affect the synthesis of IL-15. IL-17 appears to be a stronger inducer of IL-6 and IL-8 than IL-15, and even exerted activation comparable to that of IL-1β in RA FLS. IL-17-mediated induction of IL-6 and IL-8 was transduced via activation of phosphatidylinositol 3-kinase/Akt and NF-κB, while CD40 ligation and p38 MAPK (mitogen-activated protein kinase) are not likely to partake in the process. Together these results suggest that IL-17 is capable of more than accessory roles in the activation of RA FLS and provide grounds for targeting IL-17-associated pathways in therapeutic modulation of arthritis inflammation

Determination of staphylococcal exotoxins, SCCmec types, and genetic relatedness of Staphylococcus intermedius group isolates from veterinary staff, companion animals, and hospital environments in Korea

The Staphylococcus (S.) intermedius group (SIG) has been a main research subject in recent years. S. pseudintermedius causes pyoderma and otitis in companion animals as well as foodborne diseases. To prevent SIG-associated infection and disease outbreaks, identification of both staphylococcal exotoxins and staphylococcal cassette chromosome mec (SCCmec) types among SIG isolates may be helpful. In this study, it was found that a single isolate (one out of 178 SIG isolates examined) harbored the canine enterotoxin SEC gene. However, the S. intermedius exfoliative toxin gene was found in 166 SIG isolates although the S. aureus-derived exfoliative toxin genes, such as eta, etb and etd, were not detected. SCCmec typing resulted in classifying one isolate as SCCmec type IV, 41 isolates as type V (including three S. intermedius isolates), and 10 isolates as non-classifiable. Genetic relatedness of all S. pseudintermedius isolates recovered from veterinary staff, companion animals, and hospital environments was determined by pulsed-field gel electrophoresis. Strains having the same band patterns were detected in S. pseudintermedius isolates collected at 13 and 18 months, suggesting possible colonization and/or expansion of a specific S. pseudintermedius strain in a veterinary hospital

Development of Monoclonal Antibodies Against Human IRF-5 and Their Use in Identifying the Binding of IRF-5 to Nuclear Import Proteins Karyopherin-α1 and -β1

PURPOSE: IRF-5 is a direct transducer of virus-mediated and TLR-mediated signaling pathways for the expression of cytokines and chemokines which form homodimers or heterodimers with IRF-7. However, direct IRF-5-specific monoclonal antibodies (mAbs) are not available at present. These could be used to further evaluate the functions of IRF-5. In this study, we produced and characterized three mouse mAbs to human IRF-5. The binding of IRF-5 to nuclear import proteins was first identified using a mAb. MATERIALS AND METHODS: His-tagged human IRF-5 protein spanning amino acid residues 193-257 was used as an antigen and three mAbs were produced. The mAbs were tested with ELISA, Western blot analysis (WB), immunofluorescent staining (IF), and immunoprecipitation (IP). In addition, the nuclear import protein which carried phosphorylated IRF-5 was identified using one of these mAbs. RESULTS: MAbs 5IRF8, 5IRF10 and 5IRF24 which reacted with the recombinant His-IRF-5(193-257) protein were produced. All mAbs bound to human IRF-5, but not to IRF-3 or IRF-7. They could be used for WB, IF, and IP studies. The binding of phosphorylated IRF-5 to karyopherin-alpha1 and -beta1 was also identified. CONCLUSION: Human IRF-5-specific mAbs are produced for studying the immunologic roles related to IRF-5. Phosphorylated IRF-5 is transported to the nucleus by binding to nuclear import proteins karyopherin-alpha1 and -beta1.ope

Diffusion MR Imaging of Postoperative Bilateral Acute Ischemic Optic Neuropathy

A 57-year-old woman experienced bilateral acute ischemic optic neuropathy after spine surgery. Routine MR imaging sequence, T2-weighted image, showed subtle high signal intensity on bilateral optic nerves. A contrast-enhanced T1 weighted image showed enhancement along the bilateral optic nerve sheath. Moreover, diffusion-weighted image (DWI) and an apparent diffusion coefficient map showed markedly restricted diffusion on bilateral optic nerves. Although MR findings of T2-weighted and contrast enhanced T1-weighted images may be nonspecific, the DWI finding of cytotoxic edema of bilateral optic nerves will be helpful for the diagnosis of acute ischemic optic neuropathy after spine surgery