Strangeness-driven Exploration in Multi-Agent Reinforcement Learning

Efficient exploration strategy is one of essential issues in cooperative multi-agent reinforcement learning (MARL) algorithms requiring complex coordination. In this study, we introduce a new exploration method with the strangeness that can be easily incorporated into any centralized training and decentralized execution (CTDE)-based MARL algorithms. The strangeness refers to the degree of unfamiliarity of the observations that an agent visits. In order to give the observation strangeness a global perspective, it is also augmented with the the degree of unfamiliarity of the visited entire state. The exploration bonus is obtained from the strangeness and the proposed exploration method is not much affected by stochastic transitions commonly observed in MARL tasks. To prevent a high exploration bonus from making the MARL training insensitive to extrinsic rewards, we also propose a separate action-value function trained by both extrinsic reward and exploration bonus, on which a behavioral policy to generate transitions is designed based. It makes the CTDE-based MARL algorithms more stable when they are used with an exploration method. Through a comparative evaluation in didactic examples and the StarCraft Multi-Agent Challenge, we show that the proposed exploration method achieves significant performance improvement in the CTDE-based MARL algorithms.Comment: 9 pages, 7 figure

Normal stress difference-driven particle focusing in nanoparticle colloidal dispersion

Colloidal dispersion has elastic properties due to Brownian relaxation process. However, experimental evidence for the elastic properties, characterized with normal stress differences, is elusive in shearing colloidal dispersion, particularly at low Peclet numbers (Pe < 1). Here, we report that single micrometer-sized polystyrene (PS) beads, suspended in silica nanoparticle dispersion (8 nm radius; 22%, v/v), laterally migrate and form a tightly focused stream by the normal stress differences, generated in pressure-driven microtube flow at low Pe. The nanoparticle dispersion was expected to behave as a Newtonian fluid because of its ultrashort relaxation time (2 mu s), but large shear strain experienced by the PS beads causes the notable non-Newtonian behavior. We demonstrate that the unique rheological properties of the nanoparticle dispersion generate the secondary flow in perpendicular to mainstream in a noncircular conduit, and the elastic properties of blood plasma-constituting protein solutions are elucidated by the colloidal dynamics of protein molecules

Selective emitter using a screen printed etch barrier in crystalline silicon solar cell

The low level doping of a selective emitter by etch back is an easy and low cost process to obtain a better blue response from a solar cell. This work suggests that the contact resistance of the selective emitter can be controlled by wet etching with the commercial acid barrier paste that is commonly applied in screen printing. Wet etching conditions such as acid barrier curing time, etchant concentration, and etching time have been optimized for the process, which is controllable as well as fast. The acid barrier formed by screen printing was etched with HF and HNO(3) (1:200) solution for 15 s, resulting in high sheet contact resistance of 90 Ω/sq. Doping concentrations of the electrode contact portion were 2 × 10(21) cm(−3) in the low sheet resistance (Rs) region and 7 × 10(19) cm(−3) in the high Rs region. Solar cells of 12.5 × 12.5 cm(2) in dimensions with a wet etch back selective emitter J(sc) of 37 mAcm(−2), open circuit voltage (V(oc)) of 638.3 mV and efficiency of 18.13% were fabricated. The result showed an improvement of about 13 mV on V(oc) compared to those of the reference solar cell fabricated with the reactive-ion etching back selective emitter and with J(sc) of 36.90 mAcm(−2), V(oc) of 625.7 mV, and efficiency of 17.60%

Role of interleukin-10 in endochondral bone formation in mice: Anabolic effect via the bone morphogenetic protein/Smad pathway

Objective: Interleukin-10 (IL-10) is a pleiotropic immunoregulatory cytokine with a chondroprotective effect that is elevated in cartilage and synovium in patients with osteoarthritis. However, the role of IL-10 during endochondral bone formation and its mechanism of action have not been elucidated. Methods: IL-10-/- mice and IL-10-treated tibial organ cultures were used to study loss and gain of IL-10 functions, respectively, during endochondral bone formation. Primary chondrocytes from the long bones of mouse embryos were cultured with and without IL-10. To assess the role of IL-10 in chondrogenic differentiation, we conducted mesenchymal cell micromass cultures. Results: The lengths of whole skeletons from IL-10-/- mice were similar to those of their wild-type littermates, although their skull diameters were smaller. The tibial growth plates of IL-10-/- mice showed shortening of the proliferating zone. Treatment with IL-10 significantly increased tibial lengths in organ culture. IL-10 also induced chondrocyte proliferation and hypertrophic differentiation in primary chondrocytes in vitro. Mechanistically, IL-10 activated STAT-3 and the Smad1/5/8 and ERK-1/2 MAP kinase pathways and induced the expression of bone morphogenetic protein 2 (BMP-2) and BMP-6 in primary chondrocytes. Furthermore, the blocking of BMP signaling attenuated the IL-10-mediated induction of cyclin D1 and RUNX-2 in primary chondrocytes and suppressed Alcian blue and alkaline phosphatase staining in mesenchymal cell micromass cultures. Conclusion: These results indicate that IL-10 acts as a stimulator of chondrocyte proliferation and chondrogenic or hypertrophic differentiation via activation of the BMP signaling pathway. © 2013, American College of Rheumatology

Optical Imaging of Cancer-Related Proteases Using Near-Infrared Fluorescence Matrix Metalloproteinase-Sensitive and Cathepsin B-Sensitive Probes

Cathepsin B and matrix metalloproteinase (MMP) play key roles in tumor progression by controlled degradation of extracellular matrix. Consequently, these proteases have been attracted in cancer research, and many imaging probes utilizing these proteases have been developed. Our groups developed cathepsin B and MMP imaging nanoprobes based on polymer nanoparticle platform. Both cathepsin B and MMP imaging probes used near-infrared fluorescence (NIRF) dye and dark-quencher to for high sensitivity, and protease-sensitive peptide sequence in each probe authorized high specificity of the probes. We compared the bioactivities of cathepsin B and MMP sensitive probes in cancer-related environments to investigate the biological property of the probes. As a result, cathepsin B probe showed fluorescence recovery after the probe entered the cytoplasm. This property could be useful to evaluate the cytoplasmic targeted delivery by using probe-conjugated nanoparticles in vivo. On the other hand, MMP probe was superior in specificity in vivo and tissue study. This comparative study will provide precise information about peptide-based optical probes, and allow their proper application to cancer diagnosis