Strangeness-driven Exploration in Multi-Agent Reinforcement Learning

Efficient exploration strategy is one of essential issues in cooperative multi-agent reinforcement learning (MARL) algorithms requiring complex coordination. In this study, we introduce a new exploration method with the strangeness that can be easily incorporated into any centralized training and decentralized execution (CTDE)-based MARL algorithms. The strangeness refers to the degree of unfamiliarity of the observations that an agent visits. In order to give the observation strangeness a global perspective, it is also augmented with the the degree of unfamiliarity of the visited entire state. The exploration bonus is obtained from the strangeness and the proposed exploration method is not much affected by stochastic transitions commonly observed in MARL tasks. To prevent a high exploration bonus from making the MARL training insensitive to extrinsic rewards, we also propose a separate action-value function trained by both extrinsic reward and exploration bonus, on which a behavioral policy to generate transitions is designed based. It makes the CTDE-based MARL algorithms more stable when they are used with an exploration method. Through a comparative evaluation in didactic examples and the StarCraft Multi-Agent Challenge, we show that the proposed exploration method achieves significant performance improvement in the CTDE-based MARL algorithms.Comment: 9 pages, 7 figure

Improvement in the hygroscopicity of inorganic binder through a dual coating process

The use of an anti-absorbent is proposed in this work to reduce the hygroscopicity of the inorganic binder in the casting mold, in which the anti-absorbent is coated on the mold prepared with an inorganic binder. Three types of polymers were used to select material with optimal water resistance. Polystyrene (PS) and polyvinyl alcohol (PVA) were used as a water-insoluble polymer and water-soluble polymer, respectively. In addition, polyurethane (PU) prepolymer has intermediate properties between PS and PVA. PVA and PU prepolymer were used for comparative testing with PS. For this testing process, the prepared green body was dipped into a solution of inorganic binder precursor mixed with tetraethyl orthosilicate (TEOS, SiO2 precursor) and sodium methoxide (NaOMe, Na2O precursor), and then dipped into a solution of coating reagent after a drying process. Thus, these series of coating processes in a green body is called a dual coating process. Finally the sample was heat-treated at 1000 °C to generate a glass phase by an organic–inorganic conversion process. In the sample prepared with PS, the highest contact angle and a high firing strength were exhibited, independent of polymer concentration, while the sample coated with PVA showed lower green and firing strengths. When prepolymer, PU, was applied, the green strength was remarkably improved, showing lower firing strength compared with that of PS. The green and firing strengths were optimized through the dual coating process with PS. Moreover, the moisture-proof effect of the dual coating process was verified through the moisture steam test

Impact of commercial cigarette smoke condensate on brain tissue co-cultured with astrocytes and blood-brain barrier endothelial cells

The purpose of the current study was to investigate the effect of two commercial cigarette smoke condensates (CCSC) on oxidative stress and cell cytotoxicity in human brain (T98G) or astrocytes (U-373 MG) in the presence of human brain microvascular endothelial cells (HBMEC). Cell viability of mono-culture of T98G or U-373 MG was markedly decreased in a concentration-dependent manner, and T98G was more susceptible than U-373 MG to CCSC exposure. Cytotoxicity was less prominent when T98G was co-cultured with HBMEC than when T98G was co-cultured with U-373 MG. Significant reduction in trans-epithelial electric resistance (TEER), a biomarker of cellular integrity was noted in HBMEC co-cultured with T98G (HBMEC-T98G co-culture) and U-373 MG co-cultured with T98G (U-373 MG-T98G co-culture) after 24 or 48 hr CCSC exposure, respectively. TEER value of U-373 MG co-cultured with T98G (79-84%) was higher than HBMEC co-cultured with T98G (62-63%) within 120-hr incubation with CCSC. Reactive oxygen species (ROS) generated by CCSC in mono-culture of T98G and U-373 MG reached highest levels at 4 and 16 mg/ml, respectively. ROS production by T98G fell when co-cultured with HBMEC or U-373MG. These findings suggest that adverse consequences of CCSC treatment on brain cells may be protected by blood-brain barrier or astrocytes, but with chronic exposure toxicity may be worsened due to destruction of cellular integrity.

Transcutaneous medial fixation sutures for free flap inset after robot-assisted nipple-sparing mastectomy

The application of minimal invasive mastectomy has allowed surgeons to perform nipplesparing mastectomy via a shorter, inconspicuous incision under clear vision and with more precise hemostasis. However, it poses new challenges in microsurgical breast reconstruction, such as vascular anastomosis and flap insetting, which are considerably more difficult to perform through the shorter incision on the lateral breast border. We propose an innovative technique of transcutaneous medial fixation sutures to help in flap insetting and creating and maintaining the medial breast border. The sutures are placed after mastectomy and before flap transfer. Three 4-0 nylon suture loops are placed transcutaneously and into the pocket at the markings of the preferred lower medial border of the reconstructed breast. After microvascular anastomosis and temporary shaping of the flap on top of the mastectomy skin, the three corresponding points for the sutures are identified. The three nylon loops are then sutured to the dermis of the corresponding medial point of the flap. The flap is placed into the pocket by a simultaneous gentle pull on the three sutures and a combined lateral push. The stitches are then tied and buried after completion of flap inset

Monoclinic and Correlated Metal Phase in VO_2 as Evidence of the Mott Transition: Coherent Phonon Analysis

In femtosecond pump-probe measurements, the appearance of coherent phonon oscillations at 4.5 THz and 6.0 THz indicating the rutile metal phase of VO_2 does not occur simultaneously with the first-order metal-insulator transition (MIT) near 68^oC. The monoclinic and correlated metal(MCM) phase between the MIT and the structural phase transition (SPT) is generated by a photo-assisted hole excitation which is evidence of the Mott transition. The SPT between the MCM phase and the rutile metal phase occurs due to subsequent Joule heating. The MCM phase can be regarded as an intermediate non-equilibrium state.Comment: 4 pages, 2 figure

Readout-segmented echo-planar imaging in diffusion-weighted mr imaging in breast cancer: comparison with single-shot echo-planar imaging in image quality

Objective: The purpose of this study was to compare the image quality of standard single-shot echo-planar imaging (ss-EPI) and that of readout-segmented EPI (rs-EPI) in patients with breast cancer. Materials and Methods: Seventy-one patients with 74 breast cancers underwent both ss-EPI and rs-EPI. For qualitative comparison of image quality, three readers independently assessed the two sets of diffusion-weighted (DW) images. To evaluate geometric distortion, a comparison was made between lesion lengths derived from contrast enhanced MR (CE-MR) images and those obtained from the corresponding DW images. For assessment of image parameters, signal-to-noise ratio (SNR), lesion contrast, and contrast-to-noise ratio (CNR) were calculated. Results: The rs-EPI was superior to ss-EPI in most criteria regarding the qualitative image quality. Anatomical structure distinction, delineation of the lesion, ghosting artifact, and overall image quality were significantly better in rs-EPI. Regarding the geometric distortion, lesion length on ss-EPI was significantly different from that of CE-MR, whereas there were no significant differences between CE-MR and rs-EPI. The rs-EPI was superior to ss-EPI in SNR and CNR. Conclusion: Readout-segmented EPI is superior to ss-EPI in the aspect of image quality in DW MR imaging of the breast

TRAIL sensitize MDR cells to MDR-related drugs by down-regulation of P-glycoprotein through inhibition of DNA-PKcs/Akt/GSK-3β pathway and activation of caspases

<p>Abstract</p> <p>Background</p> <p>The development of new modulator possessing high efficacy, low toxicity and high selectivity is a pivotal approach to overcome P-glycoprotein (P-gp) mediated multidrug resistance (MDR) in cancer treatment. In this study, we suggest a new molecular mechanism that TRAIL (tumor necrosis factor-related apoptosis-inducing ligand) down-regulates P-glycoprotein (P-gp) through inhibition of DNA-PKcs/Akt/GSK-3β pathway and activation of caspases and thereby sensitize MDR cells to MDR-related drugs.</p> <p>Results</p> <p>MDR variants, CEM/VLB_10-2, CEM/VLB_55-8and CEM/VLB₁₀₀cells, with gradually increased levels of P-gp derived from human lymphoblastic leukemia CEM cells, were gradually more susceptible to TRAIL-induced apoptosis and cytotoxicity than parental CEM cells. The P-gp level of MDR variants was positively correlated with the levels of DNA-PKcs, pAkt, pGSK-3β and c-Myc as well as DR5 and negatively correlated with the level of c-FLIPs. Hypersensitivity of CEM/VLB₁₀₀cells to TRAIL was accompanied by the activation of mitochondrial apoptotic pathway as well as the activation of initiator caspases. In addition, TRAIL-induced down-regulation of DNA-PKcs/Akt/GSK-3β pathway and c-FLIP and up-regulation of cell surface expression of death receptors were associated with the increased susceptibility to TRAIL of MDR cells. Moreover, TRAIL inhibited P-gp efflux function via caspase-3-dependent degradation of P-gp as well as DNA-PKcs and subsequently sensitized MDR cells to MDR-related drugs such as vinblastine and doxorubicin. We also found that suppression of DNA-PKcs by siRNA enhanced the susceptibility of MDR cells to vincristine as well as TRAIL via down-regulation of c-FLIP and P-gp expression and up-regulation of DR5.</p> <p>Conclusion</p> <p>This study showed for the first time that the MDR variant of CEM cells was hypersensitive to TRAIL due to up-regulation of DR5 and concomitant down-regulation of c-FLIP, and degradation of P-gp and DNA-PKcs by activation of caspase-3 might be important determinants of TRAIL-induced sensitization of MDR cells to MDR-related drugs. Therefore, combination of TRAIL and chemotherapeutic drugs may be a good strategy for treatment of cancer with multidrug resistance.</p