Diverse Exploration via InfoMax Options

In this paper, we study the problem of autonomously discovering temporally abstracted actions, or options, for exploration in reinforcement learning. For learning diverse options suitable for exploration, we introduce the infomax termination objective defined as the mutual information between options and their corresponding state transitions. We derive a scalable optimization scheme for maximizing this objective via the termination condition of options, yielding the InfoMax Option Critic (IMOC) algorithm. Through illustrative experiments, we empirically show that IMOC learns diverse options and utilizes them for exploration. Moreover, we show that IMOC scales well to continuous control tasks.Comment: Preprint. Under revie

A novel method of cultivating cardiac myocytes in agarose microchamber chips for studying cell synchronization

We have developed a new method that enables agar microstructures to be used to cultivate cardiac myocyte cells in a manner that allows their connection patterns to be controlled. Non-contact three-dimensional photo-thermal etching with a 1064-nm infrared focused laser beam was used to form the shapes of agar microstructures. This wavelength was selected as it is not absorbed by water or agar. Identical rat cardiac myocytes were cultured in adjacent microstructures connected by microchannels and the interactions of asynchronous beating cardiac myocyte cells observed. Two isolated and independently beating cardiac myocytes were shown to form contacts through the narrow microchannels and by 90 minutes had synchronized their oscillations. This occurred by one of the two cells stopping their oscillation and following the pattern of the other cell. In contrast, when two sets of synchronized beating cells came into contact, those two sets synchronized without any observable interruptions to their rhythms. The results indicate that the synchronization process of cardiac myocytes may be dependent on the community size and network pattern of these cells

On-chip constructive cell-Network study (I): Contribution of cardiac fibroblasts to cardiomyocyte beating synchronization and community effect

<p>Abstract</p> <p>Backgrounds</p> <p>To clarify the role of cardiac fibroblasts in beating synchronization, we have made simple lined-up cardiomyocyte-fibroblast network model in an on-chip single-cell-based cultivation system.</p> <p>Results</p> <p>The synchronization phenomenon of two cardiomyocyte networks connected by fibroblasts showed (1) propagation velocity of electrophysiological signals decreased a magnitude depending on the increasing number of fibroblasts, not the lengths of fibroblasts; (2) fluctuation of interbeat intervals of the synchronized two cardiomyocyte network connected by fibroblasts did not always decreased, and was opposite from homogeneous cardiomyocyte networks; and (3) the synchronized cardiomyocytes connected by fibroblasts sometimes loses their synchronized condition and recovered to synchronized condition, in which the length of asynchronized period was shorter less than 30 beats and was independent to their cultivation time, whereas the length of synchronized period increased according to cultivation time.</p> <p>Conclusions</p> <p>The results indicated that fibroblasts can connect cardiomyocytes electrically but do not significantly enhance and contribute to beating interval stability and synchronization. This might also mean that an increase in the number of fibroblasts in heart tissue reduces the cardiomyocyte 'community effect', which enhances synchronization and stability of their beating rhythms.</p

Stability of beating frequency in cardiac myocytes by their community effect measured by agarose microchamber chip

To understand the contribution of community effect on the stability of beating frequency in cardiac myocyte cell groups, the stepwise network formation of cells as the reconstructive approach using the on-chip agarose microchamber cell microcultivation system with photo-thermal etching method was applied. In the system, the shapes of agarose microstructures were changed step by step with photo-thermal etching of agarose-layer of the chip using a 1064-nm infrared focused laser beam to increase the interaction of cardiac myocyte cells during cultivation. First, individual rat cardiac myocyte in each microstructure were cultivated under isolated condition, and then connected them one by one through newly-created microchannels by photo-thermal etching to compare the contribution of community size for the magnitude of beating stability of the cell groups. Though the isolated individual cells have 50% fluctuation of beating frequency, their stability increased as the number of connected cells increased. And finally when the number reached to eight cells, they stabilized around the 10% fluctuation, which was the same magnitude of the tissue model cultivated on the dish. The result indicates the importance of the community size of cells to stabilize their performance for making cell-network model for using cells for monitoring their functions like the tissue model

On-chip constructive cell-network study (II): on-chip quasi-in vivo cardiac toxicity assay for ventricular tachycardia/fibrillation measurement using ring-shaped closed circuit microelectrode with lined-up cardiomyocyte cell network

<p>Abstract</p> <p>Backgrounds</p> <p>Conventional <it>in vitro </it>approach using human ether-a-go-go related gene (hERG) assay has been considered worldwide as the first screening assay for cardiac repolarization safety. However, it does not always oredict the potential QT prolongation risk or pro-arrhythmic risk correctly. For adaptable preclinical strategiesto evaluate global cardiac safety, an on-chip quasi-<it>in vivo </it>cardiac toxicity assay for lethal arrhythmia (ventricular tachyarrhythmia) measurement using ring-shaped closed circuit microelectrode chip has been developed.</p> <p>Results</p> <p>The ventricular electrocardiogram (ECG)-like field potential data, which includes both the repolarization and the conductance abnormality, was acquired from the self-convolutied extracellular field potentials (FPs) of a lined-up cardiomyocyte network on a circle-shaped microelectrode in an agarose microchamber. When Astemisol applied to the closed-loop cardiomyocyte network, self-convoluted FP profile of normal beating changed into an early afterdepolarization (EAD) like waveform, and then showed ventricular tachyarrhythmias and ventricular fibrilations (VT/Vf). QT-prolongation-like self-convoluted FP duration prolongation and its fluctuation increase was also observed according to the increase of Astemizole concentration.</p> <p>Conclusions</p> <p>The results indicate that the convoluted FPs of the quasi<it>-in vivo </it>cell network assay includes both of the repolarization data and the conductance abnormality of cardiomyocyte networks has the strong potential to prediction lethal arrhythmia.</p

DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness crucially decides the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies showed the effectiveness of encouraging exploration with intrinsic rewards estimated from novelty in observations. However, there is a gap between the novelty of an observation and an exploration in general, because the stochasticity in the environment as well as the behavior of an agent may affect the observation. To estimate exploratory behaviors accurately, we propose DEIR, a novel method where we theoretically derive an intrinsic reward from a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and materialize the reward with a discriminative forward model. We conduct extensive experiments in both standard and hardened exploration games in MiniGrid to show that DEIR quickly learns a better policy than baselines. Our evaluations in ProcGen demonstrate both generalization capabilities and the general applicability of our intrinsic reward.Comment: Accepted as a conference paper to the 32nd International Joint Conference on Artificial Intelligence (IJCAI-23

Sol-Gel Preparation of Highly Water-Dispersible Silsesquioxane/Zirconium Oxide Hybrid Nanoparticles

Highly water-dispersible silsesquioxane/zirconium oxide hybrid nanoparticles (SQ/ZrO2-NPs) were prepared by the following two-step reactions. First, a mixture of 3-aminopropyltrimethoxysilane (APTMOS) and zirconium tetra-n-butoxide (ZTB) was stirred in a mixed solvent of 0.1 mol/L methanolic hydrochloric acid and n-butanol at room temperature, followed by heating in an open system until the solvent was completely evaporated. Then, the aqueous solution obtained by adding water to the resulting product was heated in an open system until the water was completely reevaporated. The products (SQ/ZrO2-NPs) obtained with a feed molar ratio of APTMOS/ZTB of more than 0.3 were dispersed well in water, and their aqueous dispersions were highly transparent, which was confirmed by UV-Vis measurements. In addition, the solid product obtained by drying its aqueous dispersion was redispersed in water. The volume-average particle sizes of SQ/ZrO2-NPs (APTMOS/ZTB ≥ 0.3) were assessed to be ca. 4.6–23.8 nm by dynamic light scattering measurements in water. The theoretical refractive index of the SQ/ZrO2-NP (APTMOS/ZTB = 0.3) was estimated to be 1.66. It was assumed that the water-dispersible property of the SQ/ZrO2-NPs probably originated from the ZrO2-SiO1.5(CH2)3Cl- core-shell structures

An innovative methodology/technology for streamflow observation

River engineeringInnovative field and laboratory instrumentatio