Solving Continual Combinatorial Selection via Deep Reinforcement Learning

We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present a deep RL algorithm to solve this issue by adopting the following key ideas. First, we convert the original S-MDP into an Iterative Select-MDP (IS-MDP), which is equivalent to the S-MDP in terms of optimal actions. IS-MDP decomposes a joint action of selecting K items simultaneously into K iterative selections resulting in the decrease of actions at the expense of an exponential increase of states. Second, we overcome this state space explo-sion by exploiting a special symmetry in IS-MDPs with novel weight shared Q-networks, which prov-ably maintain sufficient expressive power. Various experiments demonstrate that our approach works well even when the item space is large and that it scales to environments with item spaces different from those used in training.Comment: Accepted to IJCAI 2019,14 pages,8 figure

The Generation of Human Induced Pluripotent Stem Cells from Blood Cells: An Efficient Protocol Using Serial Plating of Reprogrammed Cells by Centrifugation

Human induced pluripotent stem cells (hiPSCs) have demonstrated great potential for differentiation into diverse tissues. We report a straightforward and highly efficient method for the generation of iPSCs from PBMCs. By plating the cells serially to a newly coated plate by centrifugation, this protocol provides multiple healthy iPSC colonies even from a small number of PBMCs. The generated iPSCs expressed pluripotent markers and differentiated into all three germ layer lineages. The protocol can also be used with umbilical cord blood mononuclear cells (CBMCs). In this study, we present a simple and efficient protocol that improved the yield of iPSCs from floating cells such as PBMCs and CBMCs by serial plating and centrifugation

Solving continual combinatorial selection via deep reinforcement learning

We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present a deep RL algorithm to solve this issue by adopting the following key ideas. First, we convert the original S-MDP into an Iterative Select-MDP (IS-MDP), which is equivalent to the S-MDP in terms of optimal actions. IS-MDP decomposes a joint action of selecting K items simultaneously into K iterative selections resulting in the decrease of actions at the expense of an exponential increase of states. Second, we overcome this state space explo-sion by exploiting a special symmetry in IS-MDPs with novel weight shared Q-networks, which prov-ably maintain sufficient expressive power. Various experiments demonstrate that our approach works well even when the item space is large and that it scales to environments with item spaces different from those used in training.Comment: Accepted to IJCAI 2019,14 pages,8 figure

Modeling Secondary Iron Overload Cardiomyopathy with Human Induced Pluripotent Stem Cell-Derived Cardiomyocytes.

Excessive iron accumulation in the heart causes iron overload cardiomyopathy (IOC), which initially presents as diastolic dysfunction and arrhythmia but progresses to systolic dysfunction and end-stage heart failure when left untreated. However, the mechanisms of iron-related cardiac injury and how iron accumulates in human cardiomyocytes are not well understood. Herein, using human induced pluripotent stem cell-derived cardiomyocytes (iPSC-CMs), we model IOC and screen for drugs to rescue the iron overload phenotypes. Human iPSC-CMs under excess iron exposure recapitulate early-stage IOC, including oxidative stress, arrhythmia, and contractile dysfunction. We find that iron-induced changes in calcium kinetics play a critical role in dysregulation of CM functions. We identify that ebselen, a selective divalent metal transporter 1 (DMT1) inhibitor and antioxidant, could prevent the observed iron overload phenotypes, supporting the role of DMT1 in iron uptake into the human myocardium. These results suggest that ebselen may be a potential preventive and therapeutic agent for treating patients with secondary iron overload