Leveraging Domain Knowledge for Reinforcement Learning using MMC Architectures

Despite the success of reinforcement learning methods in various simulated robotic applications, end-to-end training suffers from extensive training times due to high sample complexity and does not scale well to realistic systems. In this work, we speed up reinforcement learning by incorporating domain knowledge into policy learning. We revisit an architecture based on the mean of multiple computations (MMC) principle known from computational biology and adapt it to solve a reacher task. We approximate the policy using a simple MMC network, experimentally compare this idea to end-to-end deep learning architectures, and show that our approach reduces the number of interactions required to approximate a suitable policy by a factor of ten

Ex Vivo Optimization of Donor Lungs with Inhaled Sevoflurane during Normothermic Ex Vivo Lung Perfusion (VITALISE): A Pilot and Feasibility Study in Sheep

Volatile anesthetics have been shown in different studies to reduce ischemia reperfusion injury (IRI). Ex vivo lung perfusion (EVLP) facilitates graft evaluation, extends preservation time and potentially enables injury repair and improvement of lung quality. We hypothesized that ventilating lungs with sevoflurane during EVLP would reduce lung injury and improve lung function. We performed a pilot study to test this hypothesis in a slaughterhouse sheep DCD model. Lungs were harvested, flushed and stored on ice for 3 h, after which EVLP was performed for 4 h. Lungs were ventilated with either an FiO2 of 0.4 (EVLP, n = 5) or FiO2 of 0.4 plus sevoflurane at a 2% end-tidal concentration (Cet) (S-EVLP, n = 5). Perfusate, tissue samples and functional measurements were collected and analyzed. A steady state of the target Cet sevoflurane was reached with measurable concentrations in perfusate. Lungs in the S-EVLP group showed significantly better dynamic lung compliance than those in the EVLP group (p = 0.003). Oxygenation capacity was not different in treated lungs for delta partial oxygen pressure (PO2; +3.8 (−4.9/11.1) vs. −11.7 (−12.0/−3.2) kPa, p = 0.151), but there was a trend of a better PO2/FiO2 ratio (p = 0.054). Perfusate ASAT levels in S-EVLP were significantly reduced compared to the control group (198.1 ± 93.66 vs. 223.9 ± 105.7 IU/L, p = 0.02). We conclude that ventilating lungs with sevoflurane during EVLP is feasible and could be useful to improve graft function

Elements for a general memory structure: properties of recurrent neural networks used to form situation models

Makarov VA, Song Y, Velarde MG, Hübner D, Cruse H. Elements for a general memory structure: properties of recurrent neural networks used to form situation models. Biological Cybernetics. 2008;98(5):371-395.We study how individual memory items are stored assuming that situations given in the environment can be represented in the form of synaptic-like couplings in recurrent neural networks. Previous numerical investigations have shown that specific architectures based on suppression or max units can successfully learn static or dynamic stimuli (situations). Here we provide a theoretical basis concerning the learning process convergence and the network response to a novel stimulus. We show that, besides learning "simple" static situations, a nD network can learn and replicate a sequence of up to n different vectors or frames. We find limits on the learning rate and show coupling matrices developing during training in different cases including expansion of the network into the case of nonlinear interunit coupling. Furthermore, we show that a specific coupling matrix provides low-pass-filter properties to the units, thus connecting networks constructed by static summation units with continuous-time networks. We also show under which conditions such networks can be used to perform arithmetic calculations by means of pattern completion

Characterization of the Hepatitis C Virus NS2/3 Processing Reaction by Using a Purified Precursor Protein

The NS2-NS3 region of the hepatitis C virus polyprotein encodes a proteolytic activity that is required for processing of the NS2/3 junction. Membrane association of NS2 and the autocatalytic nature of the NS2/3 processing event have so far constituted hurdles to the detailed investigation of this reaction. We now report the first biochemical characterization of the self-processing activity of a purified NS2/3 precursor. Using multiple sequence alignments, we were able to define a minimal domain, devoid of membrane-anchoring sequences, which was still capable of performing the processing reaction. This truncated protein was efficiently expressed and processed in Escherichia coli. The processing reaction could be significantly suppressed by growth in minimal medium in the absence of added zinc ions, leading to the accumulation of an unprocessed precursor protein in inclusion bodies. This protein was purified to homogeneity, refolded, and shown to undergo processing at the authentic NS2/NS3 cleavage site with rates comparable to those observed using an in vitro-translated full-length NS2/3 precursor. Size-exclusion chromatography and a dependence of the processing rate on the concentration of truncated NS2/3 suggested a functional multimerization of the precursor protein. However, we were unable to observe trans cleavage activity between cleavage-site mutants and active-site mutants. Furthermore, the cleavage reaction of the wild-type protein was not inhibited by addition of a mutant that was unable to undergo self-processing. Site-directed mutagenesis data and the independence of the processing rate from the nature of the added metal ion argue in favor of NS2/3 being a cysteine protease having Cys993 and His952 as a catalytic dyad. We conclude that a purified protein can efficiently reproduce processing at the NS2/3 site in the absence of additional cofactors