VPE: Variational Policy Embedding for Transfer Reinforcement Learning

Reinforcement Learning methods are capable of solving complex problems, but resulting policies might perform poorly in environments that are even slightly different. In robotics especially, training and deployment conditions often vary and data collection is expensive, making retraining undesirable. Simulation training allows for feasible training times, but on the other hand suffers from a reality-gap when applied in real-world settings. This raises the need of efficient adaptation of policies acting in new environments. We consider this as a problem of transferring knowledge within a family of similar Markov decision processes. For this purpose we assume that Q-functions are generated by some low-dimensional latent variable. Given such a Q-function, we can find a master policy that can adapt given different values of this latent variable. Our method learns both the generative mapping and an approximate posterior of the latent variables, enabling identification of policies for new tasks by searching only in the latent space, rather than the space of all policies. The low-dimensional space, and master policy found by our method enables policies to quickly adapt to new environments. We demonstrate the method on both a pendulum swing-up task in simulation, and for simulation-to-real transfer on a pushing task

Global Search with Bernoulli Alternation Kernel for Task-oriented Grasping Informed by Simulation

We develop an approach that benefits from large simulated datasets and takes full advantage of the limited online data that is most relevant. We propose a variant of Bayesian optimization that alternates between using informed and uninformed kernels. With this Bernoulli Alternation Kernel we ensure that discrepancies between simulation and reality do not hinder adapting robot control policies online. The proposed approach is applied to a challenging real-world problem of task-oriented grasping with novel objects. Our further contribution is a neural network architecture and training pipeline that use experience from grasping objects in simulation to learn grasp stability scores. We learn task scores from a labeled dataset with a convolutional network, which is used to construct an informed kernel for our variant of Bayesian optimization. Experiments on an ABB Yumi robot with real sensor data demonstrate success of our approach, despite the challenge of fulfilling task requirements and high uncertainty over physical properties of objects.Comment: To appear in 2nd Conference on Robot Learning (CoRL) 201

Surface roughness and height-height correlations dependence on thickness of YBaCuO thin films

For high Tc superconducting multilayer applications, smooth interfaces between the individual layers are required. However, in general, e.g., YBaCuO grows in a 3D screw-dislocation or island nucleation growth mode, introducing a surface roughness. In this contribution we study the surface layer roughness as a function of different deposition techniques as well as deposition parameters. Special attention will be paid to the increase in film roughness with increasing film thickness. For these studies we used scanning probe microscopy. From these experiments, we obtained an island density decreasing with a square root dependence on the film thickness. Furthermore, height-height correlations indicate that the film growth can be described by a ballistic growth process, with very limited effective surface diffusion. The correlation lengths ¿ are on the order of the island size, inferring that the island size forms the mean diffusion barrier. This results in a representation of non-correlated islands, which can be considered as autonomous systems

A Simulated Microgravity Environment Causes a Sustained Defect in Epithelial Barrier Function.

Intestinal epithelial cell (IEC) junctions constitute a robust barrier to invasion by viruses, bacteria and exposure to ingested agents. Previous studies showed that microgravity compromises the human immune system and increases enteropathogen virulence. However, the effects of microgravity on epithelial barrier function are poorly understood. The aims of this study were to identify if simulated microgravity alters intestinal epithelial barrier function (permeability), and susceptibility to barrier-disrupting agents. IECs (HT-29.cl19a) were cultured on microcarrier beads in simulated microgravity using a rotating wall vessel (RWV) for 18 days prior to seeding on semipermeable supports to measure ion flux (transepithelial electrical resistance (TER)) and FITC-dextran (FD4) permeability over 14 days. RWV cells showed delayed apical junction localization of the tight junction proteins, occludin and ZO-1. The alcohol metabolite, acetaldehyde, significantly decreased TER and reduced junctional ZO-1 localization, while increasing FD4 permeability in RWV cells compared with static, motion and flask control cells. In conclusion, simulated microgravity induced an underlying and sustained susceptibility to epithelial barrier disruption upon removal from the microgravity environment. This has implications for gastrointestinal homeostasis of astronauts in space, as well as their capability to withstand the effects of agents that compromise intestinal epithelial barrier function following return to Earth

Co-transcriptional R-loops are the main cause of estrogen-induced DNA damage.

The hormone estrogen (E2) binds the estrogen receptor to promote transcription of E2-responsive genes in the breast and other tissues. E2 also has links to genomic instability, and elevated E2 levels are tied to breast cancer. Here, we show that E2 stimulation causes a rapid, global increase in the formation of R-loops, co-transcriptional RNA-DNA products, which in some instances have been linked to DNA damage. We show that E2-dependent R-loop formation and breast cancer rearrangements are highly enriched at E2-responsive genomic loci and that E2 induces DNA replication-dependent double-strand breaks (DSBs). Strikingly, many DSBs that accumulate in response to E2 are R-loop dependent. Thus, R-loops resulting from the E2 transcriptional response are a significant source of DNA damage. This work reveals a novel mechanism by which E2 stimulation leads to genomic instability and highlights how transcriptional programs play an important role in shaping the genomic landscape of DNA damage susceptibility

A new Taxonomy of Continuous Global Optimization Algorithms

Surrogate-based optimization, nature-inspired metaheuristics, and hybrid combinations have become state of the art in algorithm design for solving real-world optimization problems. Still, it is difficult for practitioners to get an overview that explains their advantages in comparison to a large number of available methods in the scope of optimization. Available taxonomies lack the embedding of current approaches in the larger context of this broad field. This article presents a taxonomy of the field, which explores and matches algorithm strategies by extracting similarities and differences in their search strategies. A particular focus lies on algorithms using surrogates, nature-inspired designs, and those created by design optimization. The extracted features of components or operators allow us to create a set of classification indicators to distinguish between a small number of classes. The features allow a deeper understanding of components of the search strategies and further indicate the close connections between the different algorithm designs. We present intuitive analogies to explain the basic principles of the search algorithms, particularly useful for novices in this research field. Furthermore, this taxonomy allows recommendations for the applicability of the corresponding algorithms.Comment: 35 pages total, 28 written pages, 4 figures, 2019 Reworked Versio