Hierarchical relative entropy policy search

Many reinforcement learning (RL) tasks, especially in robotics, consist of multiple sub-tasks that are strongly structured. Such task structures can be exploited by incorporating hierarchical policies that consist of gating networks and sub-policies. However, this concept has only been partially explored for real world settings and complete methods, derived from first principles, are needed. Real world settings are challenging due to large and continuous state-action spaces that are prohibitive for exhaustive sampling methods. We define the problem of learning sub-policies in continuous state action spaces as finding a hierarchical policy that is composed of a high-level gating policy to select the low-level sub-policies for execution by the agent. In order to efficiently share experience with all sub-policies, also called inter-policy learning, we treat these sub-policies as latent variables which allows for distribution of the update information between the sub-policies. We present three different variants of our algorithm, designed to be suitable for a wide variety of real world robot learning tasks and evaluate our algorithms in two real robot learning scenarios as well as several simulations and comparisons

Learning sequential motor tasks

Many real robot applications require the sequential use of multiple distinct motor primitives. This requirement implies the need to learn the individual primitives as well as a strategy to select the primitives sequentially. Such hierarchical learning problems are commonly either treated as one complex monolithic problem which is hard to learn, or as separate tasks learned in isolation. However, there exists a strong link between the robots strategy and its motor primitives. Consequently, a consistent framework is needed that can learn jointly on the level of the individual primitives and the robots strategy. We present a hierarchical learning method which improves individual motor primitives and, simultaneously, learns how to combine these motor primitives sequentially to solve complex motor tasks. We evaluate our method on the game of robot hockey, which is both difficult to learn in terms of the required motor primitives as well as its strategic elements

Learning to predict phases of manipulation tasks as hidden states

Phase transitions in manipulation tasks often occur when contacts between objects are made or broken. A switch of the phase can result in the robot’s actions suddenly influencing different aspects of its environment. Therefore, the boundaries between phases often correspond to constraints or subgoals of the manipulation task. In this paper, we investigate how the phases of manipulation tasks can be learned from data. The task is modeled as an autoregressive hidden Markov model, wherein the hidden phase transitions depend on the observed states. The model is learned from data using the expectation-maximization algorithm. We demonstrate the proposed method on both a pushing task and a pepper mill turning task. The proposed approach was compared to a standard autoregressive hidden Markov model. The experiments show that the learned models can accurately predict the transitions in phases during the manipulation tasks

Anodic respiration of Pseudomonas putida KT2440 in a stirred-tank bioreactor

Anodic batch production of para-hydroxybenzoic acid (pHBA) from citric acid with a genetically modified Pseudomonas putida KT2440 strain was studied in a bio-electrochemical system (BES) based on a standard lab-scale stirred-tank bioreactor at fully controlled anaerobic reaction conditions. Electron transfer to the anode was mediated by addition of KFe(CN) to the medium. Effects of varying anode surface areas (graphite rod, felt and brush), power input (stirrer speed) and mediator concentrations were investigated. The obligate aerobic P. putida grew anaerobically with mediated anodic respiration and pHBA production was observed. Anodic respiration was best applying the graphite rod electrode which showed a maximal current density of 12.5 mA cm. This is the highest measured for non-porous electrodes in BES until now. Increasing the power input to 2.93 W L (700 rpm) and online control of the redox potential E at 225 mV (vs. Ag/AgCl) in the medium by controlled addition of mediator resulted in a maximal pHBA yield of 9.91 mmolC molC which exceeds pHBA yields in the aerobic batch process by 69 % (5.87 mmolC molC )

Interaction primitives for human-robot cooperation tasks

To engage in cooperative activities with human partners, robots have to possess basic interactive abilities and skills. However, programming such interactive skills is a challenging task, as each interaction partner can have different timing or an alternative way of executing movements. In this paper, we propose to learn interaction skills by observing how two humans engage in a similar task. To this end, we introduce a new representation called Interaction Primitives. Interaction primitives build on the framework of dynamic motor primitives (DMPs) by maintaining a distribution over the parameters of the DMP. With this distribution, we can learn the inherent correlations of cooperative activities which allow us to infer the behavior of the partner and to participate in the cooperation. We will provide algorithms for synchronizing and adapting the behavior of humans and robots during joint physical activities

Synthetic induction of immunogenic cell death by genetic stimulation of endoplasmic reticulum stress.

Cis-diamminedichloridoplatinum(II) (CDDP), commonly referred to as cisplatin, is a chemotherapeutic drug used for the treatment of a wide range of solid cancers. CDDP is a relatively poor inducer of immunogenic cell death (ICD), a cell death modality that converts dying cells into a tumor vaccine, stimulating an immune response against residual cancer cells that permits long-lasting immunity and a corresponding reduction in tumor growth. The incapacity of CDDP to trigger ICD is at least partially due to its failure to stimulate the premortem endoplasmic reticulum (ER)-stress response required for the externalization of the "eat-me" signal calreticulin (CRT) on the surface of dying cancer cells. Here, we developed a murine cancer cell line genetically modified to express the ER resident protein reticulon-1c (Rtn-1c) by virtue of tetracycline induction and showed that enforced Rtn-1c expression combined with CDDP treatment promoted CRT externalization to the surface of cancer cells. In contrast to single agent treatments, the tetracycline-mediated Rtn-1c induction combined with CDDP chemotherapy stimulated ICD as measured by the capacity of dying tumor cells, inoculated into syngenic immunocompetent mice, to mount an immune response to tumor re-challenge 1 week later. More importantly, established tumors, forced to constitutively express Rtn-1c in vivo by continuous treatment with tetracycline, became responsive to CDDP and exhibited a corresponding reduction in the rate of tumor growth. The combined therapeutic effects of Rtn-1c induction with CDDP treatment was only detected in the context of an intact immune system and not in nu/nu mice lacking thymus-dependent T lymphocytes. Altogether, these results indicate that the artificial or "synthetic" induction of immunogenic cell death by genetic manipulation of the ER-stress response can improve the efficacy of chemotherapy with CDDP by stimulating anticancer immunity

Trial Watch: experimental TLR7/TLR8 agonists for oncological indications

Resiquimod (R848) and motolimod (VTX-2337) are second-generation experimental derivatives of imiquimod, an imidazoquinoline with immunostimulatory properties originally approved by the US Food and Drug Administration for the topical treatment of actinic keratosis and genital warts more than 20 years ago. Both resiquimod and motolimod operate as agonists of Toll-like receptor 7 (TLR7) and/or TLR8, in thus far delivering adjuvant-like signals to antigen-presenting cells (APCs). In line with such an activity, these compounds are currently investigated as immunostimulatory agents for the treatment of various malignancies, especially in combination with peptide-based, dendritic cell-based, cancer cell lysate-based, or DNA-based vaccines. Here, we summarize preclinical and clinical evidence recently collected to support the development of resiquimod and motolimod and other TLR7/TLR8 agonists as anticancer agents

A long non-coding RNA links calreticulin-mediated immunogenic cell removal to RB1 transcription

A subset of promoters bidirectionally expresses long non-coding RNAs (ncRNAs) of unknown function and protein-coding genes (PCGs) in parallel. Here, we define a set of 1107 highly conserved human bidirectional promoters that mediate the linked expression of long ncRNAs and PCGs. Depletion of the long ncRNA expressed from the RB1 promoter, ncRNA-RB1, reveals regulatory effects different from the RB1-controlled transcriptional program. ncRNA-RB1 positively regulates the expression of calreticulin (CALR) that in response to certain therapeutic interventions can translocate from the endoplasmic reticulum to the cell surface, hence activating anticancer immune responses. Knockdown of ncRNA-RB1 in tumor cells reduced expression of CALR, impaired the translocation of the protein to the cell surface upon treatment with anthracylines and consequently inhibited the cellular uptake by macrophages. In conclusion, co-transcription of ncRNA-RB1 and RB1 provides a positive link between the expression of the two tumor suppressors RB1 and the immune-relevant CALR protein. This regulatory interplay exemplifies disease-relevant co-regulation of two distinct gene products, in which loss of expression of one oncosuppressor protein entails the abolition of additional tumor-inhibitory mechanisms

Bim and Bmf Synergize To Induce Apoptosis in Neisseria Gonorrhoeae Infection

Bcl-2 family proteins including the pro-apoptotic BH3-only proteins are central regulators of apoptotic cell death. Here we show by a focused siRNA miniscreen that the synergistic action of the BH3-only proteins Bim and Bmf is required for apoptosis induced by infection with Neisseria gonorrhoeae (Ngo). While Bim and Bmf were associated with the cytoskeleton of healthy cells, they both were released upon Ngo infection. Loss of Bim and Bmf from the cytoskeleton fraction required the activation of Jun-N-terminal kinase-1 (JNK-1), which in turn depended on Rac-1. Depletion and inhibition of Rac-1, JNK-1, Bim, or Bmf prevented the activation of Bak and Bax and the subsequent activation of caspases. Apoptosis could be reconstituted in Bim-depleted and Bmf-depleted cells by additional silencing of antiapoptotic Mcl-1 and Bcl-X-L, respectively. Our data indicate a synergistic role for both cytoskeletal-associated BH3-only proteins, Bim, and Bmf, in an apoptotic pathway leading to the clearance of Ngo-infected cells

Newton's law for Bloch electrons, Klein factors and deviations from canonical commutation relations

The acceleration theorem for Bloch electrons in a homogenous external field is usually presented using quasiclassical arguments. In quantum mechanical versions the Heisenberg equations of motion for an operator $\hat {\vec k}(t)$ are presented mostly without properly defining this operator. This leads to the surprising fact that the generally accepted version of the theorem is incorrect for the most natural definition of $\hat {\vec k}$. This operator is shown not to obey canonical commutation relations with the position operator. A similar result is shown for the phase operators defined via the Klein factors which take care of the change of particle number in the bosonization of the field operator in the description of interacting fermions in one dimension. The phase operators are also shown not to obey canonical commutation relations with the corresponding particle number operators. Implications of this fact are discussed for Tomonaga-Luttinger type models.Comment: 9 pages,1 figur