The kernel Kalman rule: efficient nonparametric inference with recursive least squares

Nonparametric inference techniques provide promising tools for probabilistic reasoning in high-dimensional nonlinear systems. Most of these techniques embed distributions into reproducing kernel Hilbert spaces (RKHS) and rely on the kernel Bayes’ rule (KBR) to manipulate the embeddings. However, the computational demands of the KBR scale poorly with the number of samples and the KBR often suffers from numerical instabilities. In this paper, we present the kernel Kalman rule (KKR) as an alternative to the KBR. The derivation of the KKR is based on recursive least squares, inspired by the derivation of the Kalman innovation update. We apply the KKR to filtering tasks where we use RKHS embeddings to represent the belief state, resulting in the kernel Kalman filter (KKF). We show on a nonlinear state estimation task with high dimensional observations that our approach provides a significantly improved estimation accuracy while the computational demands are significantly decreased

Model-based contextual policy search for data-efficient generalization of robot skills

In robotics, lower-level controllers are typically used to make the robot solve a specific task in a fixed context. For example, the lower-level controller can encode a hitting movement while the context defines the target coordinates to hit. However, in many learning problems the context may change between task executions. To adapt the policy to a new context, we utilize a hierarchical approach by learning an upper-level policy that generalizes the lower-level controllers to new contexts. A common approach to learn such upper-level policies is to use policy search. However, the majority of current contextual policy search approaches are model-free and require a high number of interactions with the robot and its environment. Model-based approaches are known to significantly reduce the amount of robot experiments, however, current model-based techniques cannot be applied straightforwardly to the problem of learning contextual upper-level policies. They rely on specific parametrizations of the policy and the reward function, which are often unrealistic in the contextual policy search formulation. In this paper, we propose a novel model-based contextual policy search algorithm that is able to generalize lower-level controllers, and is data-efficient. Our approach is based on learned probabilistic forward models and information theoretic policy search. Unlike current algorithms, our method does not require any assumption on the parametrization of the policy or the reward function. We show on complex simulated robotic tasks and in a real robot experiment that the proposed learning framework speeds up the learning process by up to two orders of magnitude in comparison to existing methods, while learning high quality policies

Data-Efficient Generalization of Robot Skills with Contextual Policy Search

In robotics, controllers make the robot solve a task within a specific context. The context can describe the objectives of the robot or physical properties of the environment and is always specified before task execution. To generalize the controller to multiple contexts, we follow a hierarchical approach for policy learning: A lower-level policy controls the robot for a given context and an upper-level policy generalizes among contexts. Current approaches for learning such upper-level policies are based on model-free policy search, which require an excessive number of interactions of the robot with its environment. More data-efficient policy search approaches are model based but, thus far, without the capability of learning hierarchical policies. We propose a new model-based policy search approach that can also learn contextual upper-level policies. Our approach is based on learning probabilistic forward models for long-term predictions. Using these predictions, we use information-theoretic insights to improve the upper-level policy. Our method achieves a substantial improvement in learning speed compared to existing methods on simulated and real robotic tasks. Copyright © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved

Generalizing movements with information-theoretic stochastic optimal control

Stochastic optimal control is typically used to plan a movement for a specific situation. Although most stochastic optimal control methods fail to generalize this movement plan to a new situation without replanning, a stochastic optimal control method is presented that allows reuse of the obtained policy in a new situation, as the policy is more robust to slight deviations from the initial movement plan. To improve the robustness of the policy, we employ information-theoretic policy updates that explicitly operate on trajectory distributions instead of single trajectories. To ensure a stable and smooth policy update, the ”distance” is limited between the trajectory distributions of the old and the new control policies. The introduced bound offers a closed-form solution for the resulting policy and extends results from recent developments in stochastic optimal control. In contrast to many standard stochastic optimal control algorithms, the current approach can directly infer the system dynamics from data points, and hence can also be used for model-based reinforcement learning. This paper represents an extension of the paper by Lioutikov et al. (“Sample-Based Information-Theoretic Stochastic Optimal Control,” Proceedings of 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, Piscataway, NJ, 2014, pp. 3896–3902). In addition to revisiting the content, an extensive theoretical comparison is presented of the approach with related work, additional aspects of the implementation are discussed, and further evaluations are introduced

Fibroblast Growth Factor-2 Primes Human Mesenchymal Stem Cells for Enhanced Chondrogenesis

Human mesenchymal stem cells (hMSCs) are multipotent cells capable of differentiating into a variety of mature cell types, including osteoblasts, adipocytes and chondrocytes. It has previously been shown that, when expanded in medium supplemented with fibroblast growth factor-2 (FGF-2), hMSCs show enhanced chondrogenesis (CG). Previous work concluded that the enhancement of CG could be attributed to the selection of a cell subpopulation with inherent chondrogenic potential. In this study, we show that FGF-2 pretreatment actually primed hMSCs to undergo enhanced CG by increasing basal Sox9 protein levels. Our results show that Sox9 protein levels were elevated within 30 minutes of exposure to FGF-2 and progressively increased with longer exposures. Further, we show using flow cytometry that FGF-2 increased Sox9 protein levels per cell in proliferating and non-proliferating hMSCs, strongly suggesting that FGF-2 primes hMSCs for subsequent CG by regulating Sox9. Indeed, when hMSCs were exposed to FGF-2 for 2 hours and subsequently differentiated into the chondrogenic lineage using pellet culture, phosphorylated-Sox9 (pSox9) protein levels became elevated and ultimately resulted in an enhancement of CG. However, small interfering RNA (siRNA)-mediated knockdown of Sox9 during hMSC expansion was unable to negate the prochondrogenic effects of FGF-2, suggesting that the FGF-2-mediated enhancement of hMSC CG is only partly regulated through Sox9. Our findings provide new insights into the mechanism by which FGF-2 regulates predifferentiation hMSCs to undergo enhanced CG

Mitochondria-Specific Accumulation of Amyloid β Induces Mitochondrial Dysfunction Leading to Apoptotic Cell Death

Mitochondria are best known as the essential intracellular organelles that host the homeostasis required for cellular survival, but they also have relevance in diverse disease-related conditions, including Alzheimer's disease (AD). Amyloid β (Aβ) peptide is the key molecule in AD pathogenesis, and has been highlighted in the implication of mitochondrial abnormality during the disease progress. Neuronal exposure to Aβ impairs mitochondrial dynamics and function. Furthermore, mitochondrial Aβ accumulation has been detected in the AD brain. However, the underlying mechanism of how Aβ affects mitochondrial function remains uncertain, and it is questionable whether mitochondrial Aβ accumulation followed by mitochondrial dysfunction leads directly to neuronal toxicity. This study demonstrated that an exogenous Aβ1–42 treatment, when applied to the hippocampal cell line of mice (specifically HT22 cells), caused a deleterious alteration in mitochondria in both morphology and function. A clathrin-mediated endocytosis blocker rescued the exogenous Aβ1–42-mediated mitochondrial dysfunction. Furthermore, the mitochondria-targeted accumulation of Aβ1–42 in HT22 cells using Aβ1–42 with a mitochondria-targeting sequence induced the identical morphological alteration of mitochondria as that observed in the APP/PS AD mouse model and exogenous Aβ1–42-treated HT22 cells. In addition, subsequent mitochondrial dysfunctions were demonstrated in the mitochondria-specific Aβ1–42 accumulation model, which proved indistinguishable from the mitochondrial impairment induced by exogenous Aβ1–42-treated HT22 cells. Finally, cellular toxicity was directly induced by mitochondria-targeted Aβ1–42 accumulation, which mimics the apoptosis process in exogenous Aβ1–42-treated HT22 cells. Taken together, these results indicate that mitochondria-targeted Aβ1–42 accumulation is the necessary and sufficient condition for Aβ-mediated mitochondria impairments, and leads directly to cellular death rather than along with other Aβ-mediated signaling alterations

Modulating gradients in regulatory signals within mesenchymal stem cell seeded hydrogels: a novel strategy to engineer zonal articular cartilage.

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.Engineering organs and tissues with the spatial composition and organisation of their native equivalents remains a major challenge. One approach to engineer such spatial complexity is to recapitulate the gradients in regulatory signals that during development and maturation are believed to drive spatial changes in stem cell differentiation. Mesenchymal stem cell (MSC) differentiation is known to be influenced by both soluble factors and mechanical cues present in the local microenvironment. The objective of this study was to engineer a cartilaginous tissue with a native zonal composition by modulating both the oxygen tension and mechanical environment thorough the depth of MSC seeded hydrogels. To this end, constructs were radially confined to half their thickness and subjected to dynamic compression (DC). Confinement reduced oxygen levels in the bottom of the construct and with the application of DC, increased strains across the top of the construct. These spatial changes correlated with increased glycosaminoglycan accumulation in the bottom of constructs, increased collagen accumulation in the top of constructs, and a suppression of hypertrophy and calcification throughout the construct. Matrix accumulation increased for higher hydrogel cell seeding densities; with DC further enhancing both glycosaminoglycan accumulation and construct stiffness. The combination of spatial confinement and DC was also found to increase proteoglycan-4 (lubricin) deposition toward the top surface of these tissues. In conclusion, by modulating the environment through the depth of developing constructs, it is possible to suppress MSC endochondral progression and to engineer tissues with zonal gradients mimicking certain aspects of articular cartilage.Funding was provided by Science Foundation Ireland (President of Ireland Young Researcher Award: 08/Y15/B1336) and the European Research Council (StemRepair – Project number 258463)