Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discovery

Robot assembly discovery is a challenging problem that lives at the intersection of resource allocation and motion planning. The goal is to combine a predefined set of objects to form something new while considering task execution with the robot-in-the-loop. In this work, we tackle the problem of building arbitrary, predefined target structures entirely from scratch using a set of Tetris-like building blocks and a robotic manipulator. Our novel hierarchical approach aims at efficiently decomposing the overall task into three feasible levels that benefit mutually from each other. On the high level, we run a classical mixed-integer program for global optimization of block-type selection and the blocks' final poses to recreate the desired shape. Its output is then exploited to efficiently guide the exploration of an underlying reinforcement learning (RL) policy. This RL policy draws its generalization properties from a flexible graph-based representation that is learned through Q-learning and can be refined with search. Moreover, it accounts for the necessary conditions of structural stability and robotic feasibility that cannot be effectively reflected in the previous layer. Lastly, a grasp and motion planner transforms the desired assembly commands into robot joint movements. We demonstrate our proposed method's performance on a set of competitive simulated RAD environments, showcase real-world transfer, and report performance and robustness gains compared to an unstructured end-to-end approach. Videos are available at https://sites.google.com/view/rl-meets-milp

Robust Reinforcement Learning: A Review of Foundations and Recent Advances

Reinforcement learning (RL) has become a highly successful framework for learning in Markov decision processes (MDP). Due to the adoption of RL in realistic and complex environments, solution robustness becomes an increasingly important aspect of RL deployment. Nevertheless, current RL algorithms struggle with robustness to uncertainty, disturbances, or structural changes in the environment. We survey the literature on robust approaches to reinforcement learning and categorize these methods in four different ways: (i) Transition robust designs account for uncertainties in the system dynamics by manipulating the transition probabilities between states; (ii) Disturbance robust designs leverage external forces to model uncertainty in the system behavior; (iii) Action robust designs redirect transitions of the system by corrupting an agent’s output; (iv) Observation robust designs exploit or distort the perceived system state of the policy. Each of these robust designs alters a different aspect of the MDP. Additionally, we address the connection of robustness to the risk-based and entropy-regularized RL formulations. The resulting survey covers all fundamental concepts underlying the approaches to robust reinforcement learning and their recent advances

Local Online Motor Babbling: Learning Motor Abundance of a Musculoskeletal Robot Arm

Motor babbling and goal babbling has been used for sensorimotor learning of highly redundant systems in soft robotics. Recent works in goal babbling have demonstrated successful learning of inverse kinematics (IK) on such systems, and suggest that babbling in the goal space better resolves motor redundancy by learning as few yet efficient sensorimotor mappings as possible. However, for musculoskeletal robot systems, motor redundancy can provide useful information to explain muscle activation patterns, thus the term motor abundance. In this work, we introduce some simple heuristics to empirically define the unknown goal space, and learn the IK of a 10 DoF musculoskeletal robot arm using directed goal babbling. We then further propose local online motor babbling guided by Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which bootstraps on the goal babbling samples for initialization, such that motor abundance can be queried online for any static goal. Our approach leverages the resolving of redundancies and the efficient guided exploration of motor abundance in two stages of learning, allowing both kinematic accuracy and motor variability at the queried goal. The result shows that local online motor babbling guided by CMA-ES can efficiently explore motor abundance at queried goal positions on a musculoskeletal robot system and gives useful insights in terms of muscle stiffness and synergy.IEEE/RSJ International Conference on Intelligent Robots and Systems (iROS2019), November 4 - 8, 2019, Macau, Chin

Approaches to monitor and evaluate OER policies in higher education : tracing developments in Germany, Austria, and Switzerland

The 2019 UNESCO recommendation on Open Educational Resources (OER) encourages member states to monitor policies and mechanisms in OER across the world. In higher education, there are many initiatives and policies around OER. This contribution gives insights into the current situation concerning OER policy documents that are of national or institutional relevance for public higher education institutions in Germany, Switzerland, and Austria. For each country, a different approach for identifying OER policy documents was chosen, dependent on the availability of documents and different dominant forms of documentation. Whereas digital documents available on the web were found as helpful sources for Germany, and performance agreements between the national ministry and individual universities were used for analysis in Austria, a survey amongst all universities was the chosen research approach in Switzerland to give an overview about potentially OER related policy documents. All these documents are now made available via the OER World Map. With this contribution, the authors also highlight the possibility of using the OER World Map as a powerful tool to collect and evaluate OER policy documents

Approaches to monitor and evaluate OER policies in higher education : tracing developments in Germany, Austria, and Switzerland

The 2019 UNESCO recommendation on Open Educational Resources (OER) encourages member states to monitor policies and mechanisms in OER across the world. In higher education, there are many initiatives and policies around OER. This contribution gives insights into the current situation concerning OER policy documents that are of national or institutional relevance for public higher education institutions in Germany, Switzerland, and Austria. For each country, a different approach for identifying OER policy documents was chosen, dependent on the availability of documents and different dominant forms of documentation. Whereas digital documents available on the web were found as helpful sources for Germany, and performance agreements between the national ministry and individual universities were used for analysis in Austria, a survey amongst all universities was the chosen research approach in Switzerland to give an overview about potentially OER related policy documents. All these documents are now made available via the OER World Map. With this contribution, the authors also highlight the possibility of using the OER World Map as a powerful tool to collect and evaluate OER policy documents

Tacaribe Virus but Not Junin Virus Infection Induces Cytokine Release from Primary Human Monocytes and Macrophages

The mechanisms underlying the development of disease during arenavirus infection are poorly understood. However, common to all hemorrhagic fever diseases is the involvement of macrophages as primary target cells, suggesting that the immune response in these cells may be of paramount importance during infection. Thus, in order to identify features of the immune response that contribute to arenavirus pathogenesis, we have examined the growth kinetics and cytokine profiles of two closely related New World arenaviruses, the apathogenic Tacaribe virus (TCRV) and the hemorrhagic fever-causing Junin virus (JUNV), in primary human monocytes and macrophages. Both viruses grew robustly in VeroE6 cells; however, TCRV titres were decreased by approximately 10 fold compared to JUNV in both monocytes and macrophages. Infection of both monocytes and macrophages with TCRV also resulted in the release of high levels of IL-6, IL-10 and TNF-α, while levels of IFN-α, IFN-β and IL-12 were not affected. However, we could show that the presence of these cytokines had no direct effect on growth of either TCRV of JUNV in macrophages. Further analysis also showed that while the production of IL-6 and IL-10 are dependent on viral replication, production of TNF-α also occurs after exposure to UV-inactivated TCRV particles and is thus independent of productive virus infection. Surprisingly, JUNV infection did not have an effect on any of the cytokines examined indicating that, in contrast to other viral hemorrhagic fever viruses, macrophage-derived cytokine production is unlikely to play an active role in contributing to the cytokine dysregulation observed in JUNV infected patients. Rather, these results suggest that an early, controlled immune response by infected macrophages may be critical for the successful control of infection of apathogenic viruses and prevention of subsequent disease, including systemic cytokine dysregulation

Salinity variations in the northern Coorong Lagoon, South Australia: Significant changes in the ecosystem following human alteration to the natural water regime

European settlement and drought have significantly impacted the hydrology of the Coorong, a shallow coastal lagoon complex in South Australia, which is part of a terminal wetland at the mouth of the River Murray. An increased salinity associated with lower water levels and progressive isolation from ocean flushes contributed to a severe decline in ecological diversity over the past decades. Here we have conducted a molecular and stable isotopic study of a sedimentary core from the northern Coorong Lagoon spanning more than 5000 years to investigate the recent palaeoenvironmental history of the ecosystem. Major alterations were evident in many biogeochemical parameters in sediments deposited after the 1950s coinciding with the beginning of intensified water regulations. The most prominent shift occurred in δ13C profiles of C21–C33n-alkanes from average values of −23.5‰ to an average of −28.2‰.Further changes included decreases in carbon preference index (CPI) and average chain length (ACL) of the n-alkane series as well as significant increases in algal (e.g. C20 HBI, long chain alkenes and C29-alkadiene) and bacterial (e.g. 13C depleted short chain n-alkanes and hopanoids, δ13C: −35.9‰ to −30.1‰) derived hydrocarbons. Long chain n-alkanes with a strong odd/even predominance as observed here are typically attributed to terrigenous plants. In the Coorong however, terrigenous input to sedimentary OM is only minor. Therefore changes in the before mentioned parameters were attributed to a source transition from a major contribution of macrophytes towards predominantly microalgae and bacteria.δD values of C21–C33n-alkanes showed a general trend towards more enriched values in younger sediments, indicating an overall rising salinity. However, the most pronounced positive shift in these profiles again occurred after the 1950s. Altogether this study demonstrates that the recent human induced changes of the Coorong hydrology, compounded by a severe drought led to an increase in salinity and alterations of primary production which have been much more significant than natural variations occurring throughout the Holocene over several thousands of years