Reset-free Trial-and-Error Learning for Robot Damage Recovery

The high probability of hardware failures prevents many advanced robots (e.g., legged robots) from being confidently deployed in real-world situations (e.g., post-disaster rescue). Instead of attempting to diagnose the failures, robots could adapt by trial-and-error in order to be able to complete their tasks. In this situation, damage recovery can be seen as a Reinforcement Learning (RL) problem. However, the best RL algorithms for robotics require the robot and the environment to be reset to an initial state after each episode, that is, the robot is not learning autonomously. In addition, most of the RL methods for robotics do not scale well with complex robots (e.g., walking robots) and either cannot be used at all or take too long to converge to a solution (e.g., hours of learning). In this paper, we introduce a novel learning algorithm called "Reset-free Trial-and-Error" (RTE) that (1) breaks the complexity by pre-generating hundreds of possible behaviors with a dynamics simulator of the intact robot, and (2) allows complex robots to quickly recover from damage while completing their tasks and taking the environment into account. We evaluate our algorithm on a simulated wheeled robot, a simulated six-legged robot, and a real six-legged walking robot that are damaged in several ways (e.g., a missing leg, a shortened leg, faulty motor, etc.) and whose objective is to reach a sequence of targets in an arena. Our experiments show that the robots can recover most of their locomotion abilities in an environment with obstacles, and without any human intervention.Comment: 18 pages, 16 figures, 3 tables, 6 pseudocodes/algorithms, video at https://youtu.be/IqtyHFrb3BU, code at https://github.com/resibots/chatzilygeroudis_2018_rt

Classification via sequential testing

The problem of generating the sequence of tests required to reach a diagnostic conclusion with minimum average cost, which is also known as test sequencing problem, is considered. The test sequencing problem is formulated as an optimal binary AND/OR decision tree construction problem, whose solution is known to be NP-complete. The problem can be solved optimally using dynamic programming or AND/OR graph search methods (AO*, CF, and HS). However, for large systems, the associated computational effort with dynamic programming or AND/OR graph search methods is substantial, due to the rapidly increasing number of nodes in AND/OR search graph. In order to prevent the computational explosion, one-step or multistep lookahead heuristic algorithms have been developed to solve the test sequencing problem. Our approach is based on integrating concepts from the one-step lookahead heuristic algorithms and the strategies used in Huffman coding. The effectiveness of the algorithms is demonstrated on several test cases. The traditional test sequencing problem is generalized here to include asymmetrical tests. Our approach to test sequencing can be adapted to solve a wide variety of binary identification problems arising in decision table programming, medical diagnosis, database query processing, quality assurance, and pattern recognition

Exact and heuristic approaches to detect failures in failed k-out-of-n systems

This paper considers a k-out-of-n system that has just failed. There is an associated cost of testing each component. In addition, we have apriori information regarding the probabilities that a certain set of components is the reason for the failure. The goal is to identify the subset of components that have caused the failure with the minimum expected cost. In this work, we provide exact and approximate policies that detects components’ states in a failed k-out-of-n system. We propose two integer programming (IP) formulations, two novel Markov decision process (MDP) based approaches, and two heuristic algorithms. We show the limitations of exact algorithms and effectiveness of proposed heuristic approaches on a set of randomly generated test instances. Despite longer CPU times, IP formulations are flexible in incorporating further restrictions such as test precedence relationships, if need be. Numerical results illustrate that dynamic programming for the proposed MDP model is the most effective exact method, solving up to 12 components within one hour. The heuristic algorithms’ performances are presented against exact approaches for small to medium sized instances and against a lower bound for larger instances

Assessment team report on flight-critical systems research at NASA Langley Research Center

The quality, coverage, and distribution of effort of the flight-critical systems research program at NASA Langley Research Center was assessed. Within the scope of the Assessment Team's review, the research program was found to be very sound. All tasks under the current research program were at least partially addressing the industry needs. General recommendations made were to expand the program resources to provide additional coverage of high priority industry needs, including operations and maintenance, and to focus the program on an actual hardware and software system that is under development

Ray: A Distributed Execution Engine for the Machine Learning Ecosystem

In recent years, growing data volumes and more sophisticated computational procedures have greatly increased the demand for computational power. Machine learning and artificial intelligence applications, for example, are notorious for their computational requirements. At the same time, Moores law is ending and processor speeds are stalling. As a result, distributed computing has become ubiquitous. While the cloud makes distributed hardware infrastructure widely accessible and therefore offers the potential of horizontal scale, developing these distributed algorithms and applications remains surprisingly hard. This is due to the inherent complexity of concurrent algorithms, the engineering challenges that arise when communicating between many machines, the requirements like fault tolerance and straggler mitigation that arise at large scale and the lack of a general-purpose distributed execution engine that can support a wide variety of applications.In this thesis, we study the requirements for a general-purpose distributed computation model and present a solution that is easy to use yet expressive and resilient to faults. At its core our model takes familiar concepts from serial programming, namely functions and classes, and generalizes them to the distributed world, therefore unifying stateless and stateful distributed computation. This model not only supports many machine learning workloads like training or serving, but is also a good t for cross-cutting machine learning applications like reinforcement learning and data processing applications like streaming or graph processing. We implement this computational model as an open-source system called Ray, which matches or exceeds the performance of specialized systems in many application domains, while also offering horizontally scalability and strong fault tolerance properties

SHStream: Self-Healing Framework for HTTP Video-Streaming

HTTP video-streaming is leading delivery of video content over the Internet. This phenomenon is explained by the ubiquity of web browsers, the permeability of HTTP traffic and the recent video technologies around HTML5. However, the inclusion of multimedia requests imposes new requirements on web servers due to responses with lifespans that can reach dozens of minutes and timing requirements for data fragments transmitted during the response period. Consequently, web- servers require real-time performance control to avoid playback outages caused by overloading and performance anomalies. We present SHStream , a self-healing framework for web servers delivering video-streaming content that provides (1) load admit- tance to avoid server overloading; (2) prediction of performance anomalies using online data stream learning algorithms; (3) continuous evaluation and selection of the best algorithm for prediction; and (4) proactive recovery by migrating the server to other hosts using container-based virtualization techniques. Evaluation of our framework using several variants of Hoeffding trees and ensemble algorithms showed that with a small number of learning instances, it is possible to achieve approximately 98% of recall and 99% of precision for failure predictions. Additionally, proactive failover can be performed in less than 1 secon

A proposal for a global task planning architecture using the RoboEarth cloud based framework

As robotic systems become more and more capable of assisting in human domains, methods are sought to compose robot executable plans from abstract human instructions. To cope with the semantically rich and highly expressive nature of human instructions, Hierarchical Task Network planning is often being employed along with domain knowledge to solve planning problems in a pragmatic way. Commonly, the domain knowledge is specific to the planning problem at hand, impeding re-use. Therefore this paper conceptualizes a global planning architecture, based on the worldwide accessible RoboEarth cloud framework. This architecture allows environmental state inference and plan monitoring on a global level. To enable plan re-use for future requests, the RoboEarth action language has been adapted to allow semantic matching of robot capabilities with previously composed plans