Simulation-aided Learning from Demonstration for Robotic LEGO Construction

Recent advancements in manufacturing have a growing demand for fast, automatic prototyping (i.e. assembly and disassembly) capabilities to meet users' needs. This paper studies automatic rapid LEGO prototyping, which is devoted to constructing target LEGO objects that satisfy individual customization needs and allow users to freely construct their novel designs. A construction plan is needed in order to automatically construct the user-specified LEGO design. However, a freely designed LEGO object might not have an existing construction plan, and generating such a LEGO construction plan requires a non-trivial effort since it requires accounting for numerous constraints (e.g. object shape, colors, stability, etc.). In addition, programming the prototyping skill for the robot requires the users to have expert programming skills, which makes the task beyond the reach of the general public. To address the challenges, this paper presents a simulation-aided learning from demonstration (SaLfD) framework for easily deploying LEGO prototyping capability to robots. In particular, the user demonstrates constructing the customized novel LEGO object. The robot extracts the task information by observing the human operation and generates the construction plan. A simulation is developed to verify the correctness of the learned construction plan and the resulting LEGO prototype. The proposed system is deployed to a FANUC LR-mate 200id/7L robot. Experiments demonstrate that the proposed SaLfD framework can effectively correct and learn the prototyping (i.e. assembly and disassembly) tasks from human demonstrations. And the learned prototyping tasks are realized by the FANUC robot

Adaptive Robot Framework: Providing Versatility and Autonomy to Manufacturing Robots Through FSM, Skills and Agents

207 p.The main conclusions that can be extracted from an analysis of the current situation and future trends of the industry,in particular manufacturing plants, are the following: there is a growing need to provide customization of products, ahigh variation of production volumes and a downward trend in the availability of skilled operators due to the ageingof the population. Adapting to this new scenario is a challenge for companies, especially small and medium-sizedenterprises (SMEs) that are suffering first-hand how their specialization is turning against them.The objective of this work is to provide a tool that can serve as a basis to face these challenges in an effective way.Therefore the presented framework, thanks to its modular architecture, allows focusing on the different needs of eachparticular company and offers the possibility of scaling the system for future requirements. The presented platform isdivided into three layers, namely: interface with robot systems, the execution engine and the application developmentlayer.Taking advantage of the provided ecosystem by this framework, different modules have been developed in order toface the mentioned challenges of the industry. On the one hand, to address the need of product customization, theintegration of tools that increase the versatility of the cell are proposed. An example of such tools is skill basedprogramming. By applying this technique a process can be intuitively adapted to the variations or customizations thateach product requires. The use of skills favours the reuse and generalization of developed robot programs.Regarding the variation of the production volumes, a system which permits a greater mobility and a faster reconfigurationis necessary. If in a certain situation a line has a production peak, mechanisms for balancing the loadwith a reasonable cost are required. In this respect, the architecture allows an easy integration of different roboticsystems, actuators, sensors, etc. In addition, thanks to the developed calibration and set-up techniques, the system canbe adapted to new workspaces at an effective time/cost.With respect to the third mentioned topic, an agent-based monitoring system is proposed. This module opens up amultitude of possibilities for the integration of auxiliary modules of protection and security for collaboration andinteraction between people and robots, something that will be necessary in the not so distant future.For demonstrating the advantages and adaptability improvement of the developed framework, a series of real usecases have been presented. In each of them different problematic has been resolved using developed skills,demonstrating how are adapted easily to the different casuistic

Protective Behavior Detection in Chronic Pain Rehabilitation: From Data Preprocessing to Learning Model

Chronic pain (CP) rehabilitation extends beyond physiotherapist-directed clinical sessions and primarily functions in people's everyday lives. Unfortunately, self-directed rehabilitation is difficult because patients need to deal with both their pain and the mental barriers that pain imposes on routine functional activities. Physiotherapists adjust patients' exercise plans and advice in clinical sessions based on the amount of protective behavior (i.e., a sign of anxiety about movement) displayed by the patient. The goal of such modifications is to assist patients in overcoming their fears and maintaining physical functioning. Unfortunately, physiotherapists' support is absent during self-directed rehabilitation or also called self-management that people conduct in their daily life. To be effective, technology for chronic-pain self-management should be able to detect protective behavior to facilitate personalized support. Thereon, this thesis addresses the key challenges of ubiquitous automatic protective behavior detection (PBD). Our investigation takes advantage of an available dataset (EmoPain) containing movement and muscle activity data of healthy people and people with CP engaged in typical everyday activities. To begin, we examine the data augmentation methods and segmentation parameters using various vanilla neural networks in order to enable activity-independent PBD within pre-segmented activity instances. Second, by incorporating temporal and bodily attention mechanisms, we improve PBD performance and support theoretical/clinical understanding of protective behavior that the attention of a person with CP shifts between body parts perceived as risky during feared movements. Third, we use human activity recognition (HAR) to improve continuous PBD in data of various activity types. The approaches proposed above are validated against the ground truth established by majority voting from expert annotators. Unfortunately, using such majority-voted ground truth causes information loss, whereas direct learning from all annotators is vulnerable to noise from disagreements. As the final study, we improve the learning from multiple annotators by leveraging the agreement information for regularization

Adaptive Robot Framework: Providing Versatility and Autonomy to Manufacturing Robots Through FSM, Skills and Agents

207 p.The main conclusions that can be extracted from an analysis of the current situation and future trends of the industry,in particular manufacturing plants, are the following: there is a growing need to provide customization of products, ahigh variation of production volumes and a downward trend in the availability of skilled operators due to the ageingof the population. Adapting to this new scenario is a challenge for companies, especially small and medium-sizedenterprises (SMEs) that are suffering first-hand how their specialization is turning against them.The objective of this work is to provide a tool that can serve as a basis to face these challenges in an effective way.Therefore the presented framework, thanks to its modular architecture, allows focusing on the different needs of eachparticular company and offers the possibility of scaling the system for future requirements. The presented platform isdivided into three layers, namely: interface with robot systems, the execution engine and the application developmentlayer.Taking advantage of the provided ecosystem by this framework, different modules have been developed in order toface the mentioned challenges of the industry. On the one hand, to address the need of product customization, theintegration of tools that increase the versatility of the cell are proposed. An example of such tools is skill basedprogramming. By applying this technique a process can be intuitively adapted to the variations or customizations thateach product requires. The use of skills favours the reuse and generalization of developed robot programs.Regarding the variation of the production volumes, a system which permits a greater mobility and a faster reconfigurationis necessary. If in a certain situation a line has a production peak, mechanisms for balancing the loadwith a reasonable cost are required. In this respect, the architecture allows an easy integration of different roboticsystems, actuators, sensors, etc. In addition, thanks to the developed calibration and set-up techniques, the system canbe adapted to new workspaces at an effective time/cost.With respect to the third mentioned topic, an agent-based monitoring system is proposed. This module opens up amultitude of possibilities for the integration of auxiliary modules of protection and security for collaboration andinteraction between people and robots, something that will be necessary in the not so distant future.For demonstrating the advantages and adaptability improvement of the developed framework, a series of real usecases have been presented. In each of them different problematic has been resolved using developed skills,demonstrating how are adapted easily to the different casuistic

Intuitive Instruction of Industrial Robots : A Knowledge-Based Approach

With more advanced manufacturing technologies, small and medium sized enterprises can compete with low-wage labor by providing customized and high quality products. For small production series, robotic systems can provide a cost-effective solution. However, for robots to be able to perform on par with human workers in manufacturing industries, they must become flexible and autonomous in their task execution and swift and easy to instruct. This will enable small businesses with short production series or highly customized products to use robot coworkers without consulting expert robot programmers. The objective of this thesis is to explore programming solutions that can reduce the programming effort of sensor-controlled robot tasks. The robot motions are expressed using constraints, and multiple of simple constrained motions can be combined into a robot skill. The skill can be stored in a knowledge base together with a semantic description, which enables reuse and reasoning. The main contributions of the thesis are 1) development of ontologies for knowledge about robot devices and skills, 2) a user interface that provides simple programming of dual-arm skills for non-experts and experts, 3) a programming interface for task descriptions in unstructured natural language in a user-specified vocabulary and 4) an implementation where low-level code is generated from the high-level descriptions. The resulting system greatly reduces the number of parameters exposed to the user, is simple to use for non-experts and reduces the programming time for experts by 80%. The representation is described on a semantic level, which means that the same skill can be used on different robot platforms. The research is presented in seven papers, the first describing the knowledge representation and the second the knowledge-based architecture that enables skill sharing between robots. The third paper presents the translation from high-level instructions to low-level code for force-controlled motions. The two following papers evaluate the simplified programming prototype for non-expert and expert users. The last two present how program statements are extracted from unstructured natural language descriptions

Patient-specific simulation for autonomous surgery

An Autonomous Robotic Surgical System (ARSS) has to interact with the complex anatomical environment, which is deforming and whose properties are often uncertain. Within this context, an ARSS can benefit from the availability of patient-specific simulation of the anatomy. For example, simulation can provide a safe and controlled environment for the design, test and validation of the autonomous capabilities. Moreover, it can be used to generate large amounts of patient-specific data that can be exploited to learn models and/or tasks. The aim of this Thesis is to investigate the different ways in which simulation can support an ARSS and to propose solutions to favor its employability in robotic surgery. We first address all the phases needed to create such a simulation, from model choice in the pre-operative phase based on the available knowledge to its intra-operative update to compensate for inaccurate parametrization. We propose to rely on deep neural networks trained with synthetic data both to generate a patient-specific model and to design a strategy to update model parametrization starting directly from intra-operative sensor data. Afterwards, we test how simulation can assist the ARSS, both for task learning and during task execution. We show that simulation can be used to efficiently train approaches that require multiple interactions with the environment, compensating for the riskiness to acquire data from real surgical robotic systems. Finally, we propose a modular framework for autonomous surgery that includes deliberative functions to handle real anatomical environments with uncertain parameters. The integration of a personalized simulation proves fundamental both for optimal task planning and to enhance and monitor real execution. The contributions presented in this Thesis have the potential to introduce significant step changes in the development and actual performance of autonomous robotic surgical systems, making them closer to applicability to real clinical conditions

Automatische Fehlerbehandlung in industriellen Montageszenarien auf Basis menschlicher Demonstrationen

Based on a scenario where humans and robots share their workspace, a system for automatically error handling during an automated industrial assembly is presented. If an error occurs, it is first detected and then classified. If it is a previously unknown error, the human closest to the robot will be asked to perform error handling by interacting with the robot. This interaction is recorded so that it can be reapplied if the same error occurs again. If the error is already known, an appropriate error handling is selected and applied without any further human interaction required. Thus, the interaction rate decreases over time and the system learns to handle more and more errors independently. In addition, it is presented how different recorded error handlings can be optimized according to given performance criteria. For this purpose, a suitable input device for performing the error handling is required first. In addition, the Hierarchical Decomposition (HD) is introduced as the abstract representation of an assembly operation. In this case, an assembly is subdivided into different states at multiple hierarchical levels. This is done by a domain export which also defines conditions for state transition. Thus, the HD allows assembly progress monitoring, error detection and classification as well as error prediction. A strategy presentation is introduced to store and reuse demonstrated error handling interactions. One particular feature of this representation is that a strategy is always related to the robot's end-effector pose at that point of time when an error occurs. Thus, a strategy describes the movements which have been performed for error handling. The strategy's invariance against rotation or translation allows significant reduction in the amount of strategies needed to be demonstrated by a human via interaction. Four selection criteria are introduced in order to decide if a strategy matches an error. Thereby, it is possible to make a selection based on one criterion or to perform a multi-criteria optimization using all available information. By introducing a strategy optimization approach, the overall system performance can be improved. In a subsequent experiment, it is shown that the presented error handling approach can be successfully applied.Ausgehend von einem Szenario, in dem sich Menschen und Roboter einen Arbeitsraum teilen, wird ein System zur automatischen Behandlung von Fehlerzuständen in automatisierten Montageprozessen vorgestellt. Tritt ein Fehler auf, so wird dieser erkannt und klassifiziert. Handelt es sich um einen bisher unbekannten Fehler, so wird der Mensch, welcher dem Roboter am nächsten ist gebeten, eine Fehlerbehandlung durch Interaktion mit dem Roboter durchzuführen. Diese Fehlerbehandlung wird aufgezeichnet, sodass sie bei einem erneuten Auftreten des gleichen Fehlers wieder angewendet werden kann. Ist der aufgetretene Fehler jedoch bereits bekannt, so wird eine dazu passende Fehlerbehandlung ausgewählt und ausgeführt, ohne dass es zu einer Interaktion kommt. Somit sinkt die Interaktionsrate über die Zeit betrachtet und das System lernt immer mehr Fehler eigenständig zu behandeln. Zusätzlich wird vorgestellt, wie verschiedene und aufgezeichnete Fehlerbehandlungen gemäß vorgegebenen Performancemaßen optimiert werden können. Zur Realisierung eines solchen Systems wird zunächst ein passendes Eingabegerät zur Durchführung der Fehlerbehandlung benötigt. Zusätzlich wird mit der Hierarchical Decomposition (HD) ein Ansatz zur abstrakten Beschreibung von Montagevorgängen vorgestellt. Des Weiteren wird eine Strategierepräsentation eingeführt, um demonstrierte Fehlerbehandlungen speichern und wiederverwenden zu können. Eine besondere Eigenschaft der vorgestellten Strategierepräsentation ist, dass eine Strategie immer auf die End-Effektor Pose des Roboters zu dem Zeitpunkt, an welchem der Fehler auftritt, bezogen ist. Somit beschreibt eine Strategie die Bewegungen, welche zur Fehlerbehandlung durchzuführen sind. Um Strategien auswählen zu können, werden vier Auswahlkriterien vorgestellt. Dabei ist es möglich, eine Auswahl nur auf Basis eines Kriteriums zu treffen oder alle zu berücksichtigen, in dem eine Multikriterienoptimierung durchgeführt wird. Durch die Einführung eines Verfahrens zur Optimierung von Strategien kann die Systemperformance bezüglich eines vorgegebenen Performancemaßes gesteigert werden. In einem anschließenden Experiment wird gezeigt, dass der vorgestellte Ansatz zur Fehlerbehandlung erfolgreich angewendet werden kann