Supervisory teleoperation with online learning and optimal control

We present a general approach for online learning and optimal control of manipulation tasks in a supervisory teleoperation context, targeted to underwater remotely operated vehicles (ROVs). We use an online Bayesian nonparametric learning algorithm to build models of manipulation motions as task-parametrized hidden semi-Markov models (TP-HSMM) that capture the spatiotemporal characteristics of demonstrated motions in a probabilistic representation. Motions are then executed autonomously using an optimal controller, namely a model predictive control (MPC) approach in a receding horizon fashion. This way the remote system locally closes a high-frequency control loop that robustly handles noise and dynamically changing environments. Our system automates common and recurring tasks, allowing the operator to focus only on the tasks that genuinely require human intervention. We demonstrate how our solution can be used for a hot-stabbing motion in an underwater teleoperation scenario. We evaluate the performance of the system over multiple trials and compare with a state-of-the-art approach. We report that our approach generalizes well with only a few demonstrations, accurately performs the learned task and adapts online to dynamically changing task conditions

INTEGRATING USABILITY WITH TASK PERFORMANCE FOR SHARED AUTONOMY

As robots become more complex, the degrees-of-freedom (DoF) for controlling them is rapidly outpacing the degrees-of-control that can be supplied by humans via conventional interfaces. In practice, this is making it more difficult for users to control robots via teleoperation. Full autonomy offers an alternative control paradigm that lowers user burden but sacrifices adaptability to novel tasks and dynamic environments. Shared control has emerged as a way to capture the best of both worlds. In shared control, commands from a user are combined with commands from machine intelligence to provide an improved human-machine experience. In this work, I tackle two specific research challenges for shared control. First, I introduce a novel approach for controlling robots by leveraging machine learning and latent spaces. I call this approach \emph{amplified control} in reference to its ability to amplify teleoperation by enabling low-DoF inputs to control high-DoF robots. Often, machine learning based control algorithms are trained to optimize task performance while ignoring usability by human operators. This typically leads to control models that work well in simulation, but fail to transfer to effective human users. My second research contribution tackles this challenge by defining a set of novel usability metrics and incorporating them into my machine learning process. This introduces tradeoffs between optimizing my amplified control models for task performance and usability. I study these challenges in the context of assistive robotics where the tradeoffs between task performance and usability are particularly relevant. I verify the performance of my amplified control algorithms and evaluate human-robot interactions with both a physical robotic system and a simulation-based representation. Although I focus on assistive robotics, this work contributes to the more general challenge of creating meaningful user interactions as robots grow in complexity

A Scalable, High-Performance, Real-Time Control Architecture with Application to Semi-Autonomous Teleoperation

A scalable and real-time capable infrastructure is required to enable high-performance control and haptic rendering of systems with many degrees-of-freedom. The specific platform that motivates this thesis work is the open research platform da Vinci ReResearch Kit (dVRK). For the system architecture, we propose a specialized IEEE-1394 (FireWire) broadcast protocol that takes advantage of broadcast and peer-to-peer transfers to minimize the number of transactions, and thus the software overhead, on the control PC, thereby enabling fast real-time control. It has also been extended to Ethernet via a novel Ethernet-to-FireWire bridge protocol. The software architecture consists of a distributed hardware interface layer, a real-time component-based software framework, and integration with the Robot Operating System (ROS). The architecture is scalable to support multiple active manipulators, reconfigurable to enable researchers to partition a full system into multiple independent subsystems, and extensible at all levels of control. This architecture has been applied to two semi-autonomous teleoperation applications. The first application is a suturing task in Robotic Minimally Invasive Surgery (RMIS), that includes the development of virtual fixtures for the needle passing and knot tying sub-tasks, with a multi-user study to verify their effectiveness. The second application concerns time-delayed teleoperation of a robotic arm for satellite servicing. The research contribution includes the development of a line virtual fixture with augmented reality, a test for different time delay configurations and a multi-user study that evaluates the effectiveness of the system

Modeling and Improving Teleoperation Performance of Semi-Autonomous Wheeled Robots

Robotics and unmanned vehicles have allowed us to interact with environments in ways that were impossible decades ago. As perception, decision making, and control improve, it becomes possible to automate more parts of robot operation. However, humans will remain a critical part of robot control based on preference, ethical, and technical reasons. An ongoing question will be when and how to pair humans and automation to create semi-autonomous systems. The answer to this question depends on numerous factors such as the robot's task, platform, environment conditions, and the user. The work in this dissertation focuses on modeling the impact of these factors on performance and developing improved semi-autonomous control schemes, so that robot systems can be better designed. Experiments and analysis focus on wheeled robots, however the approach taken and many of the trends could be applied to a variety of platforms. Wheeled robots are often teleoperated over wireless communication networks. While this arrangement may be convenient, it introduces many challenges including time-varying delays and poor perception of the robot's environment that can lead to the robot colliding with objects or rolling over. With regards to semi-autonomous control, rollover prevention and obstacle avoidance behaviors are considered. In this area, two contributions are presented. The first is a rollover prevention method that uses an existing manipulator arm on-board a wheeled robot. The second is a method of approximating convex obstacle free regions for use in optimal control path planning problems. Teleoperation conditions, including communication delays, automation, and environment layout, are considered in modeling robot operation performance. From these considerations stem three contributions. The first is a method of relating driving performance among different communication delay distributions. The second parameterizes how driving through different arrangements of obstacles relates to performance. Lastly, based on user studies, teleoperation performance is related to different conditions of communication delay, automation level, and environment arrangement. The contributions of this dissertation will assist roboticists to implement better automation and understand when to use automation.PHDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/136951/1/jgstorms_1.pd