Deep Reinforcement Learning for Concentric Tube Robot Control with a Goal-Based Curriculum

Concentric Tube Robots (CTRs), a type of continuum robot, are a collection of concentric, pre-curved tubes composed of super elastic nickel titanium alloy. CTRs can bend and twist from the interactions between neighboring tubes causing the kinematics and therefore control of the end-effector to be very challenging to model. In this paper, we develop a control scheme for a CTR end-effector in Cartesian space with no prior kinematic model using a deep reinforcement learning (DRL) approach with a goal-based curriculum reward strategy. We explore the use of curricula by changing the goal tolerance through training with constant, linear and exponential decay functions. Also, relative and absolute joint representations as a way of improving training convergence are explored. Quantitative comparisons for combinations of curricula and joint representations are performed and the exponential decay relative approach is used for training a robust policy in a noise-induced simulation environment. Compared to a previous DRL approach, our new method reduces training time and employs a more complex simulation environment. We report mean Cartesian errors of 1.29 mm and a success rate of 0.93 with a relative decay curriculum. In path following, we report mean errors of 1.37 mm in a noise-induced path following task. Albeit in simulation, these results indicate the promise of using DRL in model free control of continuum robots and CTRs in particular

Modeling, Sensorization and Control of Concentric-Tube Robots

Since the concept of the Concentric-Tube Robot (CTR) was proposed in 2006, CTRs have been a popular research topic in the field of surgical robotics. The unique mechanical design of this robot allows it to navigate through narrow channels in the human anatomy and operate in highly constrained environments. It is therefore likely to become the next generation of surgical robots to overcome the challenges that cannot be addressed by current technologies. In CSTAR, we have had ongoing work over the past several years aimed at developing novel techniques and technologies for CTRs. This thesis describes the contributions made in this context, focusing primarily on topics such as modeling, sensorization, and control of CTRs. Prior to this work, one of the main challenges in CTRs was to develop a kinematic model that achieves a balance between the numerical accuracy and computational efficiency for surgical applications. In this thesis, a fast kinematic model of CTRs is proposed, which can be solved at a comparatively fast rate (0.2 ms) with minimal loss of accuracy (0.1 mm) for a 3-tube CTR. A Jacobian matrix is derived based on this model, leading to the development of a real-time trajectory tracking controller for CTRs. For tissue-robot interactions, a force-rejection controller is proposed for position control of CTRs under time-varying force disturbances. In contrast to rigid-link robots, instability of position control could be caused by non-unique solutions to the forward kinematics of CTRs. This phenomenon is modeled and analyzed, resulting in design criteria that can ensure kinematic stability of a CTR in its entire workspace. Force sensing is another major difficulty for CTRs. To address this issue, commercial force/torque sensors (Nano43, ATI Industrial Automation, United States) are integrated into one of our CTR prototypes. These force/torque sensors are replaced by Fiber-Bragg Grating (FBG) sensors that are helically-wrapped and embedded in CTRs. A strain-force calculation algorithm is proposed, to convert the reflected wavelength of FBGs into force measurements with 0.1 N force resolution at 100 Hz sampling rate. In addition, this thesis reports on our innovations in prototyping drive units for CTRs. Three designs of CTR prototypes are proposed, the latest one being significantly more compact and cost efficient in comparison with most designs in the literature. All of these contributions have brought this technology a few steps closer to being used in operating rooms. Some of the techniques and technologies mentioned above are not merely limited to CTRs, but are also suitable for problems arising in other types of surgical robots, for example, for sensorizing da Vinci surgical instruments for force sensing (see Appendix A)

Data-efficient Non-parametric Modelling and Control of an Extensible Soft Manipulator

Data-driven approaches have shown promising results in modeling and controlling robots, specifically soft and flexible robots where developing physics-based models are more challenging. However, these methods often require a large number of real data, and gathering such data is time-consuming and can damage the robot as well. This paper proposed a novel data-efficient and non-parametric approach to develop a continuous model using a small dataset of real robot demonstrations (only 25 points). To the best of our knowledge, the proposed approach is the most sample-efficient method for soft continuum robot. Furthermore, we employed this model to develop a controller to track arbitrary trajectories in the feasible kinematic space. To show the performance of the proposed approach, a set of trajectory-tracking experiments has been conducted. The results showed that the robot was able to track the references precisely even in presence of external loads (up to 25 grams). Moreover, fine object manipulation experiments were performed to demonstrate the effectiveness of the proposed method in real-world tasks. Finally, we compared its performance with common data-driven approaches in seen/useen-before trajectory tracking scenarios. The results validated that the proposed approach significantly outperformed the existing approaches in unseen-before scenarios and offered similar performance in seen-before scenarios

Data-driven modelling and control of concentric tube robots with application in distal lung sampling

This research aims to explore the use of Concentric Tube Robots (CTRs) as a novel alternative to needle-based interventions in order to make these procedures more accurate and repeatable. CTRs due to their small footprint, compliance, and dexterity have been proposed for several minimally-invasive robotic surgeries. As a novel flexible robot, it has the potential to reach distal parts of the human lung that are difficult or impossible to reach with conventional needle-based interventions. There are, however, still significant challenges associated with the motion and position control of CTRs. Commonly used model-based control approaches are computationally expensive to solve and often employ simplified geometric/dynamic assumptions, which could be inaccurate in the presence of unmodelled disturbances and external interaction forces. Consequently, this work explores different control strategies to overcome these limitation. This is achieved by first building a simulation environment based on a computationally improved kinematic model that enables real-time control. Then, data-driven control approaches are investigated in order to provide accurate position control in the presence of uncertainties in the system. Finally, a three-phase affordance-aware motion planner is proposed to demonstrate the feasibility of using CTRs for percutaneous lung biopsy. According to this, the first part of this work concentrates on computationally efficient real-time modelling and simulation of CTRs. In order to achieve this, two approaches are taken. The first one introduces a method that can rapidly estimate the solution of the kinematic model, while the second approach focuses on implementing the existing model in a computationally efficient way in Robot Operating System (ROS) using C++. Second, this work explores data-driven solutions to control the robot without relying on the kinematic model. Consequently, two data-driven solutions are proposed, namely the Hybrid Dual Jacobian approach and the Extended Dynamic Mode Decomposition (EDMD) algorithm. The hybrid controller combines the advantages of model-based and data-driven control approaches, while the EDMD provides a completely model-free solution to control the robot. Both controllers are capable of rapidly predicting the robot’s nonlinear dynamics from a limited data set and offer consistent control under external loading and in the presence of obstacles. The third part of the thesis explores the use of CTRs in the context of distal lung sampling. This work demonstrates that CTRs are suitable for Needle-Based Optical Endomicroscopy where a CTR steers a fluorescent imaging probe with cellular and bacterial imaging capability inside soft tissue. Then, it is also demonstrated that a CTR can be used as a Steerable Needle to reach a target deep inside the tissue. To achieve these tasks, a motion planner is essential due to the fact that a CTR is only capable of reaching specific points in its workspace and there are a number of configurations where the robot becomes unstable. Based on this, a threeii phase affordance-aware motion planner algorithm is developed. The motion planner selects the best entry point for a specific task. Based on the selected entry point it first generates a stable trajectory from the robot’s initial configuration to the selected entry point. Then, a feasible trajectory is generated from the entry point to the target. Finally, the proposed datadriven control algorithm is applied to autonomously steer the robot on the generated trajectory toward the target region for endomicroscopic imaging

Nonparametric Online Learning Control for Soft Continuum Robot: An Enabling Technique for Effective Endoscopic Navigation.

Bioinspired robotic structures comprising soft actuation units have attracted increasing research interest. Taking advantage of its inherent compliance, soft robots can assure safe interaction with external environments, provided that precise and effective manipulation could be achieved. Endoscopy is a typical application. However, previous model-based control approaches often require simplified geometric assumptions on the soft manipulator, but which could be very inaccurate in the presence of unmodeled external interaction forces. In this study, we propose a generic control framework based on nonparametric and online, as well as local, training to learn the inverse model directly, without prior knowledge of the robot's structural parameters. Detailed experimental evaluation was conducted on a soft robot prototype with control redundancy, performing trajectory tracking in dynamically constrained environments. Advanced element formulation of finite element analysis is employed to initialize the control policy, hence eliminating the need for random exploration in the robot's workspace. The proposed control framework enabled a soft fluid-driven continuum robot to follow a 3D trajectory precisely, even under dynamic external disturbance. Such enhanced control accuracy and adaptability would facilitate effective endoscopic navigation in complex and changing environments

Integrating Optimization and Sampling for Robot Motion Planning with Applications in Healthcare

Robots deployed in human-centric environments, such as a person's home in a home-assistance setting or inside a person's body in a surgical setting, have the potential to have a large, positive impact on human quality of life. However, for robots to operate in such environments they must be able to move efficiently while avoiding colliding with obstacles such as objects in the person's home or sensitive anatomical structures in the person's body. Robot motion planning aims to compute safe and efficient motions for robots that avoid obstacles, but home assistance and surgical robots come with unique challenges that can make this difficult. For instance, many state of the art surgical robots have computationally expensive kinematic models, i.e., it can be computationally expensive to predict their shape as they move. Some of these robots have hybrid dynamics, i.e., they consist of multiple stages that behave differently. Additionally, it can be difficult to plan motions for robots while leveraging real-world sensor data, such as point clouds. In this dissertation, we demonstrate and empirically evaluate methods for overcoming these challenges to compute high-quality and safe motions for robots in home-assistance and surgical settings. First, we present a motion planning method for a continuum, parallel surgical manipulator that accounts for its computationally expensive kinematics. We then leverage this motion planner to optimize its kinematic design chosen prior to a surgical procedure. Next, we present a motion planning method for a 3-stage lung tumor biopsy robot that accounts for its hybrid dynamics and evaluate the robot and planner in simulation and in inflated porcine lung tissue. Next, we present a motion planning method for a home-assistance robot that leverages real-world, point-cloud obstacle representations. We then expand this method to work with a type of continuum surgical manipulator, a concentric tube robot, with point-cloud anatomical representations. Finally, we present a data-driven machine learning method for more accurately estimating the shape of concentric tube robots. By effectively addressing challenges associated with home assistance and surgical robots operating in human-centric environments, we take steps toward enabling robots to have a positive impact on human quality of life.Doctor of Philosoph