Deep Reinforcement Learning in Surgical Robotics: Enhancing the Automation Level

Surgical robotics is a rapidly evolving field that is transforming the landscape of surgeries. Surgical robots have been shown to enhance precision, minimize invasiveness, and alleviate surgeon fatigue. One promising area of research in surgical robotics is the use of reinforcement learning to enhance the automation level. Reinforcement learning is a type of machine learning that involves training an agent to make decisions based on rewards and punishments. This literature review aims to comprehensively analyze existing research on reinforcement learning in surgical robotics. The review identified various applications of reinforcement learning in surgical robotics, including pre-operative, intra-body, and percutaneous procedures, listed the typical studies, and compared their methodologies and results. The findings show that reinforcement learning has great potential to improve the autonomy of surgical robots. Reinforcement learning can teach robots to perform complex surgical tasks, such as suturing and tissue manipulation. It can also improve the accuracy and precision of surgical robots, making them more effective at performing surgeries

What is This?

Downloaded from ijr.sagepub.com at UNIV CALIFORNIA BERKELEY LIB on June 18, 2014Article Motion planning with sequential convex optimization and convex collision checkin

Toward certifiable optimal motion planning for medical steerable needles

Medical steerable needles can follow 3D curvilinear trajectories to avoid anatomical obstacles and reach clinically significant targets inside the human body. Automating steerable needle procedures can enable physicians and patients to harness the full potential of steerable needles by maximally leveraging their steerability to safely and accurately reach targets for medical procedures such as biopsies. For the automation of medical procedures to be clinically accepted, it is critical from a patient care, safety, and regulatory perspective to certify the correctness and effectiveness of the planning algorithms involved in procedure automation. In this paper, we take an important step toward creating a certifiable optimal planner for steerable needles. We present an efficient, resolution-complete motion planner for steerable needles based on a novel adaptation of multi-resolution planning. This is the first motion planner for steerable needles that guarantees to compute in finite time an obstacle-avoiding plan (or notify the user that no such plan exists), under clinically appropriate assumptions. Based on this planner, we then develop the first resolution-optimal motion planner for steerable needles that further provides theoretical guarantees on the quality of the computed motion plan, that is, global optimality, in finite time. Compared to state-of-the-art steerable needle motion planners, we demonstrate with clinically realistic simulations that our planners not only provide theoretical guarantees but also have higher success rates, have lower computation times, and result in higher quality plans

Robotic System Development for Precision MRI-Guided Needle-Based Interventions

This dissertation describes the development of a methodology for implementing robotic systems for interventional procedures under intraoperative Magnetic Resonance Imaging (MRI) guidance. MRI is an ideal imaging modality for surgical guidance of diagnostic and therapeutic procedures, thanks to its ability to perform high resolution, real-time, and high soft tissue contrast imaging without ionizing radiation. However, the strong magnetic field and sensitivity to radio frequency signals, as well as tightly confined scanner bore render great challenges to developing robotic systems within MRI environment. Discussed are potential solutions to address engineering topics related to development of MRI-compatible electro-mechanical systems and modeling of steerable needle interventions. A robotic framework is developed based on a modular design approach, supporting varying MRI-guided interventional procedures, with stereotactic neurosurgery and prostate cancer therapy as two driving exemplary applications. A piezoelectrically actuated electro-mechanical system is designed to provide precise needle placement in the bore of the scanner under interactive MRI-guidance, while overcoming the challenges inherent to MRI-guided procedures. This work presents the development of the robotic system in the aspects of requirements definition, clinical work flow development, mechanism optimization, control system design and experimental evaluation. A steerable needle is beneficial for interventional procedures with its capability to produce curved path, avoiding anatomical obstacles or compensating for needle placement errors. Two kinds of steerable needles are discussed, i.e. asymmetric-tip needle and concentric-tube cannula. A novel Gaussian-based ContinUous Rotation and Variable-curvature (CURV) model is proposed to steer asymmetric-tip needle, which enables variable curvature of the needle trajectory with independent control of needle rotation and insertion. While concentric-tube cannula is suitable for clinical applications where a curved trajectory is needed without relying on tissue interaction force. This dissertation addresses fundamental challenges in developing and deploying MRI-compatible robotic systems, and enables the technologies for MRI-guided needle-based interventions. This study applied and evaluated these techniques to a system for prostate biopsy that is currently in clinical trials, developed a neurosurgery robot prototype for interstitial thermal therapy of brain cancer under MRI guidance, and demonstrated needle steering using both asymmetric tip and pre-bent concentric-tube cannula approaches on a testbed

Integrating Optimization and Sampling for Robot Motion Planning with Applications in Healthcare

Robots deployed in human-centric environments, such as a person's home in a home-assistance setting or inside a person's body in a surgical setting, have the potential to have a large, positive impact on human quality of life. However, for robots to operate in such environments they must be able to move efficiently while avoiding colliding with obstacles such as objects in the person's home or sensitive anatomical structures in the person's body. Robot motion planning aims to compute safe and efficient motions for robots that avoid obstacles, but home assistance and surgical robots come with unique challenges that can make this difficult. For instance, many state of the art surgical robots have computationally expensive kinematic models, i.e., it can be computationally expensive to predict their shape as they move. Some of these robots have hybrid dynamics, i.e., they consist of multiple stages that behave differently. Additionally, it can be difficult to plan motions for robots while leveraging real-world sensor data, such as point clouds. In this dissertation, we demonstrate and empirically evaluate methods for overcoming these challenges to compute high-quality and safe motions for robots in home-assistance and surgical settings. First, we present a motion planning method for a continuum, parallel surgical manipulator that accounts for its computationally expensive kinematics. We then leverage this motion planner to optimize its kinematic design chosen prior to a surgical procedure. Next, we present a motion planning method for a 3-stage lung tumor biopsy robot that accounts for its hybrid dynamics and evaluate the robot and planner in simulation and in inflated porcine lung tissue. Next, we present a motion planning method for a home-assistance robot that leverages real-world, point-cloud obstacle representations. We then expand this method to work with a type of continuum surgical manipulator, a concentric tube robot, with point-cloud anatomical representations. Finally, we present a data-driven machine learning method for more accurately estimating the shape of concentric tube robots. By effectively addressing challenges associated with home assistance and surgical robots operating in human-centric environments, we take steps toward enabling robots to have a positive impact on human quality of life.Doctor of Philosoph

Modeling, Sensorization and Control of Concentric-Tube Robots

Since the concept of the Concentric-Tube Robot (CTR) was proposed in 2006, CTRs have been a popular research topic in the field of surgical robotics. The unique mechanical design of this robot allows it to navigate through narrow channels in the human anatomy and operate in highly constrained environments. It is therefore likely to become the next generation of surgical robots to overcome the challenges that cannot be addressed by current technologies. In CSTAR, we have had ongoing work over the past several years aimed at developing novel techniques and technologies for CTRs. This thesis describes the contributions made in this context, focusing primarily on topics such as modeling, sensorization, and control of CTRs. Prior to this work, one of the main challenges in CTRs was to develop a kinematic model that achieves a balance between the numerical accuracy and computational efficiency for surgical applications. In this thesis, a fast kinematic model of CTRs is proposed, which can be solved at a comparatively fast rate (0.2 ms) with minimal loss of accuracy (0.1 mm) for a 3-tube CTR. A Jacobian matrix is derived based on this model, leading to the development of a real-time trajectory tracking controller for CTRs. For tissue-robot interactions, a force-rejection controller is proposed for position control of CTRs under time-varying force disturbances. In contrast to rigid-link robots, instability of position control could be caused by non-unique solutions to the forward kinematics of CTRs. This phenomenon is modeled and analyzed, resulting in design criteria that can ensure kinematic stability of a CTR in its entire workspace. Force sensing is another major difficulty for CTRs. To address this issue, commercial force/torque sensors (Nano43, ATI Industrial Automation, United States) are integrated into one of our CTR prototypes. These force/torque sensors are replaced by Fiber-Bragg Grating (FBG) sensors that are helically-wrapped and embedded in CTRs. A strain-force calculation algorithm is proposed, to convert the reflected wavelength of FBGs into force measurements with 0.1 N force resolution at 100 Hz sampling rate. In addition, this thesis reports on our innovations in prototyping drive units for CTRs. Three designs of CTR prototypes are proposed, the latest one being significantly more compact and cost efficient in comparison with most designs in the literature. All of these contributions have brought this technology a few steps closer to being used in operating rooms. Some of the techniques and technologies mentioned above are not merely limited to CTRs, but are also suitable for problems arising in other types of surgical robots, for example, for sensorizing da Vinci surgical instruments for force sensing (see Appendix A)

Needle Steering in 3-D Via Rapid Replanning

Steerable needles have the potential to improve the effectiveness of needle-based clinical procedures such as biopsy and drug delivery by improving targeting accuracy and reaching previously inaccessible targets that are behind sensitive or impenetrable anatomical regions. We present a new needle steering system capable of automatically reaching targets in 3-D environments while avoiding obstacles and compensating for real-world uncertainties. Given a specification of anatomical obstacles and a clinical target (e.g., from preoperative medical images), our system plans and controls needle motion in a closed-loop fashion under sensory feedback to optimize a clinical metric. We unify planning and control using a new fast algorithm that continuously replans the needle motion. Our rapid replanning approach is enabled by an efficient sampling-based rapidly exploring random tree (RRT) planner that achieves orders-of-magnitude reduction in computation time compared with prior 3-D approaches by incorporating variable curvature kinematics and a novel distance metric for planning. Our system uses an electromagnetic tracking system to sense the state of the needle tip during the procedure. We experimentally evaluate our needle steering system using tissue phantoms and animal tissue ex vivo. We demonstrate that our rapid replanning strategy successfully guides the needle around obstacles to desired 3-D targets with an average error of less than 3 mm

Closed-Loop Planning and Control of Steerable Medical Needles

Steerable needles have the potential to increase the effectiveness of needle-based clinical procedures such as biopsy, drug delivery, and radioactive seed implantation for cancer treatment. These needles can trace curved paths when inserted into tissue, thereby increasing maneuverability and targeting accuracy while reaching previously inaccessible targets that are behind sensitive or impenetrable anatomical regions. Guiding these flexible needles along an intended path requires continuously inserting and twisting the needle at its base, which is not intuitive for a human operator. In addition, the needle often deviates from its intended trajectory due to factors such as tissue deformation, needle-tissue interaction, noisy actuation and sensing, modeling errors, and involuntary patient motions. These challenges can be addressed with the assistance of robotic systems that automatically compensate for these perturbations by performing motion planning and feedback control of the needle in a closed-loop fashion under sensory feedback. We present two approaches for efficient closed-loop guidance of steerable needles to targets within clinically acceptable accuracy while safely avoiding sensitive or impenetrable anatomical structures. The first approach uses a fast motion planning algorithm that unifies planning and control by continuously replanning, enabling correction for perturbations as they occur. We evaluate our method using a needle steering system in phantom and ex vivo animal tissues. The second approach integrates motion planning and feedback control of steerable needles in highly deformable environments. We demonstrate that this approach significantly improves the probability of success compared to prior approaches that either consider uncertainty or deformations but not both simultaneously. We also propose a data-driven method to estimate parameters of stochastic models of steerable needle motion. These models can be used to create realistic medical simulators for clinicians wanting to train for steerable needle procedures and to improve the effectiveness of existing planning and control methods. This dissertation advances the state of the art in planning and control of steerable needles and is an important step towards realizing needle steering in clinical practice. The methods developed in this dissertation also generalize to important applications beyond medical needle steering, such as manipulating deformable objects and control of mobile robots.Doctor of Philosoph

Deep Reinforcement Learning for Inverse Kinematics and Path Following for Concentric Tube Robots

Concentric tube robots (CTRs) are continuum robots that allow for bending and twisting motions unattainable by traditional rigid link robots. The curvilinear backbones can benefit surgical applications by improving dexterity, enlarging the workspace, and reducing trauma at the entry point of the instrument. The curvilinear backbone that is attributed is the result of pre-curved, super-elastic tubes arranged concentrically. Each tube has a straight and pre-curved section and is actuated in rotation and translation from the tube base with the neighboring tube interactions producing the curvilinear backbone. The modeling of the neighboring tube interactions is non-trivial, and an explored topic in CTR literature. However, model-based kinematics and control can be inaccurate due to inherent manufacturing errors of the tubes, permanent deformation over time, and unmodelled physical interactions. This thesis proposes a model-free control method using deep reinforcement learning (DRL). The DRL framework aims to control the end-effector of the CTR with limited modeling information by leveraging simulation data, which is much less costly than hardware data. To develop a DRL framework, a Markov Decision Process (MDP) with states, actions, and rewards needs to be defined for the inverse kinematics task. First, action exploration was investigated with this MDP in a simpler simulation as CTRs have a unique extension degree of freedom per tube. Next, state representation, curriculum reward, and adaptation methods over multiple CTR systems were developed in a more accurate simulation. To validate the work in simulation, a noise-induced simulation environment was utilized to demonstrate the initial robustness of the learned policy. Finally, a hardware system was developed where a workspace characterization was performed to determine simulation to hardware differences. By using Sim2Real domain transfer, a simulation policy was successfully transferred to hardware for inverse kinematics and path following, validating the approach