Visual kinematic force estimation in robot-assisted surgery – application to knot tying

Robot-assisted surgery has potential advantages but lacks force feedback, which can lead to errors such as broken stitches or tissue damage. More experienced surgeons can judge the tool-tissue forces visually and an automated way of capturing this skill is desirable. Methods to measure force tend to involve complex measurement devices or visual tracking of tissue deformation. We investigate whether surgical forces can be estimated simply from the discrepancy between kinematic and visual measurement of the tool position. We show that combined visual and kinematic force estimation can be achieved without external measurements or modelling of tissue deformation. After initial alignment when no force is applied to the tool, visual and kinematic estimates of tool position diverge under force. We plot visual/kinematic displacement with force using vision and marker-based tracking. We demonstrate the ability to discern the forces involved in knot tying and visualize the displacement force using the publicly available JIGSAWS dataset as well as clinical examples of knot tying with the da Vinci surgical system. The ability to visualize or feel forces using this method may offer an advantage to those learning robotic surgery as well as adding to the information available to more experienced surgeons

Modeling of Force and Motion Transmission in Tendon-Driven Surgical Robots

Tendon-based transmission is a common approach for transferring motion and forces in surgical robots. In spite of design simplicity and compactness that comes with the tendon drives, there exists a number of issues associated with the tendon-based transmission. In particular, the elasticity of the tendons and the frictional interaction between the tendon and the routing result in substantially nonlinear behavior. Also, in surgical applications, the distal joints of the robot and instruments cannot be sensorized in most cases due to technical limitations. Therefore, direct measurement of forces and use of feedback motion/force control for compensation of uncertainties in tendon-based motion and force transmission are not possible. However, force/motion estimation and control in tendon-based robots are important in view of the need for haptic feedback in robotic surgery and growing interest in automatizing common surgical tasks. One possible solution to the above-described problem is the development of mathematical models for tendon-based force and motion transmission that can be used for estimation and control purposes. This thesis provides analysis of force and motion transmission in tendon-pulley based surgical robots and addresses various aspects of the transmission modeling problem. Due to similarities between the quasi-static hysteretic behavior of a tendon-pulley based da Vinci® instrument and that of a typical tendon-sheath mechanism, a distributed friction approach for modeling the force transmission in the instrument is developed. The approach is extended to derive a formula for the apparent stiffness of the instrument. Consequently, a method is developed that uses the formula for apparent stiffness of the instrument to determine the stiffness distribution of the tissue palpated. The force transmission hysteresis is further investigated from a phenomenological point of view. It is shown that a classic Preisach hysteresis model can accurately describe the quasi-static input-output force transmission behavior of the da Vinci® instrument. Also, in order to describe the distributed friction effect in tendon-pulley mechanisms, the creep theory from belt mechanics is adopted for the robotic applications. As a result, a novel motion transmission model is suggested for tendon-pulley mechanisms. The developed model is of pseudo-kinematic type as it relates the output displacement to both the input displacement and the input force. The model is subsequently used for position control of the tip of the instrument. Furthermore, the proposed pseudo-kinematic model is extended to compensate for the coupled-hysteresis effect in a multi-DOF motion. A dynamic transmission model is also suggested that describes system’s response to high frequency inputs. Finally, the proposed motion transmission model was used for modeling of the backlash-like hysteresis in RAVEN II surgical robot

A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery

Providing force feedback as relevant information in current Robot-Assisted Minimally Invasive Surgery systems constitutes a technological challenge due to the constraints imposed by the surgical environment. In this context, force estimation techniques represent a potential solution, enabling to sense the interaction forces between the surgical instruments and soft-tissues. Specifically, if visual feedback is available for observing soft-tissues’ deformation, this feedback can be used to estimate the forces applied to these tissues. To this end, a force estimation model, based on Convolutional Neural Networks and Long-Short Term Memory networks, is proposed in this work. This model is designed to process both, the spatiotemporal information present in video sequences and the temporal structure of tool data (the surgical tool-tip trajectory and its grasping status). A series of analyses are carried out to reveal the advantages of the proposal and the challenges that remain for real applications. This research work focuses on two surgical task scenarios, referred to as pushing and pulling tissue. For these two scenarios, different input data modalities and their effect on the force estimation quality are investigated. These input data modalities are tool data, video sequences and a combination of both. The results suggest that the force estimation quality is better when both, the tool data and video sequences, are processed by the neural network model. Moreover, this study reveals the need for a loss function, designed to promote the modeling of smooth and sharp details found in force signals. Finally, the results show that the modeling of forces due to pulling tasks is more challenging than for the simplest pushing actions.Peer ReviewedPostprint (author's final draft

Deep Causal Learning for Robotic Intelligence

This invited review discusses causal learning in the context of robotic intelligence. The paper introduced the psychological findings on causal learning in human cognition, then it introduced the traditional statistical solutions on causal discovery and causal inference. The paper reviewed recent deep causal learning algorithms with a focus on their architectures and the benefits of using deep nets and discussed the gap between deep causal learning and the needs of robotic intelligence

Dynamic Discriminant Analysis with Applications in Computational Surgery

University of Minnesota Ph.D. dissertation. May 2017. Major: Mechanical Engineering. Advisor: Timothy Kowalewski. 1 computer file (PDF); x, 185 pages.Background: The field of computational surgery involves the use of new technologies to improve surgical safety and patient outcomes. Two open problems in this field include smart surgical tools for identifying tissues via backend sensing, and classifying surgical skill level using laparoscopic tool motion. Prior work in these fields has been impeded by the lack of a dynamic discriminant analysis technique capable of classifying data given systems with overwhelming similarity. Methods: Four new machine learning algorithms were developed (DLS, DPP, RELIEF-RBF, and Intent Vectors). These algorithms were then applied to the open problems within computational surgery. These algorithms are designed with the specific goal of finding regions of data with maximum discriminating information while ignoring regions of similarity or data scarcity. The results of these techniques are contrasted with current machine learning algorithms found in the literature. Results: For the tissue identification problem, results indicate that the proposed DLS algorithm provides better classification than existing methods. For the surgical skill evaluation problem, results indicate that the Intent Vectors approach provides equivalent or better classification accuracy when compared to prior art. Interpretation: The algorithms presented in this work provide a novel approach to the classification of time-series data for systems with overwhelming similarity by focusing on separability maximization while maintaining a tractable training routine and real-time classification for unseen data

Design optimisation of shape memory alloy linear actuator applications

Shape memory alloy (SMA) actuators have drawn much attention and interest in recent decades due to their unique properties; and, are expected to be increasingly integrated within commercial automotive applications. Key advantages of SMA actuators include: potentially simplified construction, whereby the SMA can act as both sensor and actuator simultaneously; compatibility with Joule heating and convective ambient cooling; and, potential mass advantages over competing actuation technologies. These attributes potentially allow for the development of simpler, more reliable and cost effective actuation systems with significant reduction in mechanical complexity and size. SMA is readily available in commercial quantities and exhibits high wear resistance and durability, which make it an ideal candidate for application in automotive grade applications. Despite these identified advantages, SMA actuators are subject to a series of technical challenges associated with:  - Relatively small strain (displacement or stroke)  - Achievable frequency (actuation speed)  - Controllability (and stability)  - Positional accuracy  - Energy efficiency These technical challenges contribute to a relatively low success rate of commercial SMA actuator applications; and, provide motivation for this program to generate relevant research outcomes that enhance the commercialisation of SMA actuators. An extensive literature review of over 500 journal and patent documents was conducted to provide a clear roadmap for the commercial imperatives for SMA design. The formulated research methodology identifies milestones required for achieving the research objectives, which were addressed as research themes. Based on this literature review, the following research themes were identified:  - Design methods to resolve SMA actuator limitations  - Development of simple and practical numerical models for SMA actuator response  - Data for SMA linear actuator design Specific research contributions within these themes are presented within the thesis, with the objective of enhancing the commercial application of shape memory alloy (SMA) linear actuators, and include:  - A comprehensive analysis of SMAs: history, commercial applications, strength and limitations, design challenges and         opportunities.  - A novel investigation of transient heat transfer scenarios for cylindrical systems associated with their crossover and critical radii.  - Development of novel latent heat models for analytical and numerical applications, and proposal of readily applied activation and deactivation charts compatible with the requirements of SMA actuator designers.  - A novel investigation of the morphological effects of SMA-pulley systems (i.e. pulley diameter, SMA and lagging diameter) on structural and functional fatigue

A complex systems approach to education in Switzerland

The insights gained from the study of complex systems in biological, social, and engineered systems enables us not only to observe and understand, but also to actively design systems which will be capable of successfully coping with complex and dynamically changing situations. The methods and mindset required for this approach have been applied to educational systems with their diverse levels of scale and complexity. Based on the general case made by Yaneer Bar-Yam, this paper applies the complex systems approach to the educational system in Switzerland. It confirms that the complex systems approach is valid. Indeed, many recommendations made for the general case have already been implemented in the Swiss education system. To address existing problems and difficulties, further steps are recommended. This paper contributes to the further establishment complex systems approach by shedding light on an area which concerns us all, which is a frequent topic of discussion and dispute among politicians and the public, where billions of dollars have been spent without achieving the desired results, and where it is difficult to directly derive consequences from actions taken. The analysis of the education system's different levels, their complexity and scale will clarify how such a dynamic system should be approached, and how it can be guided towards the desired performance