A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery

Providing force feedback as relevant information in current Robot-Assisted Minimally Invasive Surgery systems constitutes a technological challenge due to the constraints imposed by the surgical environment. In this context, force estimation techniques represent a potential solution, enabling to sense the interaction forces between the surgical instruments and soft-tissues. Specifically, if visual feedback is available for observing soft-tissues’ deformation, this feedback can be used to estimate the forces applied to these tissues. To this end, a force estimation model, based on Convolutional Neural Networks and Long-Short Term Memory networks, is proposed in this work. This model is designed to process both, the spatiotemporal information present in video sequences and the temporal structure of tool data (the surgical tool-tip trajectory and its grasping status). A series of analyses are carried out to reveal the advantages of the proposal and the challenges that remain for real applications. This research work focuses on two surgical task scenarios, referred to as pushing and pulling tissue. For these two scenarios, different input data modalities and their effect on the force estimation quality are investigated. These input data modalities are tool data, video sequences and a combination of both. The results suggest that the force estimation quality is better when both, the tool data and video sequences, are processed by the neural network model. Moreover, this study reveals the need for a loss function, designed to promote the modeling of smooth and sharp details found in force signals. Finally, the results show that the modeling of forces due to pulling tasks is more challenging than for the simplest pushing actions.Peer ReviewedPostprint (author's final draft

Modeling of Force and Motion Transmission in Tendon-Driven Surgical Robots

Tendon-based transmission is a common approach for transferring motion and forces in surgical robots. In spite of design simplicity and compactness that comes with the tendon drives, there exists a number of issues associated with the tendon-based transmission. In particular, the elasticity of the tendons and the frictional interaction between the tendon and the routing result in substantially nonlinear behavior. Also, in surgical applications, the distal joints of the robot and instruments cannot be sensorized in most cases due to technical limitations. Therefore, direct measurement of forces and use of feedback motion/force control for compensation of uncertainties in tendon-based motion and force transmission are not possible. However, force/motion estimation and control in tendon-based robots are important in view of the need for haptic feedback in robotic surgery and growing interest in automatizing common surgical tasks. One possible solution to the above-described problem is the development of mathematical models for tendon-based force and motion transmission that can be used for estimation and control purposes. This thesis provides analysis of force and motion transmission in tendon-pulley based surgical robots and addresses various aspects of the transmission modeling problem. Due to similarities between the quasi-static hysteretic behavior of a tendon-pulley based da Vinci® instrument and that of a typical tendon-sheath mechanism, a distributed friction approach for modeling the force transmission in the instrument is developed. The approach is extended to derive a formula for the apparent stiffness of the instrument. Consequently, a method is developed that uses the formula for apparent stiffness of the instrument to determine the stiffness distribution of the tissue palpated. The force transmission hysteresis is further investigated from a phenomenological point of view. It is shown that a classic Preisach hysteresis model can accurately describe the quasi-static input-output force transmission behavior of the da Vinci® instrument. Also, in order to describe the distributed friction effect in tendon-pulley mechanisms, the creep theory from belt mechanics is adopted for the robotic applications. As a result, a novel motion transmission model is suggested for tendon-pulley mechanisms. The developed model is of pseudo-kinematic type as it relates the output displacement to both the input displacement and the input force. The model is subsequently used for position control of the tip of the instrument. Furthermore, the proposed pseudo-kinematic model is extended to compensate for the coupled-hysteresis effect in a multi-DOF motion. A dynamic transmission model is also suggested that describes system’s response to high frequency inputs. Finally, the proposed motion transmission model was used for modeling of the backlash-like hysteresis in RAVEN II surgical robot

Deep Causal Learning for Robotic Intelligence

This invited review discusses causal learning in the context of robotic intelligence. The paper introduced the psychological findings on causal learning in human cognition, then it introduced the traditional statistical solutions on causal discovery and causal inference. The paper reviewed recent deep causal learning algorithms with a focus on their architectures and the benefits of using deep nets and discussed the gap between deep causal learning and the needs of robotic intelligence

Dynamic Discriminant Analysis with Applications in Computational Surgery

University of Minnesota Ph.D. dissertation. May 2017. Major: Mechanical Engineering. Advisor: Timothy Kowalewski. 1 computer file (PDF); x, 185 pages.Background: The field of computational surgery involves the use of new technologies to improve surgical safety and patient outcomes. Two open problems in this field include smart surgical tools for identifying tissues via backend sensing, and classifying surgical skill level using laparoscopic tool motion. Prior work in these fields has been impeded by the lack of a dynamic discriminant analysis technique capable of classifying data given systems with overwhelming similarity. Methods: Four new machine learning algorithms were developed (DLS, DPP, RELIEF-RBF, and Intent Vectors). These algorithms were then applied to the open problems within computational surgery. These algorithms are designed with the specific goal of finding regions of data with maximum discriminating information while ignoring regions of similarity or data scarcity. The results of these techniques are contrasted with current machine learning algorithms found in the literature. Results: For the tissue identification problem, results indicate that the proposed DLS algorithm provides better classification than existing methods. For the surgical skill evaluation problem, results indicate that the Intent Vectors approach provides equivalent or better classification accuracy when compared to prior art. Interpretation: The algorithms presented in this work provide a novel approach to the classification of time-series data for systems with overwhelming similarity by focusing on separability maximization while maintaining a tractable training routine and real-time classification for unseen data

A complex systems approach to education in Switzerland

The insights gained from the study of complex systems in biological, social, and engineered systems enables us not only to observe and understand, but also to actively design systems which will be capable of successfully coping with complex and dynamically changing situations. The methods and mindset required for this approach have been applied to educational systems with their diverse levels of scale and complexity. Based on the general case made by Yaneer Bar-Yam, this paper applies the complex systems approach to the educational system in Switzerland. It confirms that the complex systems approach is valid. Indeed, many recommendations made for the general case have already been implemented in the Swiss education system. To address existing problems and difficulties, further steps are recommended. This paper contributes to the further establishment complex systems approach by shedding light on an area which concerns us all, which is a frequent topic of discussion and dispute among politicians and the public, where billions of dollars have been spent without achieving the desired results, and where it is difficult to directly derive consequences from actions taken. The analysis of the education system's different levels, their complexity and scale will clarify how such a dynamic system should be approached, and how it can be guided towards the desired performance