Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach

Robotic-assisted minimally invasive surgeries have gained a lot of popularity over conventional procedures as they offer many benefits to both surgeons and patients. Nonetheless, they still suffer from some limitations that affect their outcome. One of them is the lack of force feedback which restricts the surgeon's sense of touch and might reduce precision during a procedure. To overcome this limitation, we propose a novel force estimation approach that combines a vision based solution with supervised learning to estimate the applied force and provide the surgeon with a suitable representation of it. The proposed solution starts with extracting the geometry of motion of the heart's surface by minimizing an energy functional to recover its 3D deformable structure. A deep network, based on a LSTM-RNN architecture, is then used to learn the relationship between the extracted visual-geometric information and the applied force, and to find accurate mapping between the two. Our proposed force estimation solution avoids the drawbacks usually associated with force sensing devices, such as biocompatibility and integration issues. We evaluate our approach on phantom and realistic tissues in which we report an average root-mean square error of 0.02 N.Peer ReviewedPostprint (author's final draft

Biological, simulation, and robotic studies to discover principles of swimming within granular media

The locomotion of organisms whether by running, flying, or swimming is the result of multiple degree-of-freedom nervous and musculoskeletal systems interacting with an environment that often flows and deforms in response to movement. A major challenge in biology is to understand the locomotion of organisms that crawl or burrow within terrestrial substrates like sand, soil, and muddy sediments that display both solid and fluid-like behavior. In such materials, validated theories such as the Navier-Stokes equations for fluids do not exist, and visualization techniques (such as particle image velocimetry in fluids) are nearly nonexistent. In this dissertation we integrated biological experiment, numerical simulation, and a physical robot model to reveal principles of undulatory locomotion in granular media. First, we used high speed x-ray imaging techniques to reveal how a desert dwelling lizard, the sandfish, swims within dry granular media without limb use by propagating a single period sinusoidal traveling wave along its body, resulting in a wave efficiency, the ratio of its average forward speed to wave speed, of approximately 0.5. The wave efficiency was independent of the media preparation (loosely and tightly packed). We compared this observation against two complementary modeling approaches: a numerical model of the sandfish coupled to a discrete particle simulation of the granular medium, and an undulatory robot which was designed to swim within granular media. We used these mechanical models to vary the ratio of undulation amplitude (A) to wavelength (λ) and demonstrated that an optimal condition for sand-swimming exists which results from competition between A and λ. The animal simulation and robot model, predicted that for a single period sinusoidal wave, maximal speed occurs for A/ λ = 0.2, the same kinematics used by the sandfish. Inspired by the tapered head shape of the sandfish lizard, we showed that the lift forces and hence vertical position of the robot as it moves forward within granular media can be varied by designing an appropriate head shape and controlling its angle of attack, in a similar way to flaps or wings moving in fluids. These results support the biological hypotheses which propose that morphological adaptations of desert dwelling organisms aid in their subsurface locomotion. This work also demonstrates that the discovery of biological principles of high performance locomotion within sand can help create the next generation of biophysically inspired robots that could explore potentially hazardous complex flowing environments.PhDCommittee Chair: Daniel I. Goldman; Committee Member: Hang Lu; Committee Member: Jeanette Yen; Committee Member: Shella Keilholz; Committee Member: Young-Hui Chan

Technical Report: A Contact-aware Feedback CPG System for Learning-based Locomotion Control in a Soft Snake Robot

Integrating contact-awareness into a soft snake robot and efficiently controlling its locomotion in response to contact information present significant challenges. This paper aims to solve contact-aware locomotion problem of a soft snake robot through developing bio-inspired contact-aware locomotion controllers. To provide effective contact information for the controllers, we develop a scale covered sensor structure mimicking natural snakes' \textit{scale sensilla}. In the design of control framework, our core contribution is the development of a novel sensory feedback mechanism of the Matsuoka central pattern generator (CPG) network. This mechanism allows the Matsuoka CPG system to work like a "spine cord" in the whole contact-aware control scheme, which simultaneously takes the stimuli including tonic input signals from the "brain" (a goal-tracking locomotion controller) and sensory feedback signals from the "reflex arc" (the contact reactive controller), and generate rhythmic signals to effectively actuate the soft snake robot to slither through densely allocated obstacles. In the design of the "reflex arc", we develop two types of reactive controllers -- 1) a reinforcement learning (RL) sensor regulator that learns to manipulate the sensory feedback inputs of the CPG system, and 2) a local reflexive sensor-CPG network that directly connects sensor readings and the CPG's feedback inputs in a special topology. These two reactive controllers respectively facilitate two different contact-aware locomotion control schemes. The two control schemes are tested and evaluated in the soft snake robot, showing promising performance in the contact-aware locomotion tasks. The experimental results also further verify the benefit of Matsuoka CPG system in bio-inspired robot controller design.Comment: 17 pages, 19 figure

Reinforcement Learning of CPG-regulated Locomotion Controller for a Soft Snake Robot

Intelligent control of soft robots is challenging due to the nonlinear and difficult-to-model dynamics. One promising model-free approach for soft robot control is reinforcement learning (RL). However, model-free RL methods tend to be computationally expensive and data-inefficient and may not yield natural and smooth locomotion patterns for soft robots. In this work, we develop a bio-inspired design of a learning-based goal-tracking controller for a soft snake robot. The controller is composed of two modules: An RL module for learning goal-tracking behaviors given the unmodeled and stochastic dynamics of the robot, and a central pattern generator (CPG) with the Matsuoka oscillators for generating stable and diverse locomotion patterns. We theoretically investigate the maneuverability of Matsuoka CPG's oscillation bias, frequency, and amplitude for steering control, velocity control, and sim-to-real adaptation of the soft snake robot. Based on this analysis, we proposed a composition of RL and CPG modules such that the RL module regulates the tonic inputs to the CPG system given state feedback from the robot, and the output of the CPG module is then transformed into pressure inputs to pneumatic actuators of the soft snake robot. This design allows the RL agent to naturally learn to entrain the desired locomotion patterns determined by the CPG maneuverability. We validated the optimality and robustness of the control design in both simulation and real experiments, and performed extensive comparisons with state-of-art RL methods to demonstrate the benefit of our bio-inspired control design.Comment: 20 pages, 17 figures, 4 tables, in IEEE Transactions on Robotic