NOViSE: a virtual natural orifice transluminal endoscopic surgery simulator

Purpose: Natural Orifice Transluminal Endoscopic Surgery (NOTES) is a novel technique in minimally invasive surgery whereby a flexible endoscope is inserted via a natural orifice to gain access to the abdominal cavity, leaving no external scars. This innovative use of flexible endoscopy creates many new challenges and is associated with a steep learning curve for clinicians. Methods: We developed NOViSE - the first force-feedback enabled virtual reality simulator for NOTES training supporting a flexible endoscope. The haptic device is custom built and the behaviour of the virtual flexible endoscope is based on an established theoretical framework – the Cosserat Theory of Elastic Rods. Results: We present the application of NOViSE to the simulation of a hybrid trans-gastric cholecystectomy procedure. Preliminary results of face, content and construct validation have previously shown that NOViSE delivers the required level of realism for training of endoscopic manipulation skills specific to NOTES Conclusions: VR simulation of NOTES procedures can contribute to surgical training and improve the educational experience without putting patients at risk, raising ethical issues or requiring expensive animal or cadaver facilities. In the context of an experimental technique, NOViSE could potentially facilitate NOTES development and contribute to its wider use by keeping practitioners up to date with this novel surgical technique. NOViSE is a first prototype and the initial results indicate that it provides promising foundations for further development

Gazebo Plants: Simulating Plant-Robot Interaction with Cosserat Rods

Robotic harvesting has the potential to positively impact agricultural productivity, reduce costs, improve food quality, enhance sustainability, and to address labor shortage. In the rapidly advancing field of agricultural robotics, the necessity of training robots in a virtual environment has become essential. Generating training data to automatize the underlying computer vision tasks such as image segmentation, object detection and classification, also heavily relies on such virtual environments as synthetic data is often required to overcome the shortage and lack of variety of real data sets. However, physics engines commonly employed within the robotics community, such as ODE, Simbody, Bullet, and DART, primarily support motion and collision interaction of rigid bodies. This inherent limitation hinders experimentation and progress in handling non-rigid objects such as plants and crops. In this contribution, we present a plugin for the Gazebo simulation platform based on Cosserat rods to model plant motion. It enables the simulation of plants and their interaction with the environment. We demonstrate that, using our plugin, users can conduct harvesting simulations in Gazebo by simulating a robotic arm picking fruits and achieve results comparable to real-world experiments.Comment: Upon request, we are happy to share our GazeboPlants plugin open-source (MPL 2.0

Modelling and simulation of flexible instruments for minimally invasive surgical training in virtual reality

Improvements in quality and safety standards in surgical training, reduction in training hours and constant technological advances have challenged the traditional apprenticeship model to create a competent surgeon in a patient-safe way. As a result, pressure on training outside the operating room has increased. Interactive, computer based Virtual Reality (VR) simulators offer a safe, cost-effective, controllable and configurable training environment free from ethical and patient safety issues. Two prototype, yet fully-functional VR simulator systems for minimally invasive procedures relying on flexible instruments were developed and validated. NOViSE is the first force-feedback enabled VR simulator for Natural Orifice Transluminal Endoscopic Surgery (NOTES) training supporting a flexible endoscope. VCSim3 is a VR simulator for cardiovascular interventions using catheters and guidewires. The underlying mathematical model of flexible instruments in both simulator prototypes is based on an established theoretical framework – the Cosserat Theory of Elastic Rods. The efficient implementation of the Cosserat Rod model allows for an accurate, real-time simulation of instruments at haptic-interactive rates on an off-the-shelf computer. The behaviour of the virtual tools and its computational performance was evaluated using quantitative and qualitative measures. The instruments exhibited near sub-millimetre accuracy compared to their real counterparts. The proposed GPU implementation further accelerated their simulation performance by approximately an order of magnitude. The realism of the simulators was assessed by face, content and, in the case of NOViSE, construct validity studies. The results indicate good overall face and content validity of both simulators and of virtual instruments. NOViSE also demonstrated early signs of construct validity. VR simulation of flexible instruments in NOViSE and VCSim3 can contribute to surgical training and improve the educational experience without putting patients at risk, raising ethical issues or requiring expensive animal or cadaver facilities. Moreover, in the context of an innovative and experimental technique such as NOTES, NOViSE could potentially facilitate its development and contribute to its popularization by keeping practitioners up to date with this new minimally invasive technique.Open Acces

Harnessing Simulated Data with Graphs

Physically accurate simulations allow for unlimited exploration of arbitrarily crafted environments. From a scientific perspective, digital representations of the real world are useful because they make it easy validate ideas. Virtual sandboxes allow observations to be collected at-will, without intricate setting up for measurements or needing to wait on the manufacturing, shipping, and assembly of physical resources. Simulation techniques can also be utilized over and over again to test the problem without expending costly materials or producing any waste. Remarkably, this freedom to both experiment and generate data becomes even more powerful when considering the rising adoption of data-driven techniques across engineering disciplines. These are systems that aggregate over available samples to model behavior, and thus are better informed when exposed to more data. Naturally, the ability to synthesize limitless data promises to make approaches that benefit from datasets all the more robust and desirable. However, the ability to readily and endlessly produce synthetic examples also introduces several new challenges. Data must be collected in an adaptive format that can capture the complete diversity of states achievable in arbitrary simulated configurations while too remaining amenable to downstream applications. The quantity and zoology of observations must also straddle a range which prevents overfitting but is descriptive enough to produce a robust approach. Pipelines that naively measure virtual scenarios can easily be overwhelmed by trying to sample an infinite set of available configurations. Variations observed across multiple dimensions can quickly lead to a daunting expansion of states, all of which must be processed and solved. These and several other concerns must first be addressed in order to safely leverage the potential of boundless simulated data. In response to these challenges, this thesis proposes to wield graphs in order to instill structure over digitally captured data, and curb the growth of variables. The paradigm of pairing data with graphs introduced in this dissertation serves to enforce consistency, localize operators, and crucially factor out any combinatorial explosion of states. Results demonstrate the effectiveness of this methodology in three distinct areas, each individually offering unique challenges and practical constraints, and together showcasing the generality of the approach. Namely, studies observing state-of-the-art contributions in design for additive manufacturing, side-channel security threats, and large-scale physics based contact simulations are collectively achieved by harnessing simulated datasets with graph algorithms

Multi-Scale Models to Simulate Interactions between Liquid and Thin Structures

In this dissertation, we introduce a framework for simulating the dynamics between liquid and thin structures, including the effects of buoyancy, drag, capillary cohesion, dripping, and diffusion. After introducing related works, Part I begins with a discussion on the interactions between Newtonian fluid and fabrics. In this discussion, we treat both the fluid and the fabrics as continuum media; thus, the physical model is built from mixture theory. In Part II, we discuss the interactions between Newtonian fluid and hairs. To have more detailed dynamics, we no longer treat the hairs as continuum media. Instead, we treat them as discrete Kirchhoff rods. To deal with the thin layer of liquid that clings to the hairs, we augment each hair strand with a height field representation, through which we introduce a new reduced-dimensional flow model to solve the motion of liquid along the longitudinal direction of each hair. In addition, we develop a faithful model for the hairs' cohesion induced by surface tension, where a penalty force is applied to simulate the collision and cohesion between hairs. To enable the discrete strands interact with continuum-based, shear-dependent liquid, in Part III, we develop models that account for the volume change of the liquid as it passes through strands and the momentum exchange between the strands and the liquid. Accordingly, we extend the reduced-dimensional flow model to simulate liquid with elastoviscoplastic behavior. Furthermore, we use a constraint-based model to replace the penalty-force model to handle contact, which enables an accurate simulation of the frictional and adhesive effects between wet strands. We also present a principled method to preserve the total momentum of a strand and its surface flow, as well as an analytic plastic flow approach for Herschel-Bulkley fluid that enables stable semi-implicit integration at larger time steps. We demonstrate a wide range of effects, including the challenging animation scenarios involving splashing, wringing, and colliding of wet clothes, as well as flipping of hair, animals shaking, spinning roller brushes from car washes being dunked in water, and intricate hair coalescence effects. For complex liquids, we explore a series of challenging scenarios, including strands interacting with oil paint, mud, cream, melted chocolate, and pasta sauce

A Massively-Parallel 3D Simulator for Soft and Hybrid Robots

Simulation is an important step in robotics for creating control policies and testing various physical parameters. Soft robotics is a field that presents unique physical challenges for simulating its subjects due to the nonlinearity of deformable material components along with other innovative, and often complex, physical properties. Because of the computational cost of simulating soft and heterogeneous objects with traditional techniques, rigid robotics simulators are not well suited to simulating soft robots. Thus, many engineers must build their own one-off simulators tailored to their system, or use existing simulators with reduced performance. In order to facilitate the development of this exciting technology, this work presents an interactive-speed, accurate, and versatile simulator for a variety of types of soft robots. Cronos, our open-source 3D simulation engine, parallelizes a mass-spring model for ultra-fast performance on both deformable and rigid objects. Our approach is applicable to a wide array of nonlinear material configurations, including high deformability, volumetric actuation, or heterogenous stiffness. This versatility provides the ability to mix materials and geometric components freely within a single robot simulation. By exploiting the flexibility and scalability of nonlinear Hookean mass-spring systems, this framework simulates soft and rigid objects via a highly parallel model for near real-time speed. We describe an efficient GPU CUDA implementation, which we demonstrate to achieve computation of over 1 billion elements per second on consumer-grade GPU cards. Dynamic physical accuracy of the system is validated by comparing results to Euler-Bernoulli beam theory, natural frequency predictions, and empirical data of a soft structure under large deformation

Realistic Hair Simulation: Animation and Rendering

International audienceThe last five years have seen a profusion of innovative solutions to one of the most challenging tasks in character synthesis: hair simulation. This class covers both recent and novel research ideas in hair animation and rendering, and presents time tested industrial practices that resulted in spectacular imagery

A multi-scale model for coupling strands with shear-dependent liquid

We propose a framework for simulating the complex dynamics of strands interacting with compressible, shear-dependent liquids, such as oil paint, mud, cream, melted chocolate, and pasta sauce. Our framework contains three main components: the strands modeled as discrete rods, the bulk liquid represented as a continuum (material point method), and a reduced-dimensional flow of liquid on the surface of the strands with detailed elastoviscoplastic behavior. These three components are tightly coupled together. To enable discrete strands interacting with continuum-based liquid, we develop models that account for the volume change of the liquid as it passes through strands and the momentum exchange between the strands and the liquid. We also develop an extended constraint-based collision handling method that supports cohesion between strands. Furthermore, we present a principled method to preserve the total momentum of a strand and its surface flow, as well as an analytic plastic flow approach for Herschel-Bulkley fluid that enables stable semi-implicit integration at larger time steps. We explore a series of challenging scenarios, involving splashing, shaking, and agitating the liquid which causes the strands to stick together and become entangled.This work was supported in part by the National Science Foundation under Grant Nos.: 1717178, 1319483, CAREER-1453101, the Natu- ral Sciences and Engineering Research Council of Canada under Grant No. RGPIN-04360-2014, SoftBank Group, Pixar, Adobe, and SideFX

Modélisation dynamique inverse de tissus - Apprentissage profond à l'aide de simulations basées sur la physique

Inverse problems arise in various physical domains and solving them from real-world visual observations poses a significant challenge due to the high dimensional nature of the data. Furthermore gathering enough observations that a data driven model can accurately capture the complete distribution of a physical phenomenon is often intractable. In this work we use deep learning to solve inverse problems by applying two basic principles. Deep learning models can be trained using synthetic data generated from physics based simulations. And the employed simulator itself needs to be verified for physical accuracy thus allowing the model to learn the exact physical phenomenon that is desired.To validate the simulator, we introduce rich and compact physical protocols, originally proposed in soft matter physics literature to measure physical parameters. These protocols can be easily replicated in a simulator to test the physical correctness of the model, and the validity of the simulator.We solve the inverse measurement problem of estimating contact friction in soft-bodies which otherwise requires a specialized physics bench and entails tedious acquisition protocols. This makes the prospect of a purely non-invasive, video-based measurement technique particularly attractive. Previous works have shown that such a video-based estimation is feasible for material parameters using deep learning, but this has never been applied to the friction estimation problem which results in even more subtle visual variations. Since acquiring a large dataset for this problem is impractical, we generate it using a frictional contact simulator. As the simulator has been calibrated and verified using controlled experiments, the results are not only visually plausible, but physically-correct enough to match observations made at the macroscopic scale. We propose to our knowledge the first non-invasive measurement network and adjoining synthetic training dataset for estimating cloth friction at contact, for both cloth-hard body and cloth-cloth contacts. We also acquire an extensive dataset of real world experiments for testing. Both the training and test datasets have been made freely available to the community.We also utilize the same protocol for solving the inverse measurement problem of estimating the deformed curvature of a suspended Kirchhoff rod. In order to do such estimation on physical rods, we utilize a deep learning model to visually predict a curvature field from a suspended rod. As creating a dataset from physical rods (even if synthetically constructed), that faithfully covers a representative manifold of deformed curvatures is intractable, we rely on generating such a dataset from a verified simulator. Our work shows a promising way forward for utilizing deep learning models as part of an inversion measurement pipeline.Des problèmes inverses surviennent dans divers domaines physiques et les résoudre à partir d'observations visuelles du monde réel pose un défi important en raison de la nature hautement dimensionnelle des données. De plus, rassembler suffisamment d'observations pour qu'un modèle basé sur les données puisse capturer avec précision la distribution complète d'un phénomène physique est souvent insoluble. Dans ce travail, nous utilisons l'apprentissage profond pour résoudre des problèmes inverses en appliquant deux principes de base. Les modèles d'apprentissage profond peuvent être entraînés à l'aide de données synthétiques générées à partir de simulations basées sur la physique. Et la précision physique du simulateur employé, lui-même, doit être vérifiée, permettant ainsi au modèle d'apprendre le phénomène physique exact souhaité.Afin de valider le simulateur, nous introduisons des protocoles physiques riches et compacts, proposés à l'origine dans la littérature de physique de la matière molle pour mesurer des paramètres physiques. Ces protocoles peuvent être facilement répliqués dans un simulateur pour tester l'exactitude physique du modèle et la validité du simulateur.Nous résolvons le problème de mesure inverse de l'estimation du frottement de contact dans les corps mous qui nécessite sinon un banc de physique spécialisé et un protocole d'acquisition fastidieux. Cela rend la perspective d'une technique de mesure purement non invasive basée sur la vidéo particulièrement attrayante. Des travaux antérieurs ont montré qu'une telle estimation basée sur la vidéo est réalisable pour les paramètres de matériaux en utilisant l'apprentissage profond, mais cela n'a jamais été appliqué au problème d'estimation de la friction qui entraîne des variations visuelles encore plus subtiles. Étant donné qu'il n'est pas pratique d'acquérir un grand ensemble de données pour ce problème, nous le générons à l'aide d'un simulateur de contact frictionnel. Comme le simulateur a été calibré et vérifié à l'aide d'expériences contrôlées, les résultats sont non seulement visuellement plausibles, mais suffisamment corrects physiquement pour correspondre aux observations faites à l'échelle macroscopique. Nous proposons à notre connaissance le premier réseau de mesure non invasif et un jeu de données d'entraînement synthétique adjacent pour estimer le frottement du tissu au contact, à la fois pour les contacts tissu-corps dur et tissu-tissu. Nous acquérons également un vaste ensemble de données d'expériences du monde réel pour les tests. Les ensembles de données de formation et de test ont été mis gratuitement à la disposition de la communauté.Nous utilisons également le même protocole pour résoudre le problème de mesure inverse de l'estimation de la courbure déformée d'une tige de Kirchhoff suspendue. Afin de faire une telle estimation sur des tiges physiques, nous utilisons un modèle d'apprentissage profond pour prédire visuellement un champ de courbure à partir d'une tige suspendue. Comme la création d'un ensemble de données à partir de tiges physiques (même si elles sont synthétiquement construites), qui couvre fidèlement une variété représentative de courbures déformées est insoluble, nous comptons sur la génération d'un tel ensemble de données à partir d'un simulateur vérifié. Notre travail montre une voie prometteuse pour l'utilisation de modèles d'apprentissage profond dans le cadre d'un pipeline de mesure d'inversion