Single View Augmentation of 3D Elastic Objects

International audienceThis paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a me chanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young's modulus where we show in experiments that a rough estimate of its value is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided

Simulation of hyperelastic materials in real-time using Deep Learning

The finite element method (FEM) is among the most commonly used numerical methods for solving engineering problems. Due to its computational cost, various ideas have been introduced to reduce computation times, such as domain decomposition, parallel computing, adaptive meshing, and model order reduction. In this paper we present U-Mesh: a data-driven method based on a U-Net architecture that approximates the non-linear relation between a contact force and the displacement field computed by a FEM algorithm. We show that deep learning, one of the latest machine learning methods based on artificial neural networks, can enhance computational mechanics through its ability to encode highly non-linear models in a compact form. Our method is applied to two benchmark examples: a cantilever beam and an L-shape subject to moving punctual loads. A comparison between our method and proper orthogonal decomposition (POD) is done through the paper. The results show that U-Mesh can perform very fast simulations on various geometries, mesh resolutions and number of input forces with very small errors

Real-time Biomechanical Modeling for Intraoperative Soft Tissue Registration

Computer assisted surgery systems intraoperatively support the surgeon by providing information on the location of hidden risk and target structures during surgery. However, soft tissue deformations make intraoperative registration (and thus intraoperative navigation) difficult. In this work, a novel, biomechanics based approach for real-time soft tissue registration from sparse intraoperative sensor data such as stereo endoscopic images is presented to overcome this problem

Calipso: Physics-based Image and Video Editing through CAD Model Proxies

We present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. Running these simulations allows us to apply new, unseen forces to move or deform selected objects, change physical parameters such as mass or elasticity, or even add entire new objects that interact with the rest of the underlying scene. In Calipso, the user makes edits directly in 3D; these edits are processed by the simulation and then transfered to the target 2D content using shape-to-image correspondences in a photo-realistic rendering process. To align the CAD models, we introduce an efficient CAD-to-image alignment procedure that jointly minimizes for rigid and non-rigid alignment while preserving the high-level structure of the input shape. Moreover, the user can choose to exploit image flow to estimate scene motion, producing coherent physical behavior with ambient dynamics. We demonstrate Calipso's physics-based editing on a wide range of examples producing myriad physical behavior while preserving geometric and visual consistency.Comment: 11 page

Nouvelles méthodes numériques pour la simulation temps-réel des déformations des tissus mous dans le cadre de l’assistance peropératoire

This thesis addresses the problem soft tissue simulation for augmented reality applications in liver surgery assistance and, more specifically, the implementation of a non-rigid registration pipeline to be used by the medical staff to generate interactive deformations of a patient specific liver three-dimensional virtual representation. A formal physics-based framework is first defined and used as the basis for the construction of a biomechanical model capable of producing realistic deformations. Four basic requirements guided the development of the model: accuracy, speed, stability and simplicity of implementation. Meshless and immersed-boundary methods are both considered as alternatives to the traditional finite element method. A formal non-rigid registration algorithm is finally documented and tested with real-life scenarios. A comparison with new and rising machine learning and neural network solutions is also provided.Cette thèse aborde le problème de simulation des tissus mous pour les applications de réalité augmentée en assistance peropératoire du foie et, plus précisément, la mise en oeuvre d'une procédure automatique de recalage non rigide entre une reconstruction préopératoire du foie d'un patient et les données acquises en temps réel pendant la chirurgie. Un cadre formel basé sur la physique est d'abord défini et utilisé comme base pour la construction d'un modèle biomécanique capable de reproduire les déformations du foie. Quatre directives de recherche ont guidé le développement du modèle : la précision, la rapidité, la stabilité et la simplicité de mise en oeuvre. Les méthodes sans maillage et les méthodes aux frontières immergées sont deux considérées comme des alternatives à la méthode traditionnelle des éléments finis. Un algorithme complet de recalage non rigide est documenté et testé avec des scénarios réels. Finalement, une introduction des émergentes en apprentissage automatique et réseaux de neurones est également fournie

Real-time simulation of soft tissue deformation for surgical simulation

Surgical simulation plays an important role in the training, planning and evaluation of many surgical procedures. It requires realistic and real-time simulation of soft tissue deformation under interaction with surgical tools. However, it is challenging to satisfy both of these conflicting requirements. On one hand, biological soft tissues are complex in terms of material compositions, structural formations, and mechanical behaviours, resulting in nonlinear deformation characteristics under an external load. Due to the involvement of both material and geometric nonlinearities, the use of nonlinear elasticity causes a highly expensive computational load, leading to the difficulty to achieve the real-time computational performance required by surgical simulation. On the other hand, in order to satisfy the real-time computational requirement, most of the existing methods are mainly based on linear elasticity under the assumptions of small deformation and homogeneity to describe deformation of soft tissues. Such simplifications allow reduced runtime computation; however, they are inadequate for modelling nonlinear material properties such as anisotropy, heterogeneity and large deformation of soft tissues. In general, the two conflicting requirements of surgical simulation raise immense complexity in modelling of soft tissue deformation. This thesis focuses on establishment of new methodologies for modelling of soft tissue deformation for surgical simulation. Due to geometric and material nonlinearities in soft tissue deformation, the existing methods have only limited capabilities in achieving nonlinear soft tissue deformation in real-time. In this thesis, the main focus is devoted to the real-time and realistic modelling of nonlinear soft tissue deformation for surgical simulation. New methodologies, namely new ChainMail algorithms, energy propagation method, and energy balance method, are proposed to address soft tissue deformation. Results demonstrate that the proposed methods can simulate the typical soft tissue mechanical properties, accommodate isotropic and homogeneous, anisotropic and heterogeneous materials, handle incompressibility and viscoelastic behaviours, conserve system energy, and achieve realistic, real-time and stable deformation. In the future, it is projected to extend the proposed methodologies to handle surgical operations, such as cutting, joining and suturing, for topology changes occurred in surgical simulation

A comprehensive assessment of accuracy of adaptive integration of cut cells for laminar fluid-structure interaction problems

Finite element methods based on cut-cells are becoming increasingly popular because of their advantages over formulations based on body-fitted meshes for problems with moving interfaces. In such methods, the cells (or elements) which are cut by the interface between two different domains need to be integrated using special techniques in order to obtain optimal convergence rates and accurate fluxes across the interface. The adaptive integration technique in which the cells are recursively subdivided is one of the popular techniques for the numerical integration of cut-cells due to its advantages over tessellation, particularly for problems involving complex geometries in three dimensions. Although adaptive integration does not impose any limitations on the representation of the geometry of immersed solids as it requires only point location algorithms, it becomes computationally expensive for recovering optimal convergence rates. This paper presents a comprehensive assessment of the adaptive integration of cut-cells for applications in computational fluid dynamics and fluid-structure interaction. We assess the effect of the accuracy of integration of cut cells on convergence rates in velocity and pressure fields, and then on forces and displacements for fluid-structure interaction problems by studying several examples in two and three dimensions. By taking the computational cost and the accuracy of forces and displacements into account, we demonstrate that numerical results of acceptable accuracy for FSI problems involving laminar flows can be obtained with only fewer levels of refinement. In particular, we show that three levels of adaptive refinement are sufficient for obtaining force and displacement values of acceptable accuracy for laminar fluid-structure interaction problems

A comprehensive assessment of accuracy of adaptive integration of cut cells for laminar fluid-structure interaction problems

Finite element methods based on cut-cells are becoming increasingly popular because of their advantages over formulations based on body-fitted meshes for problems with moving interfaces. In such methods, the cells (or elements) which are cut by the interface between two different domains need to be integrated using special techniques in order to obtain optimal convergence rates and accurate fluxes across the interface. The adaptive integration technique in which the cells are recursively subdivided is one of the popular techniques for the numerical integration of cut-cells due to its advantages over tessellation, particularly for problems involving complex geometries in three dimensions. Although adaptive integration does not impose any limitations on the representation of the geometry of immersed solids as it requires only point location algorithms, it becomes computationally expensive for recovering optimal convergence rates. This paper presents a comprehensive assessment of the adaptive integration of cut-cells for applications in computational fluid dynamics and fluid-structure interaction. We assess the effect of the accuracy of integration of cut cells on convergence rates in velocity and pressure fields, and then on forces and displacements for fluid-structure interaction problems by studying several examples in two and three dimensions. By taking the computational cost and the accuracy of forces and displacements into account, we demonstrate that numerical results of acceptable accuracy for FSI problems involving laminar flows can be obtained with only fewer levels of refinement. In particular, we show that three levels of adaptive refinement are sufficient for obtaining force and displacement values of acceptable accuracy for laminar fluid-structure interaction problems