Simulation Method for the Physical Deformation of a Three-Dimensional Soft Body in Augmented Reality-Based External Ventricular Drainage

Objectives Intraoperative navigation reduces the risk of major complications and increases the likelihood of optimal surgical outcomes. This paper presents an augmented reality (AR)-based simulation technique for ventriculostomy that visualizes brain deformations caused by the movements of a surgical instrument in a three-dimensional brain model. This is achieved by utilizing a position-based dynamics (PBD) physical deformation method on a preoperative brain image. Methods An infrared camera-based AR surgical environment aligns the real-world space with a virtual space and tracks the surgical instruments. For a realistic representation and reduced simulation computation load, a hybrid geometric model is employed, which combines a high-resolution mesh model and a multiresolution tetrahedron model. Collision handling is executed when a collision between the brain and surgical instrument is detected. Constraints are used to preserve the properties of the soft body and ensure stable deformation. Results The experiment was conducted once in a phantom environment and once in an actual surgical environment. The tasks of inserting the surgical instrument into the ventricle using only the navigation information presented through the smart glasses and verifying the drainage of cerebrospinal fluid were evaluated. These tasks were successfully completed, as indicated by the drainage, and the deformation simulation speed averaged 18.78 fps. Conclusions This experiment confirmed that the AR-based method for external ventricular drain surgery was beneficial to clinicians

Interactively Cutting and Constraining Vertices in Meshes Using Augmented Matrices

We present a finite-element solution method that is well suited for interactive simulations of cutting meshes in the regime of linear elastic models. Our approach features fast updates to the solution of the stiffness system of equations to account for real-time changes in mesh connectivity and boundary conditions. Updates are accomplished by augmenting the stiffness matrix to keep it consistent with changes to the underlying model, without refactoring the matrix at each step of cutting. The initial stiffness matrix and its Cholesky factors are used to implicitly form and solve a Schur complement system using an iterative solver. As changes accumulate over many simulation timesteps, the augmented solution method slows down due to the size of the augmented matrix. However, by periodically refactoring the stiffness matrix in a concurrent background process, fresh Cholesky factors that incorporate recent model changes can replace the initial factors. This controls the size of the augmented matrices and provides a way to maintain a fast solution rate as the number of changes to a model grows. We exploit sparsity in the stiffness matrix, the right-hand-side vectors and the solution vectors to compute the solutions fast, and show that the time complexity of the update steps is bounded linearly by the size of the Cholesky factor of the initial matrix. Our complexity analysis and experimental results demonstrate that this approach scales well with problem size. Results for cutting and deformation of 3D linear elastic models are reported for meshes representing the brain, eye, and model problems with element counts up to 167,000; these show the potential of this method for real-time interactivity. An application to limbal incisions for surgical correction of astigmatism, for which linear elastic models and small deformations are sufficient, is included

2D and 3D surface image processing algorithms and their applications

This doctoral dissertation work aims to develop algorithms for 2D image segmentation application of solar filament disappearance detection, 3D mesh simplification, and 3D image warping in pre-surgery simulation. Filament area detection in solar images is an image segmentation problem. A thresholding and region growing combined method is proposed and applied in this application. Based on the filament area detection results, filament disappearances are reported in real time. The solar images in 1999 are processed with this proposed system and three statistical results of filaments are presented. 3D images can be obtained by passive and active range sensing. An image registration process finds the transformation between each pair of range views. To model an object, a common reference frame in which all views can be transformed must be defined. After the registration, the range views should be integrated into a non-redundant model. Optimization is necessary to obtain a complete 3D model. One single surface representation can better fit to the data. It may be further simplified for rendering, storing and transmitting efficiently, or the representation can be converted to some other formats. This work proposes an efficient algorithm for solving the mesh simplification problem, approximating an arbitrary mesh by a simplified mesh. The algorithm uses Root Mean Square distance error metric to decide the facet curvature. Two vertices of one edge and the surrounding vertices decide the average plane. The simplification results are excellent and the computation speed is fast. The algorithm is compared with six other major simplification algorithms. Image morphing is used for all methods that gradually and continuously deform a source image into a target image, while producing the in-between models. Image warping is a continuous deformation of a: graphical object. A morphing process is usually composed of warping and interpolation. This work develops a direct-manipulation-of-free-form-deformation-based method and application for pre-surgical planning. The developed user interface provides a friendly interactive tool in the plastic surgery. Nose augmentation surgery is presented as an example. Displacement vector and lattices resulting in different resolution are used to obtain various deformation results. During the deformation, the volume change of the model is also considered based on a simplified skin-muscle model

Design of Virtual Hands for Natural Interaction in the Metaverse

[EN] The emergence of the Metaverse is raising important questions in the field of human-machine interaction that must be addressed for a successful implementation of the new paradigm. Therefore, the exploration and integration of both technology and human interaction within this new framework are needed. This paper describes an innovative and technically viable proposal for virtual shopping in the fashion field. Virtual hands directly scanned from the real world have been integrated, after a retopology process, in a virtual environment created for the Metaverse, and have been integrated with digital nails. Human interaction with the Metaverse has been carried out through the acquisition of the real posture of the user's hands using an infrared-based sensor and mapping it in its virtualized version, achieving natural identification. The technique has been successfully tested in an immersive shopping experience with the Meta Quest 2 headset as a pilot project, where a transactions mechanism based on the blockchain technology (non-fungible tokens, NFTs) has allowed for the development of a feasible solution for massive audiences. The consumers' reactions were extremely positive, with a total of 250 in-person participants and 120 remote accesses to the Metaverse. Very interesting technical guidelines are raised in this project, the resolution of which may be useful for future implementations.This research is part of the R+D project PID2020-118021RB-I00, funded by MICIN/AEI/10.13039/501100011033, and R+D project UJI-B2022-48 funded by University Jaume I.Cerdá Boluda, J.; Mora, MC.; Lloret Romero, MN.; Scarani, S.; Sastre, J. (2024). Design of Virtual Hands for Natural Interaction in the Metaverse. Sensors. 24(3). https://doi.org/10.3390/s2403074124

Modeling Real-Time 3-D Lung Deformations for Medical Visualization

In this paper, we propose a physics-based and physiology-based approach for modeling real-time deformations of 3-D high-resolution polygonal lung models obtained from high-resolution computed tomography (HRCT) images of normal human subjects. The physics-based deformation operator is nonsymmetric, which accounts for the heterogeneous elastic properties of the lung tissue and spatial-dynamic flow properties of the air. An iterative approach is used to estimate the deformation with the deformation operator initialized based on the regional alveolar expandability, a key physiology-based parameter. The force applied on each surface node is based on the airflow pattern inside the lungs, which is known to be based on the orientation of the human subject. The validation of lung dynamics is done by resimulating the lung deformation and comparing it with HRCT data and computing force applied on each node derived from a 4-D HRCT dataset of a normal human subject using the proposed deformation operator and verifying its gradient with the orientation

Device-based decision-making for adaptation of three-dimensional content

The goal of this research was the creation of an adaptation mechanism for the delivery of three-dimensional content. The adaptation of content, for various network and terminal capabilities - as well as for different user preferences, is a key feature that needs to be investigated. Current state-of-the art research of the adaptation shows promising results for specific tasks and limited types of content, but is still not well-suited for massive heterogeneous environments. In this research, we present a method for transmitting adapted three-dimensional content to multiple target devices. This paper presents some theoretical and practical methods for adapting three-dimensional content, which includes shapes and animation. We also discuss practical details of the integration of our methods into MPEG-21 and MPEG-4 architecture

Geometric Surface Processing and Virtual Modeling

In this work we focus on two main topics "Geometric Surface Processing" and "Virtual Modeling". The inspiration and coordination for most of the research work contained in the thesis has been driven by the project New Interactive and Innovative Technologies for CAD (NIIT4CAD), funded by the European Eurostars Programme. NIIT4CAD has the ambitious aim of overcoming the limitations of the traditional approach to surface modeling of current 3D CAD systems by introducing new methodologies and technologies based on subdivision surfaces in a new virtual modeling framework. These innovations will allow designers and engineers to transform quickly and intuitively an idea of shape in a high-quality geometrical model suited for engineering and manufacturing purposes. One of the objective of the thesis is indeed the reconstruction and modeling of surfaces, representing arbitrary topology objects, starting from 3D irregular curve networks acquired through an ad-hoc smart-pen device. The thesis is organized in two main parts: "Geometric Surface Processing" and "Virtual Modeling". During the development of the geometric pipeline in our Virtual Modeling system, we faced many challenges that captured our interest and opened new areas of research and experimentation. In the first part, we present these theories and some applications to Geometric Surface Processing. This allowed us to better formalize and give a broader understanding on some of the techniques used in our latest advancements on virtual modeling and surface reconstruction. The research on both topics led to important results that have been published and presented in articles and conferences of international relevance