Surgical Augmented Reality with Topological Changes

International audienceThe visualization of internal structures of organs in minimally invasive surgery is an important avenue for improving the perception of the surgeon, or for supporting planning and decision systems. However, current methods dealing with non-rigid augmented reality only provide augmentation when the topology of the organ is not modified. In this paper we solve this shortcoming by introducing a method for physics-based non-rigid augmented reality. Singularities caused by topo-logical changes are detected and propagated to the pre-operative model. This significantly improves the coherence between the actual laparascopic view and the model, and provides added value in terms of navigation and decision making. Our real time augmentation algorithm is assessed on a video showing the cut of a porcine liver's lobe in minimal invasive surgery

Handling Topological Changes during Elastic Registration: Application to Augmented Reality in Laparoscopic Surgery

International audiencePurpose: Locating the internal structures of an organ is a critical aspect of many surgical procedures. Minimally invasive surgery, associated with augmented reality techniques, offers the potential to visualize inner structures, allowing for improved analysis, depth perception or for supporting planning and decision systems.Methods: Most of the current methods dealing with rigid or non-rigid augmented reality make the assumption that the topology of the organ is not modified. As surgery relies essentially on cutting and dissection of anatomical structures, such methods are limited to the early stages of the surgery.We solve this shortcoming with the introduction of a method for physics-based elastic registration using a single view from a monocular camera.Singularities caused by topological changes are detected and propagated to the pre-operative model. This significantly improves the coherence between the actual laparoscopic view and the model, and provides added value in terms of navigation and decision-making, e.g. by overlaying the internal structures of an organ on the laparoscopic view.Results: Our real time augmentation method is assessed on several scenarios, using synthetic objects and real organs. In all cases, the impact of our approach is demonstrated, both qualitatively and quantitatively.Conclusion: The presented approach tackles the challenge of localizing internal structures throughout a complete surgical procedure, even after surgical cuts. This information is crucial for surgeons to improve the outcome for their surgical procedure and avoid complications

Interactively Cutting and Constraining Vertices in Meshes Using Augmented Matrices

We present a finite-element solution method that is well suited for interactive simulations of cutting meshes in the regime of linear elastic models. Our approach features fast updates to the solution of the stiffness system of equations to account for real-time changes in mesh connectivity and boundary conditions. Updates are accomplished by augmenting the stiffness matrix to keep it consistent with changes to the underlying model, without refactoring the matrix at each step of cutting. The initial stiffness matrix and its Cholesky factors are used to implicitly form and solve a Schur complement system using an iterative solver. As changes accumulate over many simulation timesteps, the augmented solution method slows down due to the size of the augmented matrix. However, by periodically refactoring the stiffness matrix in a concurrent background process, fresh Cholesky factors that incorporate recent model changes can replace the initial factors. This controls the size of the augmented matrices and provides a way to maintain a fast solution rate as the number of changes to a model grows. We exploit sparsity in the stiffness matrix, the right-hand-side vectors and the solution vectors to compute the solutions fast, and show that the time complexity of the update steps is bounded linearly by the size of the Cholesky factor of the initial matrix. Our complexity analysis and experimental results demonstrate that this approach scales well with problem size. Results for cutting and deformation of 3D linear elastic models are reported for meshes representing the brain, eye, and model problems with element counts up to 167,000; these show the potential of this method for real-time interactivity. An application to limbal incisions for surgical correction of astigmatism, for which linear elastic models and small deformations are sufficient, is included

Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems

Background: Compared with traditional thoracotomy, video-assisted thoracoscopic surgery (VATS) has less minor trauma, faster recovery, higher patient compliance, but higher requirements for surgeons. Virtual surgery training simulation systems are important and have been widely used in Europe and America. Augmented reality (AR) in surgical training simulation systems significantly improve the training effect of virtual surgical training, although AR technology is still in its initial stage. Mixed reality has gained increased attention in technology-driven modern medicine but has yet to be used in everyday practice. Methods: This study proposed an immersive AR lobectomy within a thoracoscope surgery training system, using visual and haptic modeling to study the potential benefits of this critical technology. The content included immersive AR visual rendering, based on the cluster-based extended position-based dynamics algorithm of soft tissue physical modeling. Furthermore, we designed an AR haptic rendering systems, whose model architecture consisted of multi-touch interaction points, including kinesthetic and pressure-sensitive points. Finally, based on the above theoretical research, we developed an AR interactive VATS surgical training platform. Results: Twenty-four volunteers were recruited from the First People's Hospital of Yunnan Province to evaluate the VATS training system. Face, content, and construct validation methods were used to assess the tactile sense, visual sense, scene authenticity, and simulator performance. Conclusions: The results of our construction validation demonstrate that the simulator is useful in improving novice and surgical skills that can be retained after a certain period of time. The video-assisted thoracoscopic system based on AR developed in this study is effective and can be used as a training device to assist in the development of thoracoscopic skills for novices

Real-time Error Control for Surgical Simulation

Objective: To present the first real-time a posteriori error-driven adaptive finite element approach for real-time simulation and to demonstrate the method on a needle insertion problem. Methods: We use corotational elasticity and a frictional needle/tissue interaction model. The problem is solved using finite elements within SOFA. The refinement strategy relies upon a hexahedron-based finite element method, combined with a posteriori error estimation driven local $h$-refinement, for simulating soft tissue deformation. Results: We control the local and global error level in the mechanical fields (e.g. displacement or stresses) during the simulation. We show the convergence of the algorithm on academic examples, and demonstrate its practical usability on a percutaneous procedure involving needle insertion in a liver. For the latter case, we compare the force displacement curves obtained from the proposed adaptive algorithm with that obtained from a uniform refinement approach. Conclusions: Error control guarantees that a tolerable error level is not exceeded during the simulations. Local mesh refinement accelerates simulations. Significance: Our work provides a first step to discriminate between discretization error and modeling error by providing a robust quantification of discretization error during simulations.Comment: 12 pages, 16 figures, change of the title, submitted to IEEE TBM

A Surgery Issue: Cutting through the Architectural Fabric

This essay examines the material ontologies of surgical trans-modifications. Focusing on incisions and subsequent scars, the essay argues for the queering of architecture and design as an act of cutting through structures and processes. I start by rereading a topological body plan, used by surgeons as a guide for performing incisions. I suggest that this plan constitutes a variation within topological representations. It is thus reconceptualised as an internally contradictory representation, calling for dis/continuous cutting acts upon the represented body; that is an amphi-topological representation. The notion of the cut is further approached from the point of view of queer theory and ‘agential realism’. This perspective offers affirmative ways of discussing about acts of cutting. When performed into self-organizing fabrics, cuts appear to act as ‘agents of dis/continuity’. Then, the discussion passes through the genealogy of the architectural section and the building cuts of Gordon Matta Clark, so as to show that the production of buildings by ‘cutting through them to come to matter’ is deep-rooted in the architectural discipline. Next, the philosophical origins of the idea of cutting through the material fabric of the world are sought. It is argued that the latter, beginning as a gnosiological tool, was transformed into a fabricating tool and then into a tool of smoothing the striated. The essay concludes with the presentation of SCARchiCAD; a computational design tool which takes the skin’s wound healing process as a model, offering an interpretation of what the ‘cutting through a virtual form’ could suggest for the design of architectural bodies and the queering of architecture

Laparoscopic Video Analysis for Training and Image Guided Surgery

Automatic analysis of Minimally Invasive Surgical video has the potential to drive new solutions for alleviating needs of safe and reproducible training programs, objective and transparent evaluation systems and navigation tools to assist surgeons and improve patient safety. Surgical video is an always available source of information, which can be used without any additional intrusive hardware in the operating room. This paper is focused on surgical video analysis methods and techniques. It describes authors' contributions in two key aspects, the 3D reconstruction of the surgical field and the segmentation and tracking of tools and organs based on laparoscopic video images. Results are given to illustrate the potential of this field of research, like the calculi of the 3D position and orientation of a tool from its 2D image, or the translation of a preoperative resection plan into a hepatectomy surgical procedure using the shading information of the image. Research efforts are required to further develop these technologies in order to harness all the valuable information available in any video-based surgery

Real Estate using AR, VR, DR, Geo-referencing and other mobile applications

Purpose - The use of Mobile Applications, Geo-referencing, Augmented Reality (AR) or Virtual Reality (VR) brings knowledge and understanding of how we possible deal with Real Estate (RE) nowadays. Clients want fast access to information, real details, and sometimes a sample of a tour.Design/methodology/approach – In this paper, first it is analysed in some interviews with owners of small and medium enterprises in RA. Then it is presented some pieces of evidence on the use of those technologies and applications. Finally, we point out some conclusions, some limitations and recommendations for future studies.Findings – The AR and VR technologies and new applications for personal computers (PCs) or mobile engines are rising sharply in the area of medicine. We also see some advances in architecture and the exhibition of properties, in particular in views of interiors, guide visits to apartments, or houses. These issues can easily be into the RE.Practical implications – Including Mobile Applications, Geo-referencing, AR or VR in RE business we can expect a better understanding of what consumers want, accurate details or proximity with a real word in a near future.Originality/value – We have few investigations both in RE and very new Technologies and Mobile Applications, mainly looking towards clients. Few articles are referring to those applications in RE and considering Portugal studies are rare. This kind of research should be more frequent and compared, as possible, with other countries and other situations