Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions

Non-rigid registration under anisotropic deformations

Non-rigid registration of deformed 3D shapes is a challenging and fundamental task in geometric processing, which aims to non-rigidly deform a source shape into alignment with a target shape. Current state-of-the-art methods assume deformations to be near-isometric. This assumption does not reflect real-world conditions, for example in large-scale deformation, where moderate anisotropic deformations (e.g., stretches) are common. In this paper we propose two significant changes to a typical registration pipeline to address such challenging deformations. First, we introduce a method to estimate anisotropic non-isometric deformations and incorporate this into an iterative non-rigid registration pipeline. Second, we compute additional correspondences in non-isometrically deforming regions using reliable correspondences as landmarks and prune inconsistent correspondences. We compare the performance of our proposed algorithm to several state-of-the-art methods using existing benchmarks. Experimental results show that our method outperforms existing methods

Real Time Sequential Non Rigid Structure from motion using a single camera

En la actualidad las aplicaciones que basan su funcionamiento en una correcta localización y reconstrucción dentro de un entorno real en 3D han experimentado un gran interés en los últimos años, tanto por la comunidad investigadora como por la industrial. Estas aplicaciones varían desde la realidad aumentada, la robótica, la simulación, los videojuegos, etc. Dependiendo de la aplicación y del nivel de detalle de la reconstrucción, se emplean diversos dispositivos, algunos específicos, más complejos y caros como las cámaras estéreo, cámara y profundidad (RGBD) con Luz estructurada y Time of Flight (ToF), así como láser y otros más avanzados. Para aplicaciones sencillas es suficiente con dispositivos de uso común, como los smartphones, en los que aplicando técnicas de visión artificial, se pueden obtener modelos 3D del entorno para, en el caso de la realidad aumentada, mostrar información aumentada en la ubicación seleccionada.En robótica, la localización y generación simultáneas de un mapa del entorno en 3D es una tarea fundamental para conseguir la navegación autónoma. Este problema se conoce en el estado del arte como Simultaneous Localization And Mapping (SLAM) o Structure from Motion (SfM). Para la aplicación de estas técnicas, el objeto no ha de cambiar su forma a lo largo del tiempo. La reconstrucción es unívoca salvo factor de escala en captura monocular sin referencia. Si la condición de rigidez no se cumple, es porque la forma del objeto cambia a lo largo del tiempo. El problema sería equivalente a realizar una reconstrucción por fotograma, lo cual no se puede hacer de manera directa, puesto que diferentes formas, combinadas con diferentes poses de cámara pueden dar proyecciones similares. Es por esto que el campo de la reconstrucción de objetos deformables es todavía un área en desarrollo. Los métodos de SfM se han adaptado aplicando modelos físicos, restricciones temporales, espaciales, geométricas o de otros tipos para reducir la ambigüedad en las soluciones, naciendo así las técnicas conocidas como Non-Rigid SfM (NRSfM).En esta tesis se propone partir de una técnica de reconstrucción rígida bien conocida en el estado del arte como es PTAM (Parallel Tracking and Mapping) y adaptarla para incluir técnicas de NRSfM, basadas en modelo de bases lineales para estimar las deformaciones del objeto modelado dinámicamente y aplicar restricciones temporales y espaciales para mejorar las reconstrucciones, además de ir adaptándose a cambios de deformación que se presenten en la secuencia. Para ello, hay que realizar cambios de manera que cada uno de sus hilos de ejecución procesen datos no rígidos.El hilo encargado del seguimiento ya realizaba seguimiento basado en un mapa de puntos 3D, proporcionado a priori. La modificación más importante aquí es la integración de un modelo de deformación lineal para que se realice el cálculo de la deformación del objeto en tiempo real, asumiendo fijas las formas básicas de deformación. El cálculo de la pose de la cámara está basado en el sistema de estimación rígido, por lo que la estimación de pose y coeficientes de deformación se hace de manera alternada usando el algoritmo E-M (Expectation-Maximization). También, se imponen restricciones temporales y de forma para restringir las ambigüedades inherentes en las soluciones y mejorar la calidad de la estimación 3D.Respecto al hilo que gestiona el mapa, se actualiza en función del tiempo para que sea capaz de mejorar las bases de deformación cuando éstas no son capaces de explicar las formas que se ven en las imágenes actuales. Para ello, se sustituye la optimización de modelo rígido incluida en este hilo por un método de procesamiento exhaustivo NRSfM, para mejorar las bases acorde a las imágenes con gran error de reconstrucción desde el hilo de seguimiento. Con esto, el modelo se consigue adaptar a nuevas deformaciones, permitiendo al sistema evolucionar y ser estable a largo plazo.A diferencia de una gran parte de los métodos de la literatura, el sistema propuesto aborda el problema de la proyección perspectiva de forma nativa, minimizando los problemas de ambigüedad y de distancia al objeto existente en la proyección ortográfica. El sistema propuesto maneja centenares de puntos y está preparado para cumplir con restricciones de tiempo real para su aplicación en sistemas con recursos hardware limitados

Non-isometric 3D shape registration.

3D shape registration is an important task in computer graphics and computer vision. It has been widely used in the area of film industry, 3D animation, video games and AR/VR assets creation. Manually creating the 3D model of a character from scratch is tedious and time consuming, and it can only be completed by professional trained artists. With the development of 3D geometry acquisition technology, it becomes easier and cheaper to capture high-resolution and highly detailed 3D geometries. However, the scanned data are often incomplete or noisy and therefore cannot be employed directly. To deal with the above two problems, one typical and efficient solution is to deform an existing high-quality model (template) to fit the scanned data (target). Shape registration as an essential technique to do so has been arousing intensive attention. In last decades, various shape registration approaches have been proposed for accurate template fitting. However, there are still some remaining challenges. It is well known that the template can be largely different with the target in respect of size and pose. With the large (usually non-isometric) deformation between them, the shear distortion can easily occur, which may lead to poor results, such as degenerated triangles, fold-overs. Before deforming the template towards the target, reliable correspondences between them should be found first. Incorrect correspondences give the wrong deformation guidance, which can also easily produce fold-overs. As mentioned before, the target always comes with noise. This is the part we want to filter out and try not to fit the template on it. Hence, non-isometric shape registration robust to noise is highly desirable in the scene of geometry modelling from the scanned data. In this PhD research, we address existing challenges in shape registration, including how to prevent the deformation distortion, how to reduce the foldover occurrence and how to deal with the noise in the target. Novel methods including consistent as-similar as-possible surface deformation and robust Huber-L1 surface registration are proposed, which are validated through experimental comparison with state-of-the-arts. The deformation technique plays an important role in shape registration. In this research, a consistent as similar-as-possible (CASAP) surface deformation approach is proposed. Starting from investigating the continuous deformation energy, we analyse the existing term to make the discrete energy converge to the continuous one, whose property we called as energy consistency. Based on the deformation method, a novel CASAP non-isometric surface registration method is proposed. The proposed registration method well preserves the angles of triangles in the template surface so that least distortion is introduced during the surface deformation and thus reduce the risk of fold-over and self-intersection. To reduce the noise influence, a Huber-L1 based non-isometric surface registration is proposed, where a Huber-L1 regularized model constrained on the transformation variation and position difference. The proposed method is robust to noise and produces piecewise smooth results while still preserving fine details on the target. We evaluate and validate our methods through extensive experiments, whose results have demonstrated that the proposed methods in this thesis are more accurate and robust to noise in comparison of the state-of-the arts and enable us to produce high quality models with little efforts

Multi-Modal Partial Surface Matching for Intra-Operative Registration

An important task for computer-assisted surgical interventions is the alignment of pre- and intra-operative spaces allowing the transfer of pre-operative information to the current patient situation, known as intra-operative registration. Registration is usually performed by using markers or image-based techniques. Another approach is the intra-operative acquisition of organ surfaces by 3D range scanners, which are then matched to pre-operatively generated surfaces. However, this approach is not trivial, as methods for intra-operative surface matching must be able to deal with noise, distortions, deformations, and the availability of only partially overlapping, nearly flat surfaces. For these reasons, surface matching for intra-operative registration has so far only been used to account for displacements that occur in local scales, while the actual alignment is still performed manually. The main contributions of this thesis are two different approaches for automatic surface matching in intra-operative environments. The focus here is the registration of surfaces acquired by different modalities, dealing with the aforementioned issues and without relying on unique landmarks. For the first approach, surfaces are converted to graph representations and correspondences between them are identified by means of graph matching. Graphs are obtained automatically by segmenting the surfaces into regions with similar properties. As the graph matching problem is known to be NP-hard, it was solved by iteratively computing node similarity scores, and converting it to a linear assignment problem. In the second approach, correspondences are identified by the selection of two spatial configurations of landmarks that can be better fitted to each other, according to an error metric. This error metric does not only incorporate a fitting error, but also a new measure for spatial configuration reliability. The optimization problem is solved by means of a greedy algorithm. Evaluation of the two approaches was performed with several experiments, simulating intra-operative conditions. While the graph matching approach proved to be robust for the registration of small partial data, the point-based approach proved to be more reliable for noisy surfaces. Apart from being a significant contribution to the field of feature-less partial surface matching, this work represents a great effort towards the achievement of a fully automatic, marker-less, registration system for computer-assisted surgery guidance

A novel procedure for medial axis reconstruction of vessels from Medical Imaging segmentation

A procedure for reconstructing the central axis from diagnostic image processing is presented here, capable of solving the widespread problem of stepped shape effect that characterizes the most common algorithmic tools for processing the central axis for diagnostic imaging applications through the development of an algorithm correcting the spatial coordinates of each point belonging to the axis from the use of a common discrete image skeleton algorithm. The procedure is applied to the central axis traversing the vascular branch of the cerebral system, appropriately reconstructed from the processing of diagnostic images, using investigations of the local intensity values identified in adjacent voxels. The percentage intensity of the degree of adherence to a specific anatomical tissue acts as an attraction pole in the identification of the spatial center on which to place each point of the skeleton crossing the investigated anatomical structure. The results were shown in terms of the number of vessels identified overall compared to the original reference model. The procedure demonstrates high accuracy margin in the correction of the local coordinates of the central points that permits to allocate precise dimensional measurement of the anatomy under examination. The reconstruction of a central axis effectively centered in the region under examination represents a fundamental starting point in deducing, with a high margin of accuracy, key informations of a geometric and dimensional nature that favours the recognition of phenomena of shape alterations ascribable to the presence of clinical pathologies

Modeling and Simulation in Engineering

This book provides an open platform to establish and share knowledge developed by scholars, scientists, and engineers from all over the world, about various applications of the modeling and simulation in the design process of products, in various engineering fields. The book consists of 12 chapters arranged in two sections (3D Modeling and Virtual Prototyping), reflecting the multidimensionality of applications related to modeling and simulation. Some of the most recent modeling and simulation techniques, as well as some of the most accurate and sophisticated software in treating complex systems, are applied. All the original contributions in this book are jointed by the basic principle of a successful modeling and simulation process: as complex as necessary, and as simple as possible. The idea is to manipulate the simplifying assumptions in a way that reduces the complexity of the model (in order to make a real-time simulation), but without altering the precision of the results

Surveying and Three-Dimensional Modeling for Preservation and Structural Analysis of Cultural Heritage

Dense point clouds can be used for three important steps in structural analysis, in the field of cultural heritage, regardless of which instrument it was used for acquisition data. Firstly, they allow deriving the geometric part of a finite element (FE) model automatically or semi-automatically. User input is mainly required to complement invisible parts and boundaries of the structure, and to assign meaningful approximate physical parameters. Secondly, FE model obtained from point clouds can be used to estimate better and more precise parameters of the structural analysis, i.e., to train the FE model. Finally, the definition of a correct Level of Detail about the three-dimensional model, deriving from the initial point cloud, can be used to define the limit beyond which the structural analysis is compromised, or anyway less precise. In this work of research, this will be demonstrated using three different case studies of buildings, consisting mainly of masonry, measured through terrestrial laser scanning and photogrammetric acquisitions. This approach is not a typical study for geomatics analysis, but its challenges allow studying benefits and limitations. The results and the proposed approaches could represent a step towards a multidisciplinary approach where Geomatics can play a critical role in the monitoring and civil engineering field. Furthermore, through a geometrical reconstruction, different analyses and comparisons are possible, in order to evaluate how the numerical model is accurate. In fact, the discrepancies between the different results allow to evaluate how, from a geometric and simplified modeling, important details can be lost. This causes, for example, modifications in terms of mass and volume of the structure

Automatic registration of 3D models to laparoscopic video images for guidance during liver surgery

Laparoscopic liver interventions offer significant advantages over open surgery, such as less pain and trauma, and shorter recovery time for the patient. However, they also bring challenges for the surgeons such as the lack of tactile feedback, limited field of view and occluded anatomy. Augmented reality (AR) can potentially help during laparoscopic liver interventions by displaying sub-surface structures (such as tumours or vasculature). The initial registration between the 3D model extracted from the CT scan and the laparoscopic video feed is essential for an AR system which should be efficient, robust, intuitive to use and with minimal disruption to the surgical procedure. Several challenges of registration methods in laparoscopic interventions include the deformation of the liver due to gas insufflation in the abdomen, partial visibility of the organ and lack of prominent geometrical or texture-wise landmarks. These challenges are discussed in detail and an overview of the state of the art is provided. This research project aims to provide the tools to move towards a completely automatic registration. Firstly, the importance of pre-operative planning is discussed along with the characteristics of the liver that can be used in order to constrain a registration method. Secondly, maximising the amount of information obtained before the surgery, a semi-automatic surface based method is proposed to recover the initial rigid registration irrespective of the position of the shapes. Finally, a fully automatic 3D-2D rigid global registration is proposed which estimates a global alignment of the pre-operative 3D model using a single intra-operative image. Moving towards incorporating the different liver contours can help constrain the registration, especially for partial surfaces. Having a robust, efficient AR system which requires no manual interaction from the surgeon will aid in the translation of such approaches to the clinics