Geometrical Calibration of X-Ray Imaging With RGB Cameras for 3D Reconstruction

(c) 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.We present a methodology to recover the geometrical calibration of conventional X-ray settings with the help of an ordinary video camera and visible fiducials that are present in the scene. After calibration, equivalent points of interest can be easily identifiable with the help of the epipolar geometry. The same procedure also allows the measurement of real anatomic lengths and angles and obtains accurate 3D locations from image points. Our approach completely eliminates the need for X-ray-opaque reference marks (and necessary supporting frames) which can sometimes be invasive for the patient, occlude the radiographic picture, and end up projected outside the imaging sensor area in oblique protocols. Two possible frameworks are envisioned: a spatially shifting X-ray anode around the patient/object and a moving patient that moves/rotates while the imaging system remains fixed. As a proof of concept, experiences with a device under test (DUT), an anthropomorphic phantom and a real brachytherapy session have been carried out. The results show that it is possible to identify common points with a proper level of accuracy and retrieve three-dimensional locations, lengths and shapes with a millimetric level of precision. The presented approach is simple and compatible with both current and legacy widespread diagnostic X-ray imaging deployments and it can represent a good and inexpensive alternative to other radiological modalities like CT.This work was carried out with the support of Information Storage S.L., University of Valencia (grant #CPI-15-170), CSD2007-00042 Consolider Ingenio CPAN (grant #CPAN13-TR01) as well as with the support of the Spanish Ministry of Industry, Energy and Tourism (Grant TSI-100101-2013-019).Albiol Colomer, F.; Corbi, A.; Albiol Colomer, A. (2016). Geometrical Calibration of X-Ray Imaging With RGB Cameras for 3D Reconstruction. IEEE Transactions on Medical Imaging. 35(8):1952-1961. https://doi.org/10.1109/TMI.2016.2540929S1952196135

Medical image registration by neural networks: a regression-based registration approach

This thesis focuses on the development and evaluation of a registration-by-regression approach for the 3D/2D registration of coronary Computed Tomography Angiography (CTA) and X-ray angiography. This regression-based method relates image features of 2D projection images to the transformation parameters of the 3D image by a nonlinear regression. It treats registration as a regression problem, as an alternative for the traditional iterative approach that often comes with high computational costs and limited capture range. First we presented a survey of the methods with a regression-based registration approach for medical applications, as well as a summary of their main characteristics (Chapter 2). Second, we studied the registration methodology, addressing the input features and the choice of regression model (Chapter 3 and Chapter 4). For that purpose, we evaluated different options using simulated X-ray images generated from coronary artery tree models derived from 3D CTA scans. We also compared the registration-by-regression results with a method based on iterative optimization. Different image features of 2D projections and seven regression techniques were considered. The regression approach for simulated X-rays was shown to be slightly less accurate, but much more robust than the method based on an iterative optimization approach. Neural Networks obtained accurate results and showed to be robust to large initial misalignment. Third, we evaluated the registration-by-regression method using clinical data, integrating the 3D preoperative CTA of the coronary arteries with intraoperative 2D X-ray angiography images (Chapter 5). For the evaluation of the image registration, a gold standard registration was established using an exhaustive search followed by a multi-observer visual scoring procedure. The influence of preprocessing options for the simulated images and the real X-rays was studied. Several image features were also compared. The coronary registration–by-regression results were not satisfactory, resembling manual initialization accuracy. Therefore, the proposed method for this concrete problem and in its current configuration is not sufficiently accurate to be used in the clinical practice. The framework developed enables us to better understand the dependency of the proposed method on the differences between simulated and real images. The main difficulty lies in the substantial differences in appearance between the images used for training (simulated X-rays from 3D coronary models) and the actual images obtained during the intervention (real X-ray angiography). We suggest alternative solutions and recommend to evaluate the registration-by-regression approach in other applications where training data is available that has similar appearance to the eventual test data

Multi-modal matching of 2D images with 3D medical data

Image registration is the process of aligning images of the same object taken at different time points or with different imaging modalities with the aim to compare them in one coordinate system. Image registration is particularly important in biomedical imaging, where a multitude of imaging modalities exist. For example, images can be obtained with X-ray computed tomography (CT) which is based on the object’s X-ray beam attenuation whereas magnetic resonance imaging (MRI) underlines its local proton density. The gold standard in pathology for tissue analysis is histology. Histology, however, provides only 2D information in the selected sections of the 3D tissue. To evaluate the tissue’s 3D structure, volume imaging techniques, such as CT or MRI, are preferable. The combination of functional information from histology with 3D morphological data from CT is essential for tissue analysis. Furthermore, histology can validate anatomical features identified in CT data. Therefore, the registration of these two modalities is indispensable to provide a more complete overview of the tissue. Previously proposed algorithms for the registration of histological slides into 3D volumes usually rely on manual interactions, which is time-consuming and prone to bias. The high complexity of this type of registration originates from the large number of degrees of freedom. The goal of my thesis was to develop an automatic method for histology to 3D volume registration to master these challenges. The first stage of the developed algorithm uses a scale-invariant feature detector to find common matches between the histology slide and each tomography slice in a 3D dataset. A plane of the most likely position is then fitted into the feature point cloud using a robust model fitting algorithm. The second stage builds upon the first one and introduces fine-tuning of the slice position using normalized Mutual Information (NMI). Additionally, using previously developed 2D-2D registration techniques we find the rotation and translation of the histological slide within the plane. Moreover, the framework takes into account any potential nonlinear deformations of the histological slides that might occur during tissue preparation. The application of the algorithm to MRI data is investigated in our third work. The developed extension of the multi-modal feature detector showed promising results, however, the registration of a histological slide to the direct MRI volume remains a challenging task

Learning to extract features for 2D – 3D multimodal registration

The ability to capture depth information form an scene has greatly increased in the recent years. 3D sensors, traditionally high cost and low resolution sensors, are being democratized and 3D scans of indoor and outdoor scenes are becoming more and more common. However, there is still a great data gap between the amount of captures being performed with 2D and 3D sensors. Although the 3D sensors provide more information about the scene, 2D sensors are still more accessible and widely used. This trade-off between availability and information between sensors brings us to a multimodal scenario of mixed 2D and 3D data. This thesis explores the fundamental block of this multimodal scenario: the registration between a single 2D image and a single unorganized point cloud. An unorganized 3D point cloud is the basic representation of a 3D capture. In this representation the surveyed points are represented only by their real word coordinates and, optionally, by their colour information. This simplistic representation brings multiple challenges to the registration, since most of the state of the art works leverage the existence of metadata about the scene or prior knowledges. Two different techniques are explored to perform the registration: a keypoint-based technique and an edge-based technique. The keypoint-based technique estimates the transformation by means of correspondences detected using Deep Learning, whilst the edge-based technique refines a transformation using a multimodal edge detection to establish anchor points to perform the estimation. An extensive evaluation of the proposed methodologies is performed. Albeit further research is needed to achieve adequate performances, the obtained results show the potential of the usage of deep learning techniques to learn 2D and 3D similarities. The results also show the good performance of the proposed 2D-3D iterative refinement, up to the state of the art on 3D-3D registration.La capacitat de captar informació de profunditat d’una escena ha augmentat molt els darrers anys. Els sensors 3D, tradicionalment d’alt cost i baixa resolució, s’estan democratitzant i escànners 3D d’escents interiors i exteriors són cada vegada més comuns. Tot i això, encara hi ha una gran bretxa entre la quantitat de captures que s’estan realitzant amb sensors 2D i 3D. Tot i que els sensors 3D proporcionen més informació sobre l’escena, els sensors 2D encara són més accessibles i àmpliament utilitzats. Aquesta diferència entre la disponibilitat i la informació entre els sensors ens porta a un escenari multimodal de dades mixtes 2D i 3D. Aquesta tesi explora el bloc fonamental d’aquest escenari multimodal: el registre entre una sola imatge 2D i un sol núvol de punts no organitzat. Un núvol de punts 3D no organitzat és la representació bàsica d’una captura en 3D. En aquesta representació, els punts mesurats es representen només per les seves coordenades i, opcionalment, per la informació de color. Aquesta representació simplista aporta múltiples reptes al registre, ja que la majoria dels algoritmes aprofiten l’existència de metadades sobre l’escena o coneixements previs. Per realitzar el registre s’exploren dues tècniques diferents: una tècnica basada en punts clau i una tècnica basada en contorns. La tècnica basada en punts clau estima la transformació mitjançant correspondències detectades mitjançant Deep Learning, mentre que la tècnica basada en contorns refina una transformació mitjançant una detecció multimodal de la vora per establir punts d’ancoratge per realitzar l’estimació. Es fa una avaluació àmplia de les metodologies proposades. Tot i que es necessita més investigació per obtenir un rendiment adequat, els resultats obtinguts mostren el potencial de l’ús de tècniques d’aprenentatge profund per aprendre similituds 2D i 3D. Els resultats també mostren l’excel·lent rendiment del perfeccionament iteratiu 2D-3D proposat, similar al dels algoritmes de registre 3D-3D.La capacidad de captar información de profundidad de una escena ha aumentado mucho en los últimos años. Los sensores 3D, tradicionalmente de alto costo y baja resolución, se están democratizando y escáneres 3D de escents interiores y exteriores son cada vez más comunes. Sin embargo, todavía hay una gran brecha entre la cantidad de capturas que se están realizando con sensores 2D y 3D. Aunque los sensores 3D proporcionan más información sobre la escena, los sensores 2D todavía son más accesibles y ampliamente utilizados. Esta diferencia entre la disponibilidad y la información entre los sensores nos lleva a un escenario multimodal de datos mixtos 2D y 3D. Esta tesis explora el bloque fundamental de este escenario multimodal: el registro entre una sola imagen 2D y una sola nube de puntos no organizado. Una nube de puntos 3D no organizado es la representación básica de una captura en 3D. En esta representación, los puntos medidos se representan sólo por sus coordenadas y, opcionalmente, por la información de color. Esta representación simplista aporta múltiples retos en el registro, ya que la mayoría de los algoritmos aprovechan la existencia de metadatos sobre la escena o conocimientos previos. Para realizar el registro se exploran dos técnicas diferentes: una técnica basada en puntos clave y una técnica basada en contornos. La técnica basada en puntos clave estima la transformación mediante correspondencias detectadas mediante Deep Learning, mientras que la técnica basada en contornos refina una transformación mediante una detección multimodal del borde para establecer puntos de anclaje para realizar la estimación. Se hace una evaluación amplia de las metodologías propuestas. Aunque se necesita más investigación para obtener un rendimiento adecuado, los resultados obtenidos muestran el potencial del uso de técnicas de aprendizaje profundo para aprender similitudes 2D y 3D. Los resultados también muestran el excelente rendimiento del perfeccionamiento iterativo 2D-3D propuesto, similar al de los algoritmos de registro 3D-3D

Learning to extract features for 2D – 3D multimodal registration

The ability to capture depth information form an scene has greatly increased in the recent years. 3D sensors, traditionally high cost and low resolution sensors, are being democratized and 3D scans of indoor and outdoor scenes are becoming more and more common. However, there is still a great data gap between the amount of captures being performed with 2D and 3D sensors. Although the 3D sensors provide more information about the scene, 2D sensors are still more accessible and widely used. This trade-off between availability and information between sensors brings us to a multimodal scenario of mixed 2D and 3D data. This thesis explores the fundamental block of this multimodal scenario: the registration between a single 2D image and a single unorganized point cloud. An unorganized 3D point cloud is the basic representation of a 3D capture. In this representation the surveyed points are represented only by their real word coordinates and, optionally, by their colour information. This simplistic representation brings multiple challenges to the registration, since most of the state of the art works leverage the existence of metadata about the scene or prior knowledges. Two different techniques are explored to perform the registration: a keypoint-based technique and an edge-based technique. The keypoint-based technique estimates the transformation by means of correspondences detected using Deep Learning, whilst the edge-based technique refines a transformation using a multimodal edge detection to establish anchor points to perform the estimation. An extensive evaluation of the proposed methodologies is performed. Albeit further research is needed to achieve adequate performances, the obtained results show the potential of the usage of deep learning techniques to learn 2D and 3D similarities. The results also show the good performance of the proposed 2D-3D iterative refinement, up to the state of the art on 3D-3D registration.La capacitat de captar informació de profunditat d’una escena ha augmentat molt els darrers anys. Els sensors 3D, tradicionalment d’alt cost i baixa resolució, s’estan democratitzant i escànners 3D d’escents interiors i exteriors són cada vegada més comuns. Tot i això, encara hi ha una gran bretxa entre la quantitat de captures que s’estan realitzant amb sensors 2D i 3D. Tot i que els sensors 3D proporcionen més informació sobre l’escena, els sensors 2D encara són més accessibles i àmpliament utilitzats. Aquesta diferència entre la disponibilitat i la informació entre els sensors ens porta a un escenari multimodal de dades mixtes 2D i 3D. Aquesta tesi explora el bloc fonamental d’aquest escenari multimodal: el registre entre una sola imatge 2D i un sol núvol de punts no organitzat. Un núvol de punts 3D no organitzat és la representació bàsica d’una captura en 3D. En aquesta representació, els punts mesurats es representen només per les seves coordenades i, opcionalment, per la informació de color. Aquesta representació simplista aporta múltiples reptes al registre, ja que la majoria dels algoritmes aprofiten l’existència de metadades sobre l’escena o coneixements previs. Per realitzar el registre s’exploren dues tècniques diferents: una tècnica basada en punts clau i una tècnica basada en contorns. La tècnica basada en punts clau estima la transformació mitjançant correspondències detectades mitjançant Deep Learning, mentre que la tècnica basada en contorns refina una transformació mitjançant una detecció multimodal de la vora per establir punts d’ancoratge per realitzar l’estimació. Es fa una avaluació àmplia de les metodologies proposades. Tot i que es necessita més investigació per obtenir un rendiment adequat, els resultats obtinguts mostren el potencial de l’ús de tècniques d’aprenentatge profund per aprendre similituds 2D i 3D. Els resultats també mostren l’excel·lent rendiment del perfeccionament iteratiu 2D-3D proposat, similar al dels algoritmes de registre 3D-3D.La capacidad de captar información de profundidad de una escena ha aumentado mucho en los últimos años. Los sensores 3D, tradicionalmente de alto costo y baja resolución, se están democratizando y escáneres 3D de escents interiores y exteriores son cada vez más comunes. Sin embargo, todavía hay una gran brecha entre la cantidad de capturas que se están realizando con sensores 2D y 3D. Aunque los sensores 3D proporcionan más información sobre la escena, los sensores 2D todavía son más accesibles y ampliamente utilizados. Esta diferencia entre la disponibilidad y la información entre los sensores nos lleva a un escenario multimodal de datos mixtos 2D y 3D. Esta tesis explora el bloque fundamental de este escenario multimodal: el registro entre una sola imagen 2D y una sola nube de puntos no organizado. Una nube de puntos 3D no organizado es la representación básica de una captura en 3D. En esta representación, los puntos medidos se representan sólo por sus coordenadas y, opcionalmente, por la información de color. Esta representación simplista aporta múltiples retos en el registro, ya que la mayoría de los algoritmos aprovechan la existencia de metadatos sobre la escena o conocimientos previos. Para realizar el registro se exploran dos técnicas diferentes: una técnica basada en puntos clave y una técnica basada en contornos. La técnica basada en puntos clave estima la transformación mediante correspondencias detectadas mediante Deep Learning, mientras que la técnica basada en contornos refina una transformación mediante una detección multimodal del borde para establecer puntos de anclaje para realizar la estimación. Se hace una evaluación amplia de las metodologías propuestas. Aunque se necesita más investigación para obtener un rendimiento adecuado, los resultados obtenidos muestran el potencial del uso de técnicas de aprendizaje profundo para aprender similitudes 2D y 3D. Los resultados también muestran el excelente rendimiento del perfeccionamiento iterativo 2D-3D propuesto, similar al de los algoritmos de registro 3D-3D.Postprint (published version

Differential geometry methods for biomedical image processing : from segmentation to 2D/3D registration

This thesis establishes a biomedical image analysis framework for the advanced visualization of biological structures. It consists of two important parts: 1) the segmentation of some structures of interest in 3D medical scans, and 2) the registration of patient-specific 3D models with 2D interventional images. Segmenting biological structures results in 3D computational models that are simple to visualize and that can be analyzed quantitatively. Registering a 3D model with interventional images permits to position the 3D model within the physical world. By combining the information from a 3D model and 2D interventional images, the proposed framework can improve the guidance of surgical intervention by reducing the ambiguities inherent to the interpretation of 2D images. Two specific segmentation problems are considered: 1) the segmentation of large structures with low frequency intensity nonuniformity, and 2) the detection of fine curvilinear structures. First, we directed our attention toward the segmentation of relatively large structures with low frequency intensity nonuniformity. Such structures are important in medical imaging since they are commonly encountered in MRI. Also, the nonuniform diffusion of the contrast agent in some other modalities, such as CTA, leads to structures of nonuniform appearance. A level-set method that uses a local-linear region model is defined, and applied to the challenging problem of segmenting brain tissues in MRI. The unique characteristics of the proposed method permit to account for important image nonuniformity implicitly. To the best of our knowledge, this is the first time a region-based level-set model has been used to perform the segmentation of real world MRI brain scans with convincing results. The second segmentation problem considered is the detection of fine curvilinear structures in 3D medical images. Detecting those structures is crucial since they can represent veins, arteries, bronchi or other important tissues. Unfortunately, most currently available curvilinear structure detection filters incur significant signal lost at bifurcations of two structures. This peculiarity limits the performance of all subsequent processes, whether it be understanding an angiography acquisition, computing an accurate tractography, or automatically classifying the image voxels. This thesis presents a new curvilinear structure detection filter that is robust to the presence of X- and Y-junctions. At the same time, it is conceptually simple and deterministic, and allows for an intuitive representation of the structure’s principal directions. Once a 3D computational model is available, it can be used to enhance surgical guidance. A 2D/3D non-rigid method is proposed that brings a 3D centerline model of the coronary arteries into correspondence with bi-plane fluoroscopic angiograms. The registered model is overlaid on top of the interventional angiograms to provide surgical assistance during image-guided chronic total occlusion procedures, which reduces the uncertainty inherent in 2D interventional images. A fully non-rigid registration model is proposed and used to compensate for any local shape discrepancy. This method is based on a variational framework, and uses a simultaneous matching and reconstruction process. With a typical run time of less than 3 seconds, the algorithms are fast enough for interactive applications

Recalage rigide 3D-2D par intensité pour le traitement percutané des cardiopathies congénitales

Les cardiopathies congénitales cyanogènes sont des malformations cardiaques infantiles qui, dans leurs formes les plus complexes, sont aggravées par des artères morbides partant de l’aorte et appelées collatérales aorto-pulmonaires majeures (MAPCAs). Pour corriger ces malformations, les cardiologues insèrent un cathéter dans une artère du patient puis, le guident jusqu’à atteindre la structure vasculaire d’intérêt. Le cathéter est visualisé grâce à des angiographies acquises lors de l’opération. Néanmoins, ces interventions, dîtes percutanées, sont délicates à réaliser. L’emploi des angiographies 2D limite le champ de vision des cardiologues et les oblige à mentalement reconstruire la structure vasculaire en mouvement. Afin d’améliorer les conditions d’intervention, des techniques d’imagerie médicale exploitant des données tomographiques acquis avant l’intervention sont développées. Les données tomographiques forment un modèle 3D fiable de la structure vasculaire qui, une fois précisément aligné avec les angiographies, définit un outil de navigation virtuel 3D qui augmente le champ de vision des cardiologues. Dans ce mémoire, une nouvelle méthode automatique de recalage rigide 3D-2D par intensité de données tomographiques 3D avec des angiographies 2D est présentée. Aussi, une technique d’alignement semi-automatique permettant d’accélérer l’initialisation de la méthode automatique est développée. Les résultats de la méthode de recalage proposée, obtenus avec deux jeux de données de patient atteints de malformations cardiaques, sont prometteurs. Un alignement précis et robuste des données tomographique de l’artère aorte et des MAPCAs (0;265�0;647mm et 99 % de succès) à partir d’un déplacement rigide d’amplitude maximale (20mm et 20°) est obtenu en un temps de calcul raisonnable (13,7 secondes)