Automatic landmark annotation and dense correspondence registration for 3D human facial images

Dense surface registration of three-dimensional (3D) human facial images holds great potential for studies of human trait diversity, disease genetics, and forensics. Non-rigid registration is particularly useful for establishing dense anatomical correspondences between faces. Here we describe a novel non-rigid registration method for fully automatic 3D facial image mapping. This method comprises two steps: first, seventeen facial landmarks are automatically annotated, mainly via PCA-based feature recognition following 3D-to-2D data transformation. Second, an efficient thin-plate spline (TPS) protocol is used to establish the dense anatomical correspondence between facial images, under the guidance of the predefined landmarks. We demonstrate that this method is robust and highly accurate, even for different ethnicities. The average face is calculated for individuals of Han Chinese and Uyghur origins. While fully automatic and computationally efficient, this method enables high-throughput analysis of human facial feature variation.Comment: 33 pages, 6 figures, 1 tabl

Dilated FCN for Multi-Agent 2D/3D Medical Image Registration

2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.Comment: AAAI 201

3D-2D image registration by nonlinear regression

We propose a 3D-2D image registration method that relates image features of 2D projection images to the transformation parameters of the 3D image by nonlinear regression. The method is compared with a conventional registration method based on iterative optimization. For evaluation, simulated X-ray images (DRRs) were generated from coronary artery tree models derived from 3D CTA scans. Registration of nine vessel trees was performed, and the alignment quality was measured by the mean target registration error (mTRE). The regression approach was shown to be slightly less accurate, but much more robust than the method based on an iterative optimization approach

Medical image registration by neural networks: a regression-based registration approach

This thesis focuses on the development and evaluation of a registration-by-regression approach for the 3D/2D registration of coronary Computed Tomography Angiography (CTA) and X-ray angiography. This regression-based method relates image features of 2D projection images to the transformation parameters of the 3D image by a nonlinear regression. It treats registration as a regression problem, as an alternative for the traditional iterative approach that often comes with high computational costs and limited capture range. First we presented a survey of the methods with a regression-based registration approach for medical applications, as well as a summary of their main characteristics (Chapter 2). Second, we studied the registration methodology, addressing the input features and the choice of regression model (Chapter 3 and Chapter 4). For that purpose, we evaluated different options using simulated X-ray images generated from coronary artery tree models derived from 3D CTA scans. We also compared the registration-by-regression results with a method based on iterative optimization. Different image features of 2D projections and seven regression techniques were considered. The regression approach for simulated X-rays was shown to be slightly less accurate, but much more robust than the method based on an iterative optimization approach. Neural Networks obtained accurate results and showed to be robust to large initial misalignment. Third, we evaluated the registration-by-regression method using clinical data, integrating the 3D preoperative CTA of the coronary arteries with intraoperative 2D X-ray angiography images (Chapter 5). For the evaluation of the image registration, a gold standard registration was established using an exhaustive search followed by a multi-observer visual scoring procedure. The influence of preprocessing options for the simulated images and the real X-rays was studied. Several image features were also compared. The coronary registration–by-regression results were not satisfactory, resembling manual initialization accuracy. Therefore, the proposed method for this concrete problem and in its current configuration is not sufficiently accurate to be used in the clinical practice. The framework developed enables us to better understand the dependency of the proposed method on the differences between simulated and real images. The main difficulty lies in the substantial differences in appearance between the images used for training (simulated X-rays from 3D coronary models) and the actual images obtained during the intervention (real X-ray angiography). We suggest alternative solutions and recommend to evaluate the registration-by-regression approach in other applications where training data is available that has similar appearance to the eventual test data

Computer assisted determination of acetabular cup orientation using 2D-3D image registration

Background: 2D-3D image-based registration methods have been developed to measure acetabular cup orientation after total hip arthroplasty (THA). These methods require registration of both the prosthesis and the CT images to 2D radiographs and compute implant position with respect to a reference. The application of these methods is limited in clinical practice due to two limitations: (1) the requirement of a computer-aided design (CAD) model of the prosthesis, which may be unavailable due to the proprietary concerns of the manufacturer, and (2) the requirement of either multiple radiographs or radiograph-specific calibration, usually unavailable for retrospective studies. In this paper, we propose a new method to address these limitations. Methods: A new formulation for determination of post-operative cup orientation, which couples a radiographic measurement with 2D-3D image matching, was developed. In our formulation, the radiographic measurement can be obtained with known methods so that the challenge lies in the 2D-3D image matching. To solve this problem, a hybrid 2D-3D registration scheme combining a landmark-to-ray 2D-3D alignment with a robust intensity-based 2D-3D registration was used. The hybrid 2D-3D registration scheme allows computing both the post-operative cup orientation with respect to an anatomical reference and the pelvic tilt and rotation with respect to the X-ray imaging table/plate. The method was validated using 2D adult cadaver hips. Results: Using the hybrid 2D-3D registration scheme, our method showed a mean accuracy of 1.0 ± 0.7° (range from 0.1°to2.0°) for inclination and 1.7 ± 1.2° (range from 0.0° to 3.9°) for anteversion, taking the measurements from post-operative CT images as ground truths. Conclusions: Our new solution formulation and the hybrid 2D-3D registration scheme facilitate estimation of post-operative cup orientation and measurement of pelvic tilt and rotatio

A surface registration approach for video-based analysis of intraoperative brain surface deformations.

Anatomical intra operative deformation is a major limitation of accuracy in image guided neurosurgery. Approaches to quantify these deforamations based on 3D reconstruction of surfaces have been introduced. For accurate quantification of surface deformation, a robust surface registration method is required. In this paper, we propose a new surface registration for video-based analysis of intraoperative brain deformations. This registration method includes three terms: the first term is related to image intensities, the second to Euclidean distance and the third to anatomical landmarks continuously tracked in 2D video. This new surface registration method can be used with any cortical surface textured point cloud computed by stereoscopic or laser range approaches. We have shown the global method, including textured point cloud reconstruction, had a precision within 2 millimeters, which is within the usual rigid registration error of the neuronavigation system before deformations

Navigace mobilních robotů v neznámém prostředí s využitím měření vzdáleností

The ability of a robot to navigate itself in the environment is a crucial step towards its autonomy. Navigation as a subtask of the development of autonomous robots is the subject of this thesis, focusing on the development of a method for simultaneous localization an mapping (SLAM) of mobile robots in six degrees of freedom (DOF). As a part of this research, a platform for 3D range data acquisition based on a continuously inclined laser rangefinder was developed. This platform is presented, evaluating the measurements and also presenting the robotic equipment on which the platform can be fitted. The localization and mapping task is equal to the registration of multiple 3D images into a common frame of reference. For this purpose, a method based on the Iterative Closest Point (ICP) algorithm was developed. First, the originally implemented SLAM method is presented, focusing on the time-wise performance and the registration quality issues introduced by the implemented algorithms. In order to accelerate and improve the quality of the time-demanding 6DOF image registration, an extended method was developed. The major extension is the introduction of a factorized registration, extracting 2D representations of vertical objects called leveled maps from the 3D point sets, ensuring these representations are 3DOF invariant. The extracted representations are registered in 3DOF using ICP algorithm, allowing pre-alignment of the 3D data for the subsequent robust 6DOF ICP based registration. The extended method is presented, showing all important modifications to the original method. The developed registration method was evaluated using real 3D data acquired in different indoor environments, examining the benefits of the factorization and other extensions as well as the performance of the original ICP based method. The factorization gives promising results compared to a single phase 6DOF registration in vertically structured environments. Also, the disadvantages of the method are discussed, proposing possible solutions. Finally, the future prospects of the research are presented.Schopnost lokalizace a navigace je podmínkou autonomního provozu mobilních robotů. Předmětem této disertační práce jsou navigační metody se zaměřením na metodu pro simultánní lokalizaci a mapování (SLAM) mobilních robotů v šesti stupních volnosti (6DOF). Nedílnou součástí tohoto výzkumu byl vývoj platformy pro sběr 3D vzdálenostních dat s využitím kontinuálně naklápěného laserového řádkového scanneru. Tato platforma byla vyvinuta jako samostatný modul, aby mohla být umístěna na různé šasi mobilních robotů. Úkol lokalizace a mapování je ekvivalentní registraci více 3D obrazů do společného souřadného systému. Pro tyto účely byla vyvinuta metoda založená na algoritmu Iterative Closest Point Algorithm (ICP). Původně implementovaná verze navigační metody využívá ICP s akcelerací pomocí kd-stromů přičemž jsou zhodnoceny její kvalitativní a výkonnostní aspekty. Na základě této analýzy byly vyvinuty rozšíření původní metody založené na ICP. Jednou z hlavních modifikací je faktorizace registračního procesu, kdy tato faktorizace je založena na redukci dat: vytvoření 2D „leveled“ map (ve smyslu jednoúrovňových map) ze 3D vzdálenostních obrazů. Pro tuto redukci je technologicky i algoritmicky zajištěna invariantnost těchto map vůči třem stupňům volnosti. Tyto redukované mapy jsou registrovány pomocí ICP ve zbylých třech stupních volnosti, přičemž získaná transformace je aplikována na 3D data za účelem před-registrace 3D obrazů. Následně je provedena robustní 6DOF registrace. Rozšířená metoda je v disertační práci v popsána spolu se všemi podstatnými modifikacemi. Vyvinutá metoda byla otestována a zhodnocena s využitím skutečných 3D vzdálenostních dat naměřených v různých vnitřních prostředích. Jsou zhodnoceny přínosy faktorizace a jiných modifikací ve srovnání s původní jednofázovou 6DOF registrací, také jsou zmíněny nevýhody implementované metody a navrženy způsoby jejich řešení. Nakonec následuje návrh budoucího výzkumu a diskuse o možnostech dalšího rozvoje.

Registration of 3D fetal neurosonography and MRI.

We propose a method for registration of 3D fetal brain ultrasound with a reconstructed magnetic resonance fetal brain volume. This method, for the first time, allows the alignment of models of the fetal brain built from magnetic resonance images with 3D fetal brain ultrasound, opening possibilities to develop new, prior information based image analysis methods for 3D fetal neurosonography. The reconstructed magnetic resonance volume is first segmented using a probabilistic atlas and a pseudo ultrasound image volume is simulated from the segmentation. This pseudo ultrasound image is then affinely aligned with clinical ultrasound fetal brain volumes using a robust block-matching approach that can deal with intensity artefacts and missing features in the ultrasound images. A qualitative and quantitative evaluation demonstrates good performance of the method for our application, in comparison with other tested approaches. The intensity average of 27 ultrasound images co-aligned with the pseudo ultrasound template shows good correlation with anatomy of the fetal brain as seen in the reconstructed magnetic resonance image