An Information-Theory Framework for Multi-Modal Visualization

The main goal of this master thesis is the development of new fusion strategies that enhance multimodal visualization strategies

Intensity based image registration of satellite images using evolutionary techniques

Image registration is the fundamental image processing technique to determine geometrical transformation that gives the most accurate match between reference and floating images. Its main aim is to align two images. Satellite images to be fused for numerous applications must be registered before use. The main challenges in satellite image registration are finding out the optimum transformation parameters. Here in this work the non-alignment parameters are considered to be rigid and affine transformation. An intensity based satellite image registration technique is being used to register the floating image to the native co-ordinate system where the normalized mutual information (NMI) is taken as the similarity metric for optimizing and updating transform parameters. Because of no assumptions are made regarding the nature of the relationship between the image intensities in both modalities NMI is very general and powerful and can be applied automatically without prior segmentation on a large variety of data and as well works better for overlapped images as compared to mutual information(MI). In order to get maximum accuracy of registration the NMI is optimized using Genetic algorithm, particle swarm optimization and hybrid GA-PSO. The random initialization and computational complexity makes GA oppressive, whereas weak local search ability with a premature convergence is the main drawback of PSO. Hybrid GA-PSO makes a trade-off between the local and global search in order to achieve a better balance between convergence speed and computational complexity. The above registration algorithm is being validated with several satellite data sets. The hybrid GA-PSO outperforms in terms of optimized NMI value and percentage of mis-registration error

Mesh-to-raster based non-rigid registration of multi-modal images

Region of interest (ROI) alignment in medical images plays a crucial role in diagnostics, procedure planning, treatment, and follow-up. Frequently, a model is represented as triangulated mesh while the patient data is provided from CAT scanners as pixel or voxel data. Previously, we presented a 2D method for curve-to-pixel registration. This paper contributes (i) a general mesh-to-raster (M2R) framework to register ROIs in multi-modal images; (ii) a 3D surface-to-voxel application, and (iii) a comprehensive quantitative evaluation in 2D using ground truth provided by the simultaneous truth and performance level estimation (STAPLE) method. The registration is formulated as a minimization problem where the objective consists of a data term, which involves the signed distance function of the ROI from the reference image, and a higher order elastic regularizer for the deformation. The evaluation is based on quantitative light-induced fluoroscopy (QLF) and digital photography (DP) of decalcified teeth. STAPLE is computed on 150 image pairs from 32 subjects, each showing one corresponding tooth in both modalities. The ROI in each image is manually marked by three experts (900 curves in total). In the QLF-DP setting, our approach significantly outperforms the mutual information-based registration algorithm implemented with the Insight Segmentation and Registration Toolkit (ITK) and Elastix

Combined Mutual Information of Intensity and Gradient for Multi-modal Medical Image Registration

In this thesis, registration methods for multi-modal medical images are reviewed with mutual information-based methods discussed in detail. Since it was proposed, mutual information has gained intensive research and is getting very popular, however its robustness is questionable and may fail in some cases. The possible reason might be it does not consider the spatial information in the image pair. In order to improve this measure, the thesis proposes to use combined mutual information of intensity and gradient for multi-modal medical image registration. The proposed measure utilizes both the intensity and gradient information of an image pair. Maximization of this measure is assumed to correctly register an image pair. Optimization of the registration measure in a multi-dimensional space is another major issue in multi-modal medical image registration. The thesis first briefly reviews the commonly used optimization techniques and then discusses in detail the Powell\u27s conjugate direction set method, which is implemented to find the maximum of the combined mutual information of an image pair. In the experiment, we first register slice images scanned in a single patient in the same or different scanning sessions by the proposed method. Then 20 pairs of co-registered CT and PET slice images at three different resolutions are used to study the performance of the proposed measure and four other measures discussed in this thesis. Experimental results indicate that the proposed combined measure produces reliable registrations and it outperforms the intensity- and gradient-based measures at all three resolutions

HeMIS: Hetero-Modal Image Segmentation

We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, without having to learn a combinatorial number of imputation models. Evaluated on two neurological MRI datasets (brain tumors and MS lesions), the approach yields state-of-the-art segmentation results when provided with all modalities; moreover, its performance degrades remarkably gracefully when modalities are removed, significantly more so than alternative mean-filling or other synthesis approaches.Comment: Accepted as an oral presentation at MICCAI 201