Medical image colorization for better visualization and segmentation

Medical images contain precious anatomical information for clinical procedures. Improved understanding of medical modality may contribute significantly in arena of medical image analysis. This paper investigates enhancement of monochromatic medical modality into colorized images. Improving the contrast of anatomical structures facilitates precise segmentation. The proposed framework starts with pre-processing to remove noise and improve edge information. Then colour information is embedded to each pixel of a subject image. A resulting image has a potential to portray better anatomical information than a conventional monochromatic image. To evaluate the performance of colorized medical modality, the structural similarity index and the peak signal to noise ratio are computed. Supremacy of proposed colorization is validated by segmentation experiments and compared with greyscale monochromatic images

An interactive color pre-processing method to improve tumor segmentation in digital medical images

In the last few decades the medical imaging field has grown considerably, and new techniques such as computerized axial tomography (CAT) and Magnetic Resonance Imaging (MRI) are able to obtain medical images in noninvasive ways. These new technologies have opened the medical field, offering opportunities to improve patient diagnosis, education and training, treatment monitoring, and surgery planning. One of these opportunities is in the tumor segmentation field. Tumor segmentation is the process of virtually extracting the tumor from the healthy tissues of the body by computer algorithms. This is a complex process since tumors have different shapes, sizes, tissue densities, and locations. The algorithms that have been developed cannot take into account all these variations and higher accuracy is achieved with specialized methods that generally work with specific types of tissue data. In this thesis a color pre-processing method for segmentation is presented. Most tumor segmentation methods are based on grayscale values of the medical images. The method proposed in this thesis adds color information to the original values of the image. The user selects the region of interest (ROI), usually the tumor, from the grayscale medical image and from this initial selection, the image is mapped into a colored space. Tissue densities that are part of the tumor are assigned an RGB component and any tissues outside the tumor are set to black. The user can tweak the color ranges in real time to achieve better results, in cases where the tumor pixels are non-homogenous in terms of intensity. The user then places a seed in the center of the tumor and begins segmentation. A pixel in the image is segmented as part of the tumor if it\u27s within an initial 10% threshold. This threshold is determined if the seed is within the average RGB values of the tumor, and within the search region. The search region is calculated by growing or shrinking the previous region using the information or previous segmented regions of the set of slices. The method automatically segments all the slices on the set from the inputs of the first slice. All through the segmentation process the user can tweak different parameters and visualize the segmentation results in real time. The method was run on ten test cases several runs were performed for each test cases. 10 out of the 20 test runs gave false positives of 25% or less, and 10 out of the 20 test runs gave false negatives of 25% or less. Using only grayscale thresholding methods the results for the same test cases show a false positive of up to 52% on the easy cases and up to 284% on the difficult cases, and false negatives of up to 14% on the easy cases and up to 99% on the difficult cases. While the results of the grayscale and color pre-processing methods on easy cases were similar, the results of color pre-processing were much better on difficult cases, thus supporting the claim that adding color to medical images for segmentation can significantly improve accuracy of tumor segmentation

Two Decades of Colorization and Decolorization for Images and Videos

Colorization is a computer-aided process, which aims to give color to a gray image or video. It can be used to enhance black-and-white images, including black-and-white photos, old-fashioned films, and scientific imaging results. On the contrary, decolorization is to convert a color image or video into a grayscale one. A grayscale image or video refers to an image or video with only brightness information without color information. It is the basis of some downstream image processing applications such as pattern recognition, image segmentation, and image enhancement. Different from image decolorization, video decolorization should not only consider the image contrast preservation in each video frame, but also respect the temporal and spatial consistency between video frames. Researchers were devoted to develop decolorization methods by balancing spatial-temporal consistency and algorithm efficiency. With the prevalance of the digital cameras and mobile phones, image and video colorization and decolorization have been paid more and more attention by researchers. This paper gives an overview of the progress of image and video colorization and decolorization methods in the last two decades.Comment: 12 pages, 19 figure

Recovering Stereo Pairs from Anaglyphs

International audienceAn anaglyph is a single image created by selecting complementary colors from a stereo color pair; the user can perceive depth by viewing it through color-filtered glasses. We propose a technique to reconstruct the original color stereo pair given such an anaglyph. We modified SIFT-Flow and use it to initially match the different color channels across the two views. Our technique then iteratively refines the matches, selects the good matches (which defines the "anchor" colors), and propagates the anchor colors. We use a diffusion-based technique for the color propagation, and added a step to suppress unwanted colors. Results on a variety of inputs demonstrate the robustness of our technique. We also extended our method to anaglyph videos by using optic flow between time frames

Color Image Enhancement Techniques for Endoscopic images

Modern endoscopes play an important role in diagnosing various gastrointestinal (GI) tract related diseases. Although clinical findings of modern endoscopic imaging techniques are encouraging, there still remains much room for improvement of image quality. Of greatest concern, endoscopic images suffer from various degradations, such as specular highlights, non-uniform brightness and poor contrast. As a result, gastroenterologists often face difficulty in successfully identifying the subtle features, such as mucosal surface and structures, pit patterns, size and pattern of micro-vessels, tissue and vascular characteristics, superficial layer of mucosal and abnormal growths in endoscopic images. The improved visual quality of images can provide better diagnosis. This paper presents two proposed post-processing techniques for enhancing the subtle features of endoscopic images. The first proposed technique is named as endoscopic image enhancement based on adaptive sigmoid function and space-variant color reproduction (ASSVCR). It is achieved in two stages: image enhancement at gray level followed by color reproduction with the help of space variant chrominance mapping. Image enhancement is achieved by performing adaptive sigmoid function and uniform distribution of sigmoid pixels. Then color reproduction is used to generate new chrominance components. The second proposed technique is named as tri-scan. It is achieved in three stages: (1) Tissue and surface enhancement: a modified linear unsharp masking is used to sharpen the surface and edges of tissue and vascular characteristics, (2) Mucosa layer enhancement: an adaptive sigmoid function similar to the ASSVCR technique is employed on the R plane of the image to highlight the superficial layers of mucosa, (3) Color tone enhancement: the pixels are uniformly distributed to create a different color effect to highlight mucosa structures, superficial layers of mucosa and tissue characteristics. Both techniques are compared with other related works. Several performance metrics like focus value, statistic of visual representation, measurement of uniform distribution, color similarity test, color enhancement factor (CEF) and time complexity are used to assess the performance. The results showed improved performance compared to similar existing methods. In the post-processed images, we have observed that the ASSVCR can enhance and highlight pit patterns, tissue and vascular characteristics, mucosa structures and abnormal growths. It cannot highlight size and pattern of micro-vessels, and superficial layer of mucosa. In contrast, tri-scan can enhance and highlight all above mentioned features of endoscopic images

Doctor of Philosophy

dissertationConfocal microscopy has become a popular imaging technique in biology research in recent years. It is often used to study three-dimensional (3D) structures of biological samples. Confocal data are commonly multichannel, with each channel resulting from a different fluorescent staining. This technique also results in finely detailed structures in 3D, such as neuron fibers. Despite the plethora of volume rendering techniques that have been available for many years, there is a demand from biologists for a flexible tool that allows interactive visualization and analysis of multichannel confocal data. Together with biologists, we have designed and developed FluoRender. It incorporates volume rendering techniques such as a two-dimensional (2D) transfer function and multichannel intermixing. Rendering results can be enhanced through tone-mappings and overlays. To facilitate analyses of confocal data, FluoRender provides interactive operations for extracting complex structures. Furthermore, we developed the Synthetic Brainbow technique, which takes advantage of the asynchronous behavior in Graphics Processing Unit (GPU) framebuffer loops and generates random colorizations for different structures in single-channel confocal data. The results from our Synthetic Brainbows, when applied to a sequence of developing cells, can then be used for tracking the movements of these cells. Finally, we present an application of FluoRender in the workflow of constructing anatomical atlases

An evaluation of partial differential equations based digital inpainting algorithms

Partial Differential equations (PDEs) have been used to model various phenomena/tasks in different scientific and engineering endeavours. This thesis is devoted to modelling image inpainting by numerical implementations of certain PDEs. The main objectives of image inpainting include reconstructing damaged parts and filling-in regions in which data/colour information are missing. Different automatic and semi-automatic approaches to image inpainting have been developed including PDE-based, texture synthesis-based, exemplar-based, and hybrid approaches. Various challenges remain unresolved in reconstructing large size missing regions and/or missing areas with highly textured surroundings. Our main aim is to address such challenges by developing new advanced schemes with particular focus on using PDEs of different orders to preserve continuity of textural and geometric information in the surrounding of missing regions. We first investigated the problem of partial colour restoration in an image region whose greyscale channel is intact. A PDE-based solution is known that is modelled as minimising total variation of gradients in the different colour channels. We extend the applicability of this model to partial inpainting in other 3-channels colour spaces (such as RGB where information is missing in any of the two colours), simply by exploiting the known linear/affine relationships between different colouring models in the derivation of a modified PDE solution obtained by using the Euler-Lagrange minimisation of the corresponding gradient Total Variation (TV). We also developed two TV models on the relations between greyscale and colour channels using the Laplacian operator and the directional derivatives of gradients. The corresponding Euler-Lagrange minimisation yields two new PDEs of different orders for partial colourisation. We implemented these solutions in both spatial and frequency domains. We measure the success of these models by evaluating known image quality measures in inpainted regions for sufficiently large datasets and scenarios. The results reveal that our schemes compare well with existing algorithms, but inpainting large regions remains a challenge. Secondly, we investigate the Total Inpainting (TI) problem where all colour channels are missing in an image region. Reviewing and implementing existing PDE-based total inpainting methods reveal that high order PDEs, applied to each colour channel separately, perform well but are influenced by the size of the region and the quantity of texture surrounding it. Here we developed a TI scheme that benefits from our partial inpainting approach and apply two PDE methods to recover the missing regions in the image. First, we extract the (Y, Cb, Cr) of the image outside the missing region, apply the above PDE methods for reconstructing the missing regions in the luminance channel (Y), and then use the colourisation method to recover the missing (Cb, Cr) colours in the region. We shall demonstrate that compared to existing TI algorithms, our proposed method (using 2 PDE methods) performs well when tested on large datasets of natural and face images. Furthermore, this helps understanding of the impact of the texture in the surrounding areas on inpainting and opens new research directions. Thirdly, we investigate existing Exemplar-Based Inpainting (EBI) methods that do not use PDEs but simultaneously propagate the texture and structure into the missing region by finding similar patches within the rest of image and copying them into the boundary of the missing region. The order of patch propagation is determined by a priority function, and the similarity is determined by matching criteria. We shall exploit recently emerging Topological Data Analysis (TDA) tools to create innovative EBI schemes, referred to as TEBI. TDA studies shapes of data/objects to quantify image texture in terms of connectivity and closeness properties of certain data landmarks. Such quantifications help determine the appropriate size of patch propagation and will be used to modify the patch propagation priority function using the geometrical properties of curvature of isophotes, and to improve the matching criteria of patches by calculating the correlation coefficients from the spatial, gradient and Laplacian domains. The performance of this TEBI method will be tested by applying it to natural dataset images, resulting in improved inpainting when compared with other EBI methods. Fourthly, the recent hybrid-based inpainting techniques are reviewed and a number of highly performing innovative hybrid techniques that combine the use of high order PDE methods with the TEBI method for the simultaneous rebuilding of the missing texture and structure regions in an image are proposed. Such a hybrid scheme first decomposes the image into texture and structure components, and then the missing regions in these components are recovered by TEBI and PDE based methods respectively. The performance of our hybrid schemes will be compared with two existing hybrid algorithms. Fifthly, we turn our attention to inpainting large missing regions, and develop an innovative inpainting scheme that uses the concept of seam carving to reduce this problem to that of inpainting a smaller size missing region that can be dealt with efficiently using the inpainting schemes developed above. Seam carving resizes images based on content-awareness of the image for both reduction and expansion without affecting those image regions that have rich information. The missing region of the seam-carved version will be recovered by the TEBI method, original image size is restored by adding the removed seams and the missing parts of the added seams are then repaired using a high order PDE inpainting scheme. The benefits of this approach in dealing with large missing regions are demonstrated. The extensive performance testing of the developed inpainting methods shows that these methods significantly outperform existing inpainting methods for such a challenging task. However, the performance is still not acceptable in recovering large missing regions in high texture and structure images, and hence we shall identify remaining challenges to be investigated in the future. We shall also extend our work by investigating recently developed deep learning based image/video colourisation, with the aim of overcoming its limitations and shortcoming. Finally, we should also describe our on-going research into using TDA to detect recently growing serious “malicious” use of inpainting to create Fake images/videos

Inertial Alternating Generalized Forward-Backward Splitting for Image Colorization

International audienceIn this paper, we propose a novel accelerated alternating optimization scheme to solve block-biconvex nonsmooth problems whose objectives can be split into smooth (separable) regularizers and simple coupling terms. The proposed method performs a Bregman distance based generalization of the well-known forward-backward splitting for each block, along with an inertial strategy which aims at getting empirical acceleration. We discuss the theoretical convergence of the proposed scheme and provide numerical experiments on image colorization