ImageSpirit: Verbal Guided Image Parsing

Humans describe images in terms of nouns and adjectives while algorithms operate on images represented as sets of pixels. Bridging this gap between how humans would like to access images versus their typical representation is the goal of image parsing, which involves assigning object and attribute labels to pixel. In this paper we propose treating nouns as object labels and adjectives as visual attribute labels. This allows us to formulate the image parsing problem as one of jointly estimating per-pixel object and attribute labels from a set of training images. We propose an efficient (interactive time) solution. Using the extracted labels as handles, our system empowers a user to verbally refine the results. This enables hands-free parsing of an image into pixel-wise object/attribute labels that correspond to human semantics. Verbally selecting objects of interests enables a novel and natural interaction modality that can possibly be used to interact with new generation devices (e.g. smart phones, Google Glass, living room devices). We demonstrate our system on a large number of real-world images with varying complexity. To help understand the tradeoffs compared to traditional mouse based interactions, results are reported for both a large scale quantitative evaluation and a user study.Comment: http://mmcheng.net/imagespirit

Spatial Frequency Domain Imaging of Short-Wave Infrared Fluorescence for Biomedical Applications

Fluorescence imaging has become a standard in many clinical applications, such as tumor and vasculature imaging. One application that is becoming more prominent in cancer treatment is fluorescence-guided surgery (FGS). Currently, FGS allows surgeons the ability to visually navigate tumors and tissue structures intraoperatively. As a result, they can remove tumor more efficiently while maintaining critical structures within the patient, creating better outcomes and lower recovery times. However, background fluorescence and inability to localize depth create challenges when determining resection boundaries. Different techniques, such as spatially modulating the illumination and imaging at longer light wavelengths, have been developed to accurately localize position and to maximize contrast of key structures at greater penetration depths in surgeries. However, combining these two techniques offers an opportunity to capitalize on their respective advantages in fluorescence imaging. In this work, we develop instrumentation combining two cutting-edge imaging methodologies; spatial frequency domain imaging (SFDI) and short-wave infrared (SWIR) fluorescence, and evaluate the capacity of this combination to improve image quality. Using conventional flat light imaging and SFDI, images of a fluorescent tube submerged in a liquid phantom were taken in the standard “first window” of the near-infrared (NIR-I) and the SWIR. By comparing resolution and maximum signal of the resulting images, we determined whether SFDI in the SWIR can improve fluorescence imaging for surgery such as to increase sharpness and determine depth of fluorescent structures. We found that SFDI in the SWIR localizes depth and increases contrast by removing background fluorescence. When compared to NIR-I, SWIR SFDI is not as depth sensitive but can capture deeper structures. These results show that implementing SFDI can significantly improve current biomedical applications of SWIR imaging. Next steps for SWIR SFDI include analyzing the effects from altering the optical properties of the phantom, investigating the interaction between multiple fluorescent objects, and conducting in vivo efficacy studies

Object detection for UAV from color and depth image

V práci jsou prezentovány přístupy pro detekci objektů v soutěži MBZIRC 2020. Objekty jsou polystyrénové cihly dané ve-likosti a barvy. Práce obsahuje návrh algo-ritmů pro detekci objektů z barevného ob-razu pomocí barevné segmentace, metody Grab and Cut, uzavírání cyklů a neurono-vých sítí. Dále byly otestovány možnosti detekce hranolů v datech ze senzoru In-tel RealSense D435. Na závěr byly obě metody použity společně pomocí datové fúze.The work contains a review of different methods for object detection related to the MBZIRC 2020 competition. The objects are polystyrene blocks of a given size and color. The work demonstrates the design of algorithms for detection of cuboid objects from a color image using Color Segmentation, GrabCut methods, Closed Loop search and Neural Networks. Furthermore, it reviews methods of detection for depth image and geometrical verification of detected objects. Finally, both methods were used together for data fusion and its performance was reviewed. The data is obtained by Intel RealSense D435 camera

Locally Adaptive Stereo Vision Based 3D Visual Reconstruction

abstract: Using stereo vision for 3D reconstruction and depth estimation has become a popular and promising research area as it has a simple setup with passive cameras and relatively efficient processing procedure. The work in this dissertation focuses on locally adaptive stereo vision methods and applications to different imaging setups and image scenes. Solder ball height and substrate coplanarity inspection is essential to the detection of potential connectivity issues in semi-conductor units. Current ball height and substrate coplanarity inspection tools are expensive and slow, which makes them difficult to use in a real-time manufacturing setting. In this dissertation, an automatic, stereo vision based, in-line ball height and coplanarity inspection method is presented. The proposed method includes an imaging setup together with a computer vision algorithm for reliable, in-line ball height measurement. The imaging setup and calibration, ball height estimation and substrate coplanarity calculation are presented with novel stereo vision methods. The results of the proposed method are evaluated in a measurement capability analysis (MCA) procedure and compared with the ground-truth obtained by an existing laser scanning tool and an existing confocal inspection tool. The proposed system outperforms existing inspection tools in terms of accuracy and stability. In a rectified stereo vision system, stereo matching methods can be categorized into global methods and local methods. Local stereo methods are more suitable for real-time processing purposes with competitive accuracy as compared with global methods. This work proposes a stereo matching method based on sparse locally adaptive cost aggregation. In order to reduce outlier disparity values that correspond to mis-matches, a novel sparse disparity subset selection method is proposed by assigning a significance status to candidate disparity values, and selecting the significant disparity values adaptively. An adaptive guided filtering method using the disparity subset for refined cost aggregation and disparity calculation is demonstrated. The proposed stereo matching algorithm is tested on the Middlebury and the KITTI stereo evaluation benchmark images. A performance analysis of the proposed method in terms of the I0 norm of the disparity subset is presented to demonstrate the achieved efficiency and accuracy.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201

DESIGN, FABRICATION AND TESTING OF HIERARCHICAL MICRO-OPTICAL STRUCTURES AND SYSTEMS

Micro-optical systems are becoming essential components in imaging, sensing, communications, computing, and other applications. Optically based designs are replacing electronic, chemical and mechanical systems for a variety of reasons, including low power consumption, reduced maintenance, and faster operation. However, as the number and variety of applications increases, micro-optical system designs are becoming smaller, more integrated, and more complicated. Micro and nano-optical systems found in nature, such as the imaging systems found in many insects and crustaceans, can have highly integrated optical structures that vary in size by orders of magnitude. These systems incorporate components such as compound lenses, anti-reflective lens surface structuring, spectral filters, and polarization selective elements. For animals, these hybrid optical systems capable of many optical functions in a compact package have been repeatedly selected during the evolutionary process. Understanding the advantages of these designs gives motivation for synthetic optical systems with comparable functionality. However, alternative fabrication methods that deviate from conventional processes are needed to create such systems. Further complicating the issue, the resulting device geometry may not be readily compatible with existing measurement techniques. This dissertation explores several nontraditional fabrication techniques for optical components with hierarchical geometries and measurement techniques to evaluate performance of such components. A micro-transfer molding process is found to produce high-fidelity micro-optical structures and is used to fabricate a spectral filter on a curved surface. By using a custom measurement setup we demonstrate that the spectral filter retains functionality despite the nontraditional geometry. A compound lens is fabricated using similar fabrication techniques and the imaging performance is analyzed. A spray coating technique for photoresist application to curved surfaces combined with interference lithography is also investigated. Using this technique, we generate polarizers on curved surfaces and measure their performance. This work furthers an understanding of how combining multiple optical components affects the performance of each component, the final integrated devices, and leads towards realization of biomimetically inspired imaging systems

A Low-cost Depth Imaging Mobile Platform for Canola Phenotyping

To meet the high demand for supporting and accelerating progress in the breeding of novel traits, plant scientists and breeders have to measure a large number of plants and their characteristics accurately. A variety of imaging methodologies are being deployed to acquire data for quantitative studies of complex traits. When applied to a large number of plants such as canola plants, however, a complete three-dimensional (3D) model is time-consuming and expensive for high-throughput phenotyping with an enormous amount of data. In some contexts, a full rebuild of entire plants may not be necessary. In recent years, many 3D plan phenotyping techniques with high cost and large-scale facilities have been introduced to extract plant phenotypic traits, but these applications may be affected by limited research budgets and cross environments. This thesis proposed a low-cost depth and high-throughput phenotyping mobile platform to measure canola plant traits in cross environments. Methods included detecting and counting canola branches and seedpods, monitoring canola growth stages, and fusing color images to improve images resolution and achieve higher accuracy. Canola plant traits were examined in both controlled environment and field scenarios. These methodologies were enhanced by different imaging techniques. Results revealed that this phenotyping mobile platform can be used to investigate canola plant traits in cross environments with high accuracy. The results also show that algorithms for counting canola branches and seedpods enable crop researchers to analyze the relationship between canola genotypes and phenotypes and estimate crop yields. In addition to counting algorithms, fusing techniques can be helpful for plant breeders with more comfortable access plant characteristics by improving the definition and resolution of color images. These findings add value to the automation, low-cost depth and high-throughput phenotyping for canola plants. These findings also contribute a novel multi-focus image fusion that exhibits a competitive performance with outperforms some other state-of-the-art methods based on the visual saliency maps and gradient domain fast guided filter. This proposed platform and counting algorithms can be applied to not only canola plants but also other closely related species. The proposed fusing technique can be extended to other fields, such as remote sensing and medical image fusion

Método para el registro automático de imágenes basado en transformaciones proyectivas planas dependientes de las distancias y orientado a imágenes sin características comunes

Tesis inédita de la Universidad Complutense de Madrid, Facultad de Ciencias Físicas, Departamento de Arquitectura de Computadores y Automática, leída el 18-12-2015Multisensory data fusion oriented to image-based application improves the accuracy, quality and availability of the data, and consequently, the performance of robotic systems, by means of combining the information of a scene acquired from multiple and different sources into a unified representation of the 3D world scene, which is more enlightening and enriching for the subsequent image processing, improving either the reliability by using the redundant information, or the capability by taking advantage of complementary information. Image registration is one of the most relevant steps in image fusion techniques. This procedure aims the geometrical alignment of two or more images. Normally, this process relies on feature-matching techniques, which is a drawback for combining sensors that are not able to deliver common features. For instance, in the combination of ToF and RGB cameras, the robust feature-matching is not reliable. Typically, the fusion of these two sensors has been addressed from the computation of the cameras calibration parameters for coordinate transformation between them. As a result, a low resolution colour depth map is provided. For improving the resolution of these maps and reducing the loss of colour information, extrapolation techniques are adopted. A crucial issue for computing high quality and accurate dense maps is the presence of noise in the depth measurement from the ToF camera, which is normally reduced by means of sensor calibration and filtering techniques. However, the filtering methods, implemented for the data extrapolation and denoising, usually over-smooth the data, reducing consequently the accuracy of the registration procedure...La fusión multisensorial orientada a aplicaciones de procesamiento de imágenes, conocida como fusión de imágenes, es una técnica que permite mejorar la exactitud, la calidad y la disponibilidad de datos de un entorno tridimensional, que a su vez permite mejorar el rendimiento y la operatividad de sistemas robóticos. Dicha fusión, se consigue mediante la combinación de la información adquirida por múltiples y diversas fuentes de captura de datos, la cual se agrupa del tal forma que se obtiene una mejor representación del entorno 3D, que es mucho más ilustrativa y enriquecedora para la implementación de métodos de procesamiento de imágenes. Con ello se consigue una mejora en la fiabilidad y capacidad del sistema, empleando la información redundante que ha sido adquirida por múltiples sensores. El registro de imágenes es uno de los procedimientos más importantes que componen la fusión de imágenes. El objetivo principal del registro de imágenes es la consecución de la alineación geométrica entre dos o más imágenes. Normalmente, este proceso depende de técnicas de búsqueda de patrones comunes entre imágenes, lo cual puede ser un inconveniente cuando se combinan sensores que no proporcionan datos con características similares. Un ejemplo de ello, es la fusión de cámaras de color de alta resolución (RGB) con cámaras de Tiempo de Vuelo de baja resolución (Time-of-Flight (ToF)), con las cuales no es posible conseguir una detección robusta de patrones comunes entre las imágenes capturadas por ambos sensores. Por lo general, la fusión entre estas cámaras se realiza mediante el cálculo de los parámetros de calibración de las mismas, que permiten realizar la trasformación homogénea entre ellas. Y como resultado de este xii Abstract procedimiento, se obtienen mapas de profundad y de color de baja resolución. Con el objetivo de mejorar la resolución de estos mapas y de evitar la pérdida de información de color, se utilizan diversas técnicas de extrapolación de datos. Un factor crucial a tomar en cuenta para la obtención de mapas de alta calidad y alta exactitud, es la presencia de ruido en las medidas de profundidad obtenidas por las cámaras ToF. Este problema, normalmente se reduce mediante la calibración de estos sensores y con técnicas de filtrado de datos. Sin embargo, las técnicas de filtrado utilizadas, tanto para la interpolación de datos, como para la reducción del ruido, suelen producir el sobre-alisamiento de los datos originales, lo cual reduce la exactitud del registro de imágenes...Sección Deptal. de Arquitectura de Computadores y Automática (Físicas)Fac. de Ciencias FísicasTRUEunpu