Compressive Image Acquisition in Modern CMOS IC Design

Compressive Sampling (CS) offers bandwidth, power and memory size reduction compared to conventional (Nyquist) sampling. However, very few Integrated Circuit (IC) designs based on CS exist due to the missing link between the well-established CS theory on one side, and the practical aspects/effects related to physical IC design on the other side. This paper focuses on the application of compressed image acquisition in CMOS image sensor integrated circuit design. A new CS scheme is proposed which is suited for hardware implementation in CMOS IC design. All the main physical non-idealities are explained and carefully modeled. Their influences on the acquired image quality are analyzed in the general case and quantifed for the case of the proposed CS scheme. The presented methodology can also be used for different CS schemes and as a general guideline in future CS based CMOS image sensor designs

Exploring information retrieval using image sparse representations:from circuit designs and acquisition processes to specific reconstruction algorithms

New advances in the field of image sensors (especially in CMOS technology) tend to question the conventional methods used to acquire the image. Compressive Sensing (CS) plays a major role in this, especially to unclog the Analog to Digital Converters which are generally representing the bottleneck of this type of sensors. In addition, CS eliminates traditional compression processing stages that are performed by embedded digital signal processors dedicated to this purpose. The interest is twofold because it allows both to consistently reduce the amount of data to be converted but also to suppress digital processing performed out of the sensor chip. For the moment, regarding the use of CS in image sensors, the main route of exploration as well as the intended applications aims at reducing power consumption related to these components (i.e. ADC & DSP represent 99% of the total power consumption). More broadly, the paradigm of CS allows to question or at least to extend the Nyquist-Shannon sampling theory. This thesis shows developments in the field of image sensors demonstrating that is possible to consider alternative applications linked to CS. Indeed, advances are presented in the fields of hyperspectral imaging, super-resolution, high dynamic range, high speed and non-uniform sampling. In particular, three research axes have been deepened, aiming to design proper architectures and acquisition processes with their associated reconstruction techniques taking advantage of image sparse representations. How the on-chip implementation of Compressed Sensing can relax sensor constraints, improving the acquisition characteristics (speed, dynamic range, power consumption) ? How CS can be combined with simple analysis to provide useful image features for high level applications (adding semantic information) and improve the reconstructed image quality at a certain compression ratio ? Finally, how CS can improve physical limitations (i.e. spectral sensitivity and pixel pitch) of imaging systems without a major impact neither on the sensing strategy nor on the optical elements involved ? A CMOS image sensor has been developed and manufactured during this Ph.D. to validate concepts such as the High Dynamic Range - CS. A new design approach was employed resulting in innovative solutions for pixels addressing and conversion to perform specific acquisition in a compressed mode. On the other hand, the principle of adaptive CS combined with the non-uniform sampling has been developed. Possible implementations of this type of acquisition are proposed. Finally, preliminary works are exhibited on the use of Liquid Crystal Devices to allow hyperspectral imaging combined with spatial super-resolution. The conclusion of this study can be summarized as follows: CS must now be considered as a toolbox for defining more easily compromises between the different characteristics of the sensors: integration time, converters speed, dynamic range, resolution and digital processing resources. However, if CS relaxes some material constraints at the sensor level, it is possible that the collected data are difficult to interpret and process at the decoder side, involving massive computational resources compared to so-called conventional techniques. The application field is wide, implying that for a targeted application, an accurate characterization of the constraints concerning both the sensor (encoder), but also the decoder need to be defined

A review of snapshot multidimensional optical imaging: Measuring photon tags in parallel

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons’ spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition–also dubbed snapshot imaging–has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications

Light field image processing: an overview

Light field imaging has emerged as a technology allowing to capture richer visual information from our world. As opposed to traditional photography, which captures a 2D projection of the light in the scene integrating the angular domain, light fields collect radiance from rays in all directions, demultiplexing the angular information lost in conventional photography. On the one hand, this higher dimensional representation of visual data offers powerful capabilities for scene understanding, and substantially improves the performance of traditional computer vision problems such as depth sensing, post-capture refocusing, segmentation, video stabilization, material classification, etc. On the other hand, the high-dimensionality of light fields also brings up new challenges in terms of data capture, data compression, content editing, and display. Taking these two elements together, research in light field image processing has become increasingly popular in the computer vision, computer graphics, and signal processing communities. In this paper, we present a comprehensive overview and discussion of research in this field over the past 20 years. We focus on all aspects of light field image processing, including basic light field representation and theory, acquisition, super-resolution, depth estimation, compression, editing, processing algorithms for light field display, and computer vision applications of light field data

A review of snapshot multidimensional optical imaging: Measuring photon tags in parallel

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons’ spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition–also dubbed snapshot imaging–has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications

Diseño CMOS de un sistema de visión “on-chip” para aplicaciones de muy alta velocidad

Falta palabras claveEsta Tesis presenta arquitecturas, circuitos y chips para el diseño de sensores de visión CMOS con procesamiento paralelo embebido. La Tesis reporta dos chips, en concreto: El chip Q-Eye; El chip Eye-RIS_VSoC.. Y dos sistemas de visión construidos con estos chips y otros sistemas “off-chip” adicionales, como FPGAs, en concreto: El sistema Eye-RIS_v1; El sistema Eye-RIS_v2. Estos chips y sistemas están concebidos para ejecutar tareas de visión a muy alta velocidad y con consumos de potencia moderados. Los sistemas resultantes son, además, compactos y por lo tanto ventajosos en términos del factor SWaP cuando se los compara con arquitecturas convencionales formadas por sensores de imágenes convencionales seguidos de procesadores digitales. La clave de estas ventajas en términos de SWaP y velocidad radica en el uso de sensores-procesadores, en lugar de meros sensores, en la interface de los sistemas de visión. Estos sensores-procesadores embeben procesadores programables de señal-mixta dentro del pixel y son capaces tanto de adquirir imágenes como de pre-procesarlas para extraer características, eliminar información redundante y reducir el número de datos que se transmiten fuera del sensor para su procesamiento ulterior. El núcleo de la tesis es el sensor-procesador Q-Eye, que se usa como interface en los sistemas Eye-RIS. Este sensor-procesador embebe una arquitectura de procesamiento formada por procesadores de señal-mixta distribuidos por pixel. Sus píxeles son por tanto estructuras multi-funcionales complejas. De hecho, son programables, incorporan memorias e interactúan con sus vecinos para realizar una variedad de operaciones, tales como: Convoluciones lineales con máscaras programables; Difusiones controladas por tiempo y nivel de señal, a través de un “grid” resistivo embebido en el plano focal; Aritmética de imágenes; Flujo de programación dependiente de la señal; Conversión entre los dominios de datos: imagen en escala de grises e imagen binaria; Operaciones lógicas en imágenes binarias; Operaciones morfológicas en imágenes binarias. etc. Con respecto a otros píxeles multi-función y sensores-procesadores anteriores, el Q-Eye reporta entre otras las siguientes ventajas: Mayor calidad de la imagen y mejores prestaciones de las funcionalidades embebidas en el chip; Mayor velocidad de operación y mejor gestión de la energía disponible; Mayor versatilidad para integración en sistemas de visión industrial. De hecho, los sistemas Eye-RIS son los primeros sistemas de visión industriales dotados de las siguientes características: Procesamiento paralelo distribuido y progresivo; Procesadores de señal-mixta fiables, robustos y con errores controlados; Programabilidad distribuida. La Tesis incluye descripciones detalladas de la arquitectura y los circuitos usados en el pixel del Q-Eye, del propio chip Q-Eye y de los sistemas de visión construidos en base a este chip. Se incluyen también ejemplos de los distintos chips en operaciónThis Thesis presents architectures, circuits and chips for the implementation of CMOS VISION SENSORS with embedded parallel processing. The Thesis reports two chips, namely: Q-eye chip; Eye-RIS_VSoC chip, and two vision systems realized by using these chips and some additional “off-chip” circuitry, such as FPGAs. These vision systems are: Eye-RIS_v1 system; Eye-RIS_v2 system. The chips and systems reported in the Thesis are conceived to perform vision tasks at very high speed and with moderate power consumption. The proposed vision systems are also compact and advantageous in terms of SWaP factors as compared with conventional architectures consisting of standard image sensor followed by digital processors. The key of these advantages in terms of SWaP and speed lies in the use of sensors-processors, rather than mere sensors, in the front-end interface of vision systems. These sensors-processors embed mixed-signal programmable processors inside the pixel. Therefore, they are able to acquire images and process them to extract the features, removing the redundant information and reducing the data throughput for later processing. The core of the Thesis is the sensor-processor Q-Eye, which is used as front-end in the Eye-RIS systems. This sensor-processor embeds a processing architecture composed by mixed-signal processors distributed per pixel. Then, its pixels are complex multi-functional structures. In fact, they are programmable, incorporate memories and interact with its neighbors in order to carry out a set of operations, including: Linear convolutions with programmable linear masks; Time- and signal-controlled diffusions (by means of an embedded resistive grid); Image arithmetic; Signal-dependent data scheduling; Gray-scale to binary transformation; Logic operation on binary images; Mathematical morphology on binary images, etc. As compared with previous multi-function pixels and sensors-processors, the Q-Eye brings among other the following advantages: Higher image quality and better performances of functionalities embedded on chip; Higher operation speed and better management of energy budget; More versatility for integration in industrial vision systems. In fact, the Eye-RIS systems are the first industrial vision systems equipped with the following characteristics: Parallel distributed and progressive processing; Reliable, robust mixed-signal processors with handled errors; Distributed programmability. This Thesis includes detailed descriptions of architecture and circuits used in the Q-Eye pixel, in the Q-Eye chip itself and in the vision systems developed based on this chip. Also, several examples of chips and systems in operation are presented

The quantitative analysis of optical phase measurement and its application to the determination of corneal birefringence

In this thesis, a phase sensitive interferometer is successfully implemented to perform birefringent object surface-profile measurement, based on a polarisation adjustment approach. Using monochromatic light, a novel polarization interferometric method is developed, incorporating the birefringence technique and a waveplate. In our experiments, a birefringent wedge is designed for generating carrier fringes in the polariscope. Retardance is calculated from phase shifting using a phase matching technique. The accuracy of the method has been demonstrated to have an error of less than 0.02 radians. The accuracy and resolution quantitative analysis presented in this thesis can be used to determine accurately the phase-shifting interferometry for high-precision surface profile and bio-structure, such as fibre and collagen measurements with low cost. FFT technique and phase-stepping methods are described to determine birefringence within the cornea. The distribution of human corneal lamellar collagen is determined through a microscopic technique using the combination of a circular polariser and a quarter-wave retarder. A quantitative measure of corneal birefringence is achieved by phase unwrapping. The experimental findings of elliptic and hyperbolic populations of collagen fibrils may explain the optical phenomena of central corneal retardation with biaxial-like behaviour in more peripheral areas. A low cost, simple, and direct approach has been developed to make the required microscopic measurement. The traditional transmission system is improved by applying a reflection system with an LED light source and is suitable for the analysis of the birefringent cornea structures in vivo. A further instrument based upon a synthetic aperture approach has been created with the purpose of measuring the three dimensional birefringence structure of the cornea. The concept of the instrument is a combination of the parallax between individual lenses and the numerically generated planes of focus to visualise the phase structure

Optical Techniques for Fruit Firmness Assessment

This thesis describes the design and development of a new high-speed multispectral imaging (MSI) system compatible with a commercial grading line. The purpose of this system was to carry out spatially resolved spectroscopy to assess fruit firmness. Captured images were analysed using diffusion theory and modified Lorentzian models to extract a sample’s optical properties (absorption and reduced scattering coefficients) and optical parameters respectively. The high-speed MSI system was designed to capture images of fruit using a high-resolution complementary metal–oxide–semiconductor camera, 12.5 mm lens, and discrete lasers operating at 685, 850, and 904 nm. Each laser illuminates a separate fruit, and the camera captures the interacting light with a single frame encompassing all three fruit. Depending on the size of each fruit the spatial resolution with the 12.5 mm lens ranged from 0.15 to 0.22 mm/pixel. Initial measurements were made on 200 ‘Royal Gala’ apples to identify the relationships between the optical properties or parameters and either acoustic or the industry standard penetrometer firmness measurements. Performance of the high speed MSI system was poor compared to the results seen in the literature using alternative spatially resolved spectroscopic systems and other apple varieties. Only weak correlations (R = 0.33) were found between the individual optical measurements and firmness. Unsatisfactory performance from the high-speed system led to the development of a static MSI system to measure stationary fruit and the development of an inverse adding-doubling (IAD) system to provide an independent measurement of the samples optical properties. The purpose of these systems was to help understand the measurement, reduce variability, and give an indication of the upper level of performance possible. The static MSI system featured a number of improvements including the addition of a 980 nm laser, the elimination of an asymmetry caused by laser polarisation, improved temperature control, an electronic shutter system, precise location control of the fruit, and a new 25 mm lens improving spatial resolution (0.057mm/pixel). A second study was carried out using the new MSI and IAD systems on 92 ‘Royal Gala’ apples. Fruit were sliced to expose a flat measurement surface eliminating variation caused by fruit curvature and skin pigments. With these refinements and simplifications the relationships between optical properties or parameters and penetrometer firmness strengthened. As fruit softened and penetrometer firmness fell the reduced scattering coefficient measured by both the IAD and MSI system increased with correlation coefficients ranging from -0.62 to -0.70. The absorption coefficients measured by the two systems showed the expected features related to the absorption of chlorophyll and carotenoid pigments, and water absorption. As the fruit softened chlorophyll absorption decreased as the pigments are broken down and carotenoid absorption increased as new pigments are synthesised. No useful relationships were identified between the optical measurements and acoustic firmness. Multiple linear regression models were formed to predict penetrometer firmness using either the optical properties or modified Lorentzian parameters. The best performing model used a combination of the absorption and scattering coefficients, and had a correlation coefficient of 0.8 and a standard error of 5.87 N