Time-of-flight CMOS イメージセンサによる高精度・高速距離イメージングに関する研究

Tohoku University博士（工学）thesi

CMOS Sensors for Time-Resolved Active Imaging

In the past decades, time-resolved imaging such as fluorescence lifetime or time-of-flight depth imaging has been extensively explored in biomedical and industrial fields because of its non-invasive characterization of material properties and remote sensing capability. Many studies have shown its potential and effectiveness in applications such as cancer detection and tissue diagnoses from fluorescence lifetime imaging, and gesture/motion sensing and geometry sensing from time-of-flight imaging. Nonetheless, time-resolved imaging has not been widely adopted due to the high cost of the system and performance limits. The research presented in this thesis focuses on the implementation of low-cost real-time time-resolved imaging systems. Two image sensing schemes are proposed and implemented to address the major limitations. First, we propose a single-shot fluorescence lifetime image sensors for high speed and high accuracy imaging. To achieve high accuracy, previous approaches repeat the measurement for multiple sampling, resulting in long measurement time. On the other hand, the proposed method achieves both high speed and accuracy at the same time by employing a pixel-level processor that takes and compresses the multiple samples within a single measurement time. The pixels in the sensor take multiple samples from the fluorescent optical signal in sub-nanosecond resolution and compute the average photon arrival time of the optical signal. Thanks to the multiple sampling of the signal, the measurement is insensitive to the shape or the pulse-width of excitation, providing better accuracy and pixel uniformity than conventional rapid lifetime determination (RLD) methods. The proposed single-shot image sensor also improves the imaging speed by orders of magnitude compared to other conventional center-of-mass methods (CMM). Second, we propose a 3-D camera with a background light suppression scheme which is adaptable to various lighting conditions. Previous 3-D cameras are not operable in outdoor conditions because they suffer from measurement errors and saturation problems under high background light illumination. We propose a reconfigurable architecture with column-parallel discrete-time background light cancellation circuit. Implementing the processor at the column level allows an order of magnitude reduction in pixel size as compared to existing pixel-level processors. The column-level approach also provides reconfigurable operation modes for optimal performance in all lighting conditions. For example, the sensor can operate at the best frame-rate and resolution without the presence of background light. If the background light saturates the sensor or increases the shot noise, the sensor can adjust the resolution and frame-rate by pixel binning and superresolution techniques. This effectively enhances the well capacity of the pixel to compensate for the increase shot noise, and speeds up the frame processing to handle the excessive background light. A fabricated prototype sensor can suppress the background light more than 100-klx while achieving a very small pixel size of 5.9μm.PHDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/136950/1/eecho_1.pd

Real-time spatial modeling to detect and track resources on construction sites

For more than 10 years the U.S. construction industry has experienced over 1,000 fatalities annually. Many fatalities may have been prevented had the individuals and equipment involved been more aware of and alert to the physical state of the environment around them. Awareness may be improved by automatic 3D (three-dimensional) sensing and modeling of the job site environment in real-time. Existing 3D modeling approaches based on range scanning techniques are capable of modeling static objects only, and thus cannot model in real-time dynamic objects in an environment comprised of moving humans, equipment, and materials. Emerging prototype 3D video range cameras offer another alternative by facilitating affordable, wide field of view, automated static and dynamic object detection and tracking at frame rates better than 1Hz (real-time). This dissertation presents an imperical work and methodology to rapidly create a spatial model of construction sites and in particular to detect, model, and track the position, dimension, direction, and velocity of static and moving project resources in real-time, based on range data obtained from a three-dimensional video range camera in a static or moving position. Existing construction site 3D modeling approaches based on optical range sensing technologies (laser scanners, rangefinders, etc.) and 3D modeling approaches (dense, sparse, etc.) that offered potential solutions for this research are reviewed. The choice of an emerging sensing tool and preliminary experiments with this prototype sensing technology are discussed. These findings led to the development of a range data processing algorithm based on three-dimensional occupancy grids which is demonstrated in detail. Testing and validation of the proposed algorithms have been conducted to quantify the performance of sensor and algorithm through extensive experimentation involving static and moving objects. Experiments in indoor laboratory and outdoor construction environments have been conducted with construction resources such as humans, equipment, materials, or structures to verify the accuracy of the occupancy grid modeling approach. Results show that modeling objects and measuring their position, dimension, direction, and speed had an accuracy level compatible to the requirements of active safety features for construction. Results demonstrate that video rate 3D data acquisition and analysis of construction environments can support effective detection, tracking, and convex hull modeling of objects. Exploiting rapidly generated three-dimensional models for improved visualization, communications, and process control has inherent value, broad application, and potential impact, e.g. as-built vs. as-planned comparison, condition assessment, maintenance, operations, and construction activities control. In combination with effective management practices, this sensing approach has the potential to assist equipment operators to avoid incidents that result in reduce human injury, death, or collateral damage on construction sites.Civil, Architectural, and Environmental Engineerin

Fluorescence Lifetime Imaging Camera: Image Analysis, Optimization and Enhancement

Fluorescence lifetime imaging microscopy (FLIM) is an imaging technique for producing an image based on differences in fluorescence lifetimes. The present thesis is devoted to analyzing a novel Fluorescence Lifetime Imaging Camera (FLI-Cam) system. The principle of the applied camera system is based on the Time-of-Flight (ToF) technique, which was originally designed for 3D depth scene imaging. Such a camera provides a high frame rate and realizes direct nanosecond-range fluorescence lifetime sensing. The main scope of this thesis is to deliver an optimized solution and rapid sophisticated algorithm for the FLI-Cam system with high accuracy. New time-gated schemes and heterodyne modulation scheme for FLIM using the pulse-based and continuous-wave-based (phase-based) ToF camera, respectively, are presented. In order to optimize the performance of the FLI-Cam system, a thorough statistical analysis is implemented and the photon economy of our FLIM techniques is investigated. Various operation modes and experimental parameters for the measurement have been studied and optimized. The presented theoretical result is validated by numerical simulations using the Monte Carlo method and real experiments. For the enhancement of the FLIM images from our system, the vector-valued total variation technique is applied to improve the quality of FLIM images for the first time. It shows better performance than other existing approaches

Optical Synchronization of Time-of-Flight Cameras

Time-of-Flight (ToF)-Kameras erzeugen Tiefenbilder (3D-Bilder), indem sie Infrarotlicht aussenden und die Zeit messen, bis die Reflexion des Lichtes wieder empfangen wird. Durch den Einsatz mehrerer ToF-Kameras können ihre vergleichsweise geringere Auflösungen überwunden, das Sichtfeld vergrößert und Verdeckungen reduziert werden. Der gleichzeitige Betrieb birgt jedoch die Möglichkeit von Störungen, die zu fehlerhaften Tiefenmessungen führen. Das Problem der gegenseitigen Störungen tritt nicht nur bei Mehrkamerasystemen auf, sondern auch wenn mehrere unabhängige ToF-Kameras eingesetzt werden. In dieser Arbeit wird eine neue optische Synchronisation vorgestellt, die keine zusätzliche Hardware oder Infrastruktur erfordert, um ein Zeitmultiplexverfahren (engl. Time-Division Multiple Access, TDMA) für die Anwendung mit ToF-Kameras zu nutzen, um so die Störungen zu vermeiden. Dies ermöglicht es einer Kamera, den Aufnahmeprozess anderer ToF-Kameras zu erkennen und ihre Aufnahmezeiten schnell zu synchronisieren, um störungsfrei zu arbeiten. Anstatt Kabel zur Synchronisation zu benötigen, wird nur die vorhandene Hardware genutzt, um eine optische Synchronisation zu erreichen. Dazu wird die Firmware der Kamera um das Synchronisationsverfahren erweitert. Die optische Synchronisation wurde konzipiert, implementiert und in einem Versuchsaufbau mit drei ToF-Kameras verifiziert. Die Messungen zeigen die Wirksamkeit der vorgeschlagenen optischen Synchronisation. Während der Experimente wurde die Bildrate durch das zusätzliche Synchronisationsverfahren lediglich um etwa 1 Prozent reduziert.Time-of-Flight (ToF) cameras produce depth images (three-dimensional images) by measuring the time between the emission of infrared light and the reception of its reflection. A setup of multiple ToF cameras may be used to overcome their comparatively low resolution, increase the field of view, and reduce occlusion. However, the simultaneous operation of multiple ToF cameras introduces the possibility of interference resulting in erroneous depth measurements. The problem of interference is not only related to a collaborative multicamera setup but also to multiple ToF cameras operating independently. In this work, a new optical synchronization for ToF cameras is presented, requiring no additional hardware or infrastructure to utilize a time-division multiple access (TDMA) scheme to mitigate interference. It effectively enables a camera to sense the acquisition process of other ToF cameras and rapidly synchronizes its acquisition times to operate without interference. Instead of requiring cables to synchronize, only the existing hardware is utilized to enable an optical synchronization. To achieve this, the camera’s firmware is extended with the synchronization procedure. The optical synchronization has been conceptualized, implemented, and verified with an experimental setup deploying three ToF cameras. The measurements show the efficacy of the proposed optical synchronization. During the experiments, the frame rate was reduced by only about 1% due to the synchronization procedure

Design of optical fiber sensors and interrogation schemes

[ES] Las fibras ópticas son dispositivos muy utilizados en el campo de las telecomunicaciones desde su descubrimiento. En las últimas décadas, las fibras ópticas comenzaron a utilizarse como sensores fotónicos. Los primeros trabajos se centraron en la medición de unas dimensiones físicas en un punto específico. Posteriormente, surgió la posibilidad de medir las propiedades de la fibra óptica en diferentes puntos a lo largo de la fibra. Este tipo de sensores se definen como sensores distribuidos. Los componentes optoelectrónicos fueron desarrollados e investigados para telecomunicaciones. Los avances en las telecomunicaciones hicieron posible el desarrollo de sistemas de interrogación para sensores de fibra óptica, creciendo en paralelo con los avances de las telecomunicaciones. Se desarrollaron sistemas de interrogación de fibra óptica que permiten el uso de una única fibra óptica monomodo estándar como sensor que puede monitorear decenas de miles de puntos de detección al mismo tiempo. Los métodos que extraen la información de detección de la señal reflejada en la fibra óptica son los más empleados debido a la facilidad de acceso al sensor y la flexibilidad de estos sistemas. Los más estudiados son la reflectometría en dominios de tiempo y frecuencia. La reflectometría óptica en el dominio del tiempo (OTDR) fue la primera técnica utilizada para detectar la posición de los fallos en las redes de comunica-ción de fibra óptica. El OTDR sensible a la fase hizo posible detectar la elongación y la temperatura en una posición específica. Paralelamente, los gratings de Bragg (FBG) se convirtieron en los dispositivos más utilizados para implementar sensores en fibra óptica discretos. Se desarrollaron técnicas de multiplexación para realizar la detección en múltiples puntos utilizando FGBs. La reflectometría realizada interrogando arrays de FBG débiles demuestra que mejora el rendimiento del sistema en comparación al uso de una fibra monomodo. Los sistemas de interrogatorio actuales tienen algunos inconvenientes. Algunos de ellos son velocidad de interrogatorio limitada, grandes dimensiones y alto costo. En esta tesis doctoral se desarrollaron nuevos sistemas de interrogación y sensores de fibra óptica para superar algunos de estos inconvenientes. Los sensores de fibra óptica de plástico demuestran ser una plataforma innovadora para desarrollar nuevos sensores y sistemas de interrogación de bajo costo y fáciles de implementar para fibras de plástico comerciales. Se investigó la reflectometría en el dominio del tiempo y las técnicas fotónicas de microondas para la interrogación de una matriz de rejillas débiles que permitieron simplificar el sistema de interrogación para la detección de temperatura y vibración.[CA] Les fibres òptiques són dispositius molt utilitzats en el camp de les telecomunica-cions des del seu descobriment. En les últimes dècades, les fibres òptiques van començar a utilitzar-se com a sensors fotònics. Els primers treballs es van centrar en el mesurament d'unes dimensions físiques en un punt específic. Posteriorment, va sorgir la possibilitat de mesurar les propietats de la fibra òptica en diferents punts al llarg de la fibra. Aquest tipus de sensors es defineixen com a sensors distribüits. Els components optoelectrònics van ser desenvolupats i investigats per a telecomunicacions. Els avanços en les telecomunicacions van fer possi-ble el desenvolupament de sistemes d'interrogació per a sensors de fibra òptica, creixent en paral·lel amb els avanços de les telecomunicacions. Es van desenvolupar sistemes d'interrogació de fibra òptica que permeten l'ús d'una única fibra òptica monomodo estàndard com a sensor que pot monitorar desenes de milers de punts de detecció al mateix temps. Els mètodes que extreuen la informació de detecció del senyal reflectit en la fibra òptica són els més utilitzats a causa de la facilitat d'accés al sensor i la flexibilitat d'aquests sistemes. Els més estudiats són la reflectometría en dominis de temps i freqüència. La reflectometría òptica en el domini del temps (OTDR) va ser la primera tècnica utilitzada per a detectar la posició de les fallades en les xarxes de comunicació de fibra òptica. El OTDR sensible a la fase va fer possible detectar l'elongació i la temperatura en una posició específica. Paral·lelament, els gratings de Bragg (FBG) es van convertir en els dispositius més utilitzats per a implementar sensors en fibra òptica discrets. Es van desenvolupar tècniques de multiplexació per a realitzar la detecció en múltiples punts utilitzant FGBs. La reflectometría realitzada interrogant arrays de FBG febles demostra que millora el rendiment del sistema en comparació a l'ús d'una fibra monomodo. Els sistemes d'interrogatori actuals tenen alguns inconvenients. Alguns d'ells són velocitat d'interrogatori limitada, voluminositat i alt cost. En aquesta tesi doctoral es van desenvolupar nous sistemes d'interrogació i sensors de fibra òptica per a superar alguns d'aquests inconvenients. Els sensors de fibra òptica de plàstic demostren ser una plataforma innovadora per a desenvolupar nous sensors i siste-mes d'interrogació de baix cost i fàcils d'implementar per a fibres de plàstic comercials. Es va investigar la reflectometría en el domini del temps i les tècniques fotòniques de microones per a la interrogació d'una matriu de reixetes febles que van permetre simplificar el sistema d'interrogació per a la detecció de temperatura i vibració.[EN] Optical fibers are devices largely used in telecommunication field since their discovery. In the last decades, optical fibers started to be used as photonic sensors. The first works were focused on the measurement of physical dimensions to a specific point. Afterward, emerged the possibility to measure the optical fiber properties at different locations along the fiber. These kinds of sensors are defined as distributed sensors. The optoelectronic components were developed and investigated for telecommunications. The progress in telecommunication made possible the development of optical fiber sensors interrogation systems, growing in parallel with the advances of telecommunications. Optical fiber interrogation systems were developed to use a single standard monomode optical fiber as a sensor that can monitor tens of thousands of sensing points at the same time. The methods that extract the sensing information from the backscattered signal in the optical fiber are widely employed because of the easiness of access to the sensor element and the flexibility of these systems. The most studied are the reflectometry in time and frequency domains. The optical time domain reflectometry (OTDR) was the first technique used to detect the position of the failures in the optical fiber communication networks. Using phase sensitive OTDR it is possible to sense strain and temperature at a specific position. In parallel, fiber Bragg gratings (FBGs) became the most widely used devices to implement discrete optical fiber sensors. Multiplexing techniques were developed to perform multi points sensing using these gratings. The reflectometry performed interrogating weak FBGs arrays demonstrate to improve the performance of the system employing a single mode fiber. The interrogation systems nowadays have some drawbacks. Some of them are limited speed of interrogation, bulkiness, and high cost. New interrogation systems and optical fiber sensors were developed in this doctoral thesis to overcome some of these drawbacks. Plastic optical fiber sensors demonstrate to be an innovative platform to develop both new sensors and low cost, easy to implement interrogation systems for commercial plastic fibers. Reflectometry in time domain and microwave photonic techniques were investigated for the interrogation of weak gratings array allowed to simplify the interrogation system for the sensing of temperature and vibration.I would like to greatly thank the European Union’s Horizon 2020 Research and Innovation Program that funded the research described in this thesis under the Marie Sklodowska-Curie Action Grant Agreement 722509.Sartiano, D. (2021). Design of optical fiber sensors and interrogation schemes [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/161357TESI

Electric field imaging

Thesis (Ph.D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts & Sciences, 1999.Includes bibliographical references (p. 213-216).The physical user interface is an increasingly significant factor limiting the effectiveness of our interactions with and through technology. This thesis introduces Electric Field Imaging, a new physical channel and inference framework for machine perception of human action. Though electric field sensing is an important sensory modality for several species of fish, it has not been seriously explored as a channel for machine perception. Technological applications of field sensing, from the Theremin to the capacitive elevator button, have been limited to simple proximity detection tasks. This thesis presents a solution to the inverse problem of inferring geometrical information about the configuration and motion of the human body from electric field measurements. It also presents simple, inexpensive hardware and signal processing techniques for making the field measurements, and several new applications of electric field sensing. The signal processing contribution includes synchronous undersampling, a narrowband, phase sensitive detection technique that is well matched to the capabilities of contemporary microcontrollers. In hardware, the primary contributions are the School of Fish, a scalable network of microcontroller-based transceive electrodes, and the LazyFish, a small footprint integrated sensing board. Connecting n School of Fish electrodes results in an array capable of making heterodyne measurements of any or all n(n - 1) off-diagonal entries in the capacitance matrix. The LazyFish uses synchronous undersampling to provide up to 8 high signal-to-noise homodyne measurements in a very small package. The inverse electrostatics portion of the thesis presents a fast, general method for extracting geometrical information about the configuration and motion of the human body from field measurements. The method is based on the Sphere Expansion, a novel fast method for generating approximate solutions to the Laplace equation. Finally, the thesis describes a variety of applications of electric field sensing, many enabled by the small footprint of the LazyFish. To demonstrate the School of Fish hardware and the Sphere Expansion inversion method, the thesis presents 3 dimensional position and orientation tracking of two hands.by Joshua Reynolds Smith.Ph.D