A 30 µW 30 fps 110 × 110 Pixels Vision Sensor Embedding Local Binary Patterns

We present a 110 × 110 pixel vision sensor that computes the Local Binary Patterns (LBPs) of an imaged scene with a power consumption of 30 μW at 30 fps. The LBP of a given pixel is a binary vector, encoding the direction and sign of image contrast with respect to its neighbors. Each LBP provides a visual description of an image's local structure that is widely used for texture and object recognition. In the sensor proposed here, each pixel detects its corresponding LBP with respect to its four neighboring pixels and saves this information into a digital map using 6 bits to encode each pixel. The operation is executed during the exposure time and requires 83 pW/pixel · frame to be computed. The chip is implemented in a 0.35 μm CMOS featuring 34 T square pixels with 26 μm pitch. We illustrate some examples of image description based on the LBPs output by the sensor

Real-time 3-D Reconstruction by Means of Structured Light Illumination

Structured light illumination (SLI) is the process of projecting a series of light striped patterns such that, when viewed at an angle, a digital camera can reconstruct a 3-D model of a target object\u27s surface. But by relying on a series of time multiplexed patterns, SLI is not typically associated with video applications. For this purpose of acquiring 3-D video, a common SLI technique is to drive the projector/camera pair at very high frame rates such that any object\u27s motion is small over the pattern set. But at these high frame rates, the speed at which the incoming video can be processed becomes an issue. So much so that many video-based SLI systems record camera frames to memory and then apply off-line processing. In order to overcome this processing bottleneck and produce 3-D point clouds in real-time, we present a lookup-table (LUT) based solution that in our experiments, using a 640 by 480 video stream, can generate intermediate phase data at 1063.8 frames per second and full 3-D coordinate point clouds at 228.3 frames per second. These achievements are 25 and 10 times faster than previously reported studies. At the same time, a novel dual-frequency pattern is developed which combines a high-frequency sinusoid component with a unit-frequency sinusoid component, where the high-frequency component is used to generate robust phase information and the unit-frequency component is used to reduce phase unwrapping ambiguities. Finally, we developed a gamma model for SLI, which can correct the non-linear distortion caused by the optical devices. For three-step phase measuring profilometry (PMP), analysis of the root mean squared error of the corrected phase showed a 60х reduction in phase error when the gamma calibration is performed versus 33х reduction without calibration

Low-power CMOS digital-pixel Imagers for high-speed uncooled PbSe IR applications

This PhD dissertation describes the research and development of a new low-cost medium wavelength infrared MWIR monolithic imager technology for high-speed uncooled industrial applications. It takes the baton on the latest technological advances in the field of vapour phase deposition (VPD) PbSe-based medium wavelength IR (MWIR) detection accomplished by the industrial partner NIT S.L., adding fundamental knowledge on the investigation of novel VLSI analog and mixed-signal design techniques at circuit and system levels for the development of the readout integrated device attached to the detector. The work supports on the hypothesis that, by the use of the preceding design techniques, current standard inexpensive CMOS technologies fulfill all operational requirements of the VPD PbSe detector in terms of connectivity, reliability, functionality and scalability to integrate the device. The resulting monolithic PbSe-CMOS camera must consume very low power, operate at kHz frequencies, exhibit good uniformity and fit the CMOS read-out active pixels in the compact pitch of the focal plane, all while addressing the particular characteristics of the MWIR detector: high dark-to-signal ratios, large input parasitic capacitance values and remarkable mismatching in PbSe integration. In order to achieve these demands, this thesis proposes null inter-pixel crosstalk vision sensor architectures based on a digital-only focal plane array (FPA) of configurable pixel sensors. Each digital pixel sensor (DPS) cell is equipped with fast communication modules, self-biasing, offset cancellation, analog-to-digital converter (ADC) and fixed pattern noise (FPN) correction. In-pixel power consumption is minimized by the use of comprehensive MOSFET subthreshold operation. The main aim is to potentiate the integration of PbSe-based infra-red (IR)-image sensing technologies so as to widen its use, not only in distinct scenarios, but also at different stages of PbSe-CMOS integration maturity. For this purpose, we posit to investigate a comprehensive set of functional blocks distributed in two parallel approaches: • Frame-based “Smart” MWIR imaging based on new DPS circuit topologies with gain and offset FPN correction capabilities. This research line exploits the detector pitch to offer fully-digital programmability at pixel level and complete functionality with input parasitic capacitance compensation and internal frame memory. • Frame-free “Compact”-pitch MWIR vision based on a novel DPS lossless analog integrator and configurable temporal difference, combined with asynchronous communication protocols inside the focal plane. This strategy is conceived to allow extensive pitch compaction and readout speed increase by the suppression of in-pixel digital filtering, and the use of dynamic bandwidth allocation in each pixel of the FPA. In order make the electrical validation of first prototypes independent of the expensive PbSe deposition processes at wafer level, investigation is extended as well to the development of affordable sensor emulation strategies and integrated test platforms specifically oriented to image read-out integrated circuits. DPS cells, imagers and test chips have been fabricated and characterized in standard 0.15μm 1P6M, 0.35μm 2P4M and 2.5μm 2P1M CMOS technologies, all as part of research projects with industrial partnership. The research has led to the first high-speed uncooled frame-based IR quantum imager monolithically fabricated in a standard VLSI CMOS technology, and has given rise to the Tachyon series [1], a new line of commercial IR cameras used in real-time industrial, environmental and transportation control systems. The frame-free architectures investigated in this work represent a firm step forward to push further pixel pitch and system bandwidth up to the limits imposed by the evolving PbSe detector in future generations of the device.La present tesi doctoral descriu la recerca i el desenvolupament d'una nova tecnologia monolítica d'imatgeria infraroja de longitud d'ona mitja (MWIR), no refrigerada i de baix cost, per a usos industrials d'alta velocitat. El treball pren el relleu dels últims avenços assolits pel soci industrial NIT S.L. en el camp dels detectors MWIR de PbSe depositats en fase vapor (VPD), afegint-hi coneixement fonamental en la investigació de noves tècniques de disseny de circuits VLSI analògics i mixtes pel desenvolupament del dispositiu integrat de lectura unit al detector pixelat. Es parteix de la hipòtesi que, mitjançant l'ús de les esmentades tècniques de disseny, les tecnologies CMOS estàndard satisfan tots els requeriments operacionals del detector VPD PbSe respecte a connectivitat, fiabilitat, funcionalitat i escalabilitat per integrar de forma econòmica el dispositiu. La càmera PbSe-CMOS resultant ha de consumir molt baixa potència, operar a freqüències de kHz, exhibir bona uniformitat, i encabir els píxels actius CMOS de lectura en el pitch compacte del pla focal de la imatge, tot atenent a les particulars característiques del detector: altes relacions de corrent d'obscuritat a senyal, elevats valors de capacitat paràsita a l'entrada i dispersions importants en el procés de fabricació. Amb la finalitat de complir amb els requisits previs, es proposen arquitectures de sensors de visió de molt baix acoblament interpíxel basades en l'ús d'una matriu de pla focal (FPA) de píxels actius exclusivament digitals. Cada píxel sensor digital (DPS) està equipat amb mòduls de comunicació d'alta velocitat, autopolarització, cancel·lació de l'offset, conversió analògica-digital (ADC) i correcció del soroll de patró fixe (FPN). El consum en cada cel·la es minimitza fent un ús exhaustiu del MOSFET operant en subllindar. L'objectiu últim és potenciar la integració de les tecnologies de sensat d'imatge infraroja (IR) basades en PbSe per expandir-ne el seu ús, no només a diferents escenaris, sinó també en diferents estadis de maduresa de la integració PbSe-CMOS. En aquest sentit, es proposa investigar un conjunt complet de blocs funcionals distribuïts en dos enfocs paral·lels: - Dispositius d'imatgeria MWIR "Smart" basats en frames utilitzant noves topologies de circuit DPS amb correcció de l'FPN en guany i offset. Aquesta línia de recerca exprimeix el pitch del detector per oferir una programabilitat completament digital a nivell de píxel i plena funcionalitat amb compensació de la capacitat paràsita d'entrada i memòria interna de fotograma. - Dispositius de visió MWIR "Compact"-pitch "frame-free" en base a un novedós esquema d'integració analògica en el DPS i diferenciació temporal configurable, combinats amb protocols de comunicació asíncrons dins del pla focal. Aquesta estratègia es concep per permetre una alta compactació del pitch i un increment de la velocitat de lectura, mitjançant la supressió del filtrat digital intern i l'assignació dinàmica de l'ample de banda a cada píxel de l'FPA. Per tal d'independitzar la validació elèctrica dels primers prototips respecte a costosos processos de deposició del PbSe sensor a nivell d'oblia, la recerca s'amplia també al desenvolupament de noves estratègies d'emulació del detector d'IR i plataformes de test integrades especialment orientades a circuits integrats de lectura d'imatge. Cel·les DPS, dispositius d'imatge i xips de test s'han fabricat i caracteritzat, respectivament, en tecnologies CMOS estàndard 0.15 micres 1P6M, 0.35 micres 2P4M i 2.5 micres 2P1M, tots dins el marc de projectes de recerca amb socis industrials. Aquest treball ha conduït a la fabricació del primer dispositiu quàntic d'imatgeria IR d'alta velocitat, no refrigerat, basat en frames, i monolíticament fabricat en tecnologia VLSI CMOS estàndard, i ha donat lloc a Tachyon, una nova línia de càmeres IR comercials emprades en sistemes de control industrial, mediambiental i de transport en temps real.Postprint (published version

Low power CMOS vision sensor for foreground segmentation

This thesis focuses on the design of a top-ranked algorithm for background subtraction, the Pixel Adaptive Based Segmenter (PBAS), for its mapping onto a CMOS vision sensor on the focal plane processing. The redesign of PBAS into its hardware oriented version, HO-PBAS, has led to a less number of memories per pixel, along with a simpler overall model, yet, resulting in an acceptable loss of accuracy with respect to its counterpart on CPU. This thesis features two CMOS vision sensors. The first one, HOPBAS1K, has laid out a 24 x 56 pixel array onto a miniasic chip in standard 180 nm CMOS technology. The second one, HOPBAS10K, features an array of 98 x 98 pixels in standard 180 nm CMOS technology too. The second chip fixes some issues found in the first chip, and provides good hardware and background performance metrics

Bio-Inspired Optic Flow Sensors for Artificial Compound Eyes.

Compound eyes in flying insects have been studied to reveal the mysterious cues of vision-based flying mechanisms inside the smallest flying creatures in nature. Especially, researchers in the robotic area have made efforts to transfer the findings into their less than palm-sized unmanned air vehicles, micro-air-vehicles (MAVs). The miniaturized artificial compound eye is one of the key components in this system to provide visual information for navigation. Multi-directional sensing and motion estimation capabilities can give wide field-of-view (FoV) optic flows up to 360 solid angle. By deciphering the wide FoV optic flows, relevant information on the self-status of flight is parsed and utilized for flight command generation. In this work, we realize the wide-field optic flow sensing in a pseudo-hemispherical configuration realized by mounting a number of 2D array optic flow sensors on a flexible PCB module. The flexible PCBs can be bent to form a compound eye shape by origami packaging. From this scheme, the multiple 2D optic flow sensors can provide a modular, expandable configuration to meet low power constraints. The 2D optic flow sensors satisfy the low power constraint by employing a novel bio-inspired algorithm. We have modified the conventional elementary motion detector (EMD), which is known to be a basic operational unit in the insect’s visual pathways. We have implemented a bio-inspired time-stamp-based algorithm in mixed-mode circuits for robust operation. By optimal partitioning of analog to digital signal domains, we can realize the algorithm mostly in digital domain in a column-parallel circuits. Only the feature extraction algorithm is incorporated inside a pixel in analog circuits. In addition, the sensors integrate digital peripheral circuits to provide modular expandability. The on-chip data compressor can reduce the data rate by a factor of 8, so that it can connect a total of 25 optic flow sensors in a 4-wired Serial Peripheral Interface (SPI) bus. The packaged compound eye can transmit full-resolution optic flow data through the single 3MB/sec SPI bus. The fabricated 2D optic flow prototype sensor has achieved the power consumption of 243.3pJ/pixel and the maximum detectable optic flow of 1.96rad/sec at 120fps and 60 FoV.PhDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/108841/1/sssjpark_1.pd

Theory, Design and Implementation of Energy-Efficient Biotelemetry using Ultrasound Imaging

This dissertation investigates the fundamental limits of energy dissipation in establishing a communication link with implantable medical devices using ultrasound imaging-based biotelemetry. Ultrasound imaging technology has undergone a revolution during the last decade due to two primary innovations: advances in ultrasonic transducers that can operate over a broad range of frequencies and progresses in high-speed, high-resolution analog-to-digital converters and signal processors. Existing clinical and FDA approved bench-top ultrasound systems cangenerate real-time high-resolution images at frame rates as high as 10000 frames per second. On the other end of the spectrum, portable and hand-held ultrasound systems can generate high-speed real-time scans, widely used for diagnostic imaging in non-clinical environments. This dissertation’s fundamental hypothesis is to leverage the massive data acquisition and computational bandwidth afforded on these devices to establish energy-efficient bio-telemetry links with multiple in-vivo implanted devices. In the first part of the dissertation, I investigate using a commercial off-the-shelf (COTS) diagnostic ultrasound reader to achieve reliable in-vivo wireless telemetry with millimeter-sized piezoelectric crystal transducers. I propose multi-access biotelemetry methods in which several of these crystals simultaneously transmit the data using conventional modulation and coding schemes. I validated the feasibility of in-vivo operation using two piezoelectric crystals tethered to the tricuspid valve and the skin’s surface in a live ovine model. I demonstrated data rates close to 800 Kbps while consuming microwatts of power even in the presence of respiratory and cardiac motion artifacts. In the second part of the dissertation, I investigate the feasibility of energy harvesting from cardiac valvular perturbations to self-power the wireless implantable device. In this study, I explored using piezoelectric sutures implanted in proximity to the valvular regions compared to the previous studies involving piezoelectric patches or encasings attached to the cardiac or aortic surface to exploit nonlinearity in the valvular dynamics and self-power the implanted device. My study shows that power harvested from different annular planes of the tricuspid valve could range from nano-watts to milli-watts. In the final part of this dissertation, I investigate beamforming in B-scan ultrasound imaging to further reduce the biotelemetry energy-budget. In this context, I will study variance-based informatics in which the signal representation takes a form of signal variance instead of the signal mean for encoding and decoding. Using a modeling study, I show that compared to the mean-based logic representation, the variance-based representation can theoretically achieve a superior performance trade-off (in terms of energy dissipation) when operating at fundamental limits imposed by thermal-noise. I will then discuss how to extend variance-based representation to higher signal dimensions. I show that when applying variance-based encoding/decoding to B-scan biotelemetry, the power-dissipation requirements can be reducedto 100 pW even while interrogating from depths greater than 10 cm in a water medium

Electronic Systems with High Energy Efficiency for Embedded Computer Vision

Electronic systems are now widely adopted in everyday use. Moreover, nowadays there is an extensive use of embedded wearable and portable devices from industrial to consumer applications. The growing demand of embedded devices and applications has opened several new research fields due to the need of low power consumption and real time responsiveness. Focusing on this class of devices, computer vision algorithms are a challenging application target. In embedded computer vision hardware and software design have to interact to meet application specific requirements. The focus of this thesis is to study computer vision algorithms for embedded systems. The presented work starts presenting a novel algorithm for an IoT stationary use case targeting a high-end embedded device class, where power can be supplied to the platform through wires. Moreover, further contributions focus on algorithmic design and optimization on low and ultra-low power devices. Solutions are presented to gesture recognition and context change detection for wearable devices, focusing on first person wearable devices (Ego-Centric Vision), with the aim to exploit more constrained systems in terms of available power budget and computational resources. A novel gesture recognition algorithm is presented that improves state of art approaches. We then demonstrate the effectiveness of low resolution images exploitation in context change detection with real world ultra-low power imagers. The last part of the thesis deals with more flexible software models to support multiple applications linked at runtime and executed on Cortex-M device class, supporting critical isolation features typical of virtualization-ready CPUs on low-cost low-power microcontrollers and covering some defects in security and deployment capabilities of current firmwares

Persistent Phosphors for Smartphone-Based Luminescence Thermometry and Anti-Counterfeiting Applications

Leuchtstoffe anhaltender Lumineszenz im sichtbaren Spektrum eröffnen neue Möglichkeiten für Smartphone-basierte Anwendungen. Videoaufnahmen mit dem Smartphone mit 30 Bildern pro Sekunde können persistente Lumineszenzlebenszeiten einer Größenordnung von 100 ms und länger bestimmen. Die mit dem Smartphone aufgezeichneten Daten können benutzt werden um Anwendungen zu realisieren, die ansonsten nur für kurze Lebenszeiten möglich sind. Diese Alternative umgeht den Bedarf an teuren und relativ komplizierten Messinstrumenten, die für die Detektion von kurzen Lebenszeiten eingesetzt werden, wie zum Beispiel Multichannel scaling, Hochgeschwindigkeitskameras und Mikroskope zur Messung der Fluoreszenzlebenszeit. Diese Arbeit konzentriert sich auf die Detektion anhaltender Lumineszenz für Temperaturmessung und Anwendungen zur Fälschungssicherung mit dem Smartphone. Für die Smartphone-basierte Temperaturmessung wurde ein optimierter Gd2O2S: Eu3+ als Leuchtstoff verwendet, der mithilfe einer UV-Quelle angeregt werden kann. Der Leuchtstoff zeigte eine temperaturabhängige Lumineszenz, die hell und lange anhaltend genug war, um mit einer Smartphone-Kamera mit 30 Bildern pro Sekunde aufgezeichnet zu werden. Der Leuchtstoff hat eine Photolumineszenz-Quantenausbeute von 65 % und seine Lebenszeit nimmt mit steigender Temperatur ab. Dies wurde beobachtet über einen Temperaturbereich von 270 K bis 338 K, in dem die Lebenszeit von 1107 ms bis auf 100 ms abfiel. Die Analyse der zeitintegrierten Emission mit dem Smartphone nach einer Anregung mit 375 nm zeigte, dass die Temperaturen im Bereich von 270 K bis 338 K präzise gemessen werden konnten mit einer Messungenauigkeit unter 2 K. Darüber hinaus wurde die Lebenszeitmessung nicht durch Hintergrundstrahlung beeinträchtigt und ermöglichte somit eine genaue Temperaturmessung auch bei einer Hintergrundbeleuchtungsstärke von bis zu 1500 lx. Um eine Smartphone-basierte Fälschungssicherung zu realisieren wurden anhaltende Leuchtstoffe mit einstellbarer Lebenszeit bei Raumtemperatur benutzt, um dynamische, lumineszierende Etiketten zu entwickeln. Dynamische Fälschungssicherung wurde mithilfe von Ti4+-dotierten Gd2O2S: Eu3+ realisiert, wobei die Ti4+-Dotierung eine Kontrolle der Lebenszeit bei Raumtemperatur ermöglicht. Durch eine Veränderung der Kodotierung von 0 bis 0.09 mol% konnte die Lebenszeit von 1.17 ± 0.02 bis 5.95 ± 0.07 s durchgestimmt werden mit einer Anregung bei 375 nm. Durch eine Kombination von Leuchtstoffen mit verschiedenen Lebenszeiten konnten somit dynamische Etiketten zur Fälschungssicherung entwickelt werden. Die Lebenszeit der Leuchtstoffe für diese dynamischen Muster bestimmte dabei die Komplexität der Fälschungssicherung. Solche Muster, die aus einer Kombination von Leuchtstoffen mit großen Unterschieden in der Lumineszenzlebenszeit entwickelt wurden, konnten mit bloßem Auge beobachtet werden. Im Gegensatz dazu sind zeitliche Änderungen in Etiketten mit viel kürzerer Lebenszeit im Bereich von 0.2 s nur schwer mit bloßem Auge nachzuvollziehen. Mithilfe der Smartphone-Kamera mit einer Aufzeichnungsrate von 30 Bildern pro Sekunde können die versteckten Merkmale jedoch leicht entschlüsselt werden. In Hinblick auf die tatsächliche Anwendung am Verkaufsort, ist eine UV-Quelle einerseits normalerweise nicht vorhanden in einem Smartphone und andererseits stellt der Einsatz von UV-Strahlung für die Anregung der Leuchtstoffe eine Gesundheitsrisiko dar. Um die Nutzung einer UV-Quelle gänzlich zu vermeiden, wurden zweifarbige dynamische Etiketten zur Fälschungssicherung entwickelt. Diese erlauben eine Anregung mithilfe eines herkömmlichen Smartphone-Blitzlichtes während die Emission einfach mit der Kamera aufgezeichnet werden kann. Zu diesem Zweck wurden grün emittierende (SrAl2O4: Eu2+, Dy3+ (SAED)) und rot emittierende (CaS: Eu2+ und SrS: Eu2+) Leuchtstoffe entwickelt. Die Lebenszeit von SAED konnte variiert werden von 0.5 s bis 11.7 s durch Glühen des kommerziell erhältlichen Stoffes, was eine Verringerung der Störstellendichte im Material zur Folge hat. Die Lumineszenzlebenszeit von CaS: Eu2+ und SrS: Eu2+ konnte dagegen zwischen 0.1 bis 0.6 s und 150 bis 377 ms eingestellt werden mithilfe der Eu2+-Dotierdichte. Die Nutzung eines Smartphones ermöglicht nicht nur lebenszeit-basierte Temperaturmessungen ohne teure Messinstrumente, sondern eröffnet darüber hinaus eine kostengünstige Methode zur Authentifizierung von lumineszenzbasierten, dynamischen Markierungen zur Fälschungssicherung

Application-aware optimization of Artificial Intelligence for deployment on resource constrained devices

Artificial intelligence (AI) is changing people's everyday life. AI techniques such as Deep Neural Networks (DNN) rely on heavy computational models, which are in principle designed to be executed on powerful HW platforms, such as desktop or server environments. However, the increasing need to apply such solutions in people's everyday life has encouraged the research for methods to allow their deployment on embedded, portable and stand-alone devices, such as mobile phones, which exhibit relatively low memory and computational resources. Such methods targets both the development of lightweight AI algorithms and their acceleration through dedicated HW. This thesis focuses on the development of lightweight AI solutions, with attention to deep neural networks, to facilitate their deployment on resource constrained devices. Focusing on the computer vision field, we show how putting together the self learning ability of deep neural networks with application-specific knowledge, in the form of feature engineering, it is possible to dramatically reduce the total memory and computational burden, thus allowing the deployment on edge devices. The proposed approach aims to be complementary to already existing application-independent network compression solutions. In this work three main DNN optimization goals have been considered: increasing speed and accuracy, allowing training at the edge, and allowing execution on a microcontroller. For each of these we deployed the resulting algorithm to the target embedded device and measured its performance