Towards a bio-inspired mixed-signal retinal processor

Published versio

Event-based Vision: A Survey

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world

A micropower centroiding vision processor

Published versio

Compressive Sensing Based Bio-Inspired Shape Feature Detection CMOS Imager

A CMOS imager integrated circuit using compressive sensing and bio-inspired detection is presented which integrates novel functions and algorithms within a novel hardware architecture enabling efficient on-chip implementation

Neuromorphic electronics for real-time biomedical image processing

Published versio

Bio-Inspired Multi-Spectral Imaging Sensors and Algorithms for Image Guided Surgery

Image guided surgery (IGS) utilizes emerging imaging technologies to provide additional structural and functional information to the physician in clinical settings. This additional visual information can help physicians delineate cancerous tissue during resection as well as avoid damage to near-by healthy tissue. Near-infrared (NIR) fluorescence imaging (700 nm to 900 nm wavelengths) is a promising imaging modality for IGS, namely for the following reasons: First, tissue absorption and scattering in the NIR window is very low, which allows for deeper imaging and localization of tumor tissue in the range of several millimeters to a centimeter depending on the tissue surrounding the tumor. Second, spontaneous tissue fluorescence emission is minimal in the NIR region, allowing for high signal-to-background ratio imaging compared to visible spectrum fluorescence imaging. Third, decoupling the fluorescence signal from the visible spectrum allows for optimization of NIR fluorescence while attaining high quality color images. Fourth, there are two FDA approved fluorescent dyes in the NIR region—namely methylene blue (MB) and indocyanine green—which can help to identify tumor tissue due to passive accumulation in human subjects. The aforementioned advantages have led to the development of NIR fluorescence imaging systems for a variety of clinical applications, such as sentinel lymph node imaging, angiography, and tumor margin assessment. With these technological advances, secondary surgeries due to positive tumor margins or damage to healthy organs can be largely mitigated, reducing the emotional and financial toll on the patient. Currently, several NIR fluorescence imaging systems (NFIS) are available commercially or are undergoing clinical trials, such as FLARE, SPY, PDE, Fluobeam, and others. These systems capture multi-spectral images using complex optical equipment and are combined with real-time image processing to present an augmented view to the surgeon. The information is presented on a standard monitor above the operating bed, which requires the physician to stop the surgical procedure and look up at the monitor. The break in the surgical flow sometimes outweighs the benefits of fluorescence based IGS, especially in time-critical surgical situations. Furthermore, these instruments tend to be very bulky and have a large foot print, which significantly complicates their adoption in an already crowded operating room. In this document, I present the development of a compact and wearable goggle system capable of real-time sensing of both NIR fluorescence and color information. The imaging system is inspired by the ommatidia of the monarch butterfly, in which pixelated spectral filters are integrated with light sensitive elements. The pixelated spectral filters are fabricated via a carefully optimized nanofabrication procedure and integrated with a CMOS imaging array. The entire imaging system has been optimized for high signal-to-background fluorescence imaging using an analytical approach, and the efficacy of the system has been experimentally verified. The bio-inspired spectral imaging sensor is integrated with an FPGA for compact and real-time signal processing and a wearable goggle for easy integration in the operating room. The complete imaging system is undergoing clinical trials at Washington University in the St. Louis Medical School for imaging sentinel lymph nodes in both breast cancer patients and melanoma patients

Bio-inspired electronics for micropower vision processing

Vision processing is a topic traditionally associated with neurobiology; known to encode, process and interpret visual data most effectively. For example, the human retina; an exquisite sheet of neurobiological wetware, is amongst the most powerful and efficient vision processors known to mankind. With improving integrated technologies, this has generated considerable research interest in the microelectronics community in a quest to develop effective, efficient and robust vision processing hardware with real-time capability. This thesis describes the design of a novel biologically-inspired hybrid analogue/digital vision chip ORASIS1 for centroiding, sizing and counting of enclosed objects. This chip is the first two-dimensional silicon retina capable of centroiding and sizing multiple objects2 in true parallel fashion. Based on a novel distributed architecture, this system achieves ultra-fast and ultra-low power operation in comparison to conventional techniques. Although specifically applied to centroid detection, the generalised architecture in fact presents a new biologically-inspired processing paradigm entitled: distributed asynchronous mixed-signal logic processing. This is applicable to vision and sensory processing applications in general that require processing of large numbers of parallel inputs, normally presenting a computational bottleneck. Apart from the distributed architecture, the specific centroiding algorithm and vision chip other original contributions include: an ultra-low power tunable edge-detection circuit, an adjustable threshold local/global smoothing network and an ON/OFF-adaptive spiking photoreceptor circuit. Finally, a concise yet comprehensive overview of photodiode design methodology is provided for standard CMOS technologies. This aims to form a basic reference from an engineering perspective, bridging together theory with measured results. Furthermore, an approximate photodiode expression is presented, aiming to provide vision chip designers with a basic tool for pre-fabrication calculations