Design and construction of a configurable full-field range imaging system for mobile robotic applications

Mobile robotic devices rely critically on extrospection sensors to determine the range to objects in the robot’s operating environment. This provides the robot with the ability both to navigate safely around obstacles and to map its environment and hence facilitate path planning and navigation. There is a requirement for a full-field range imaging system that can determine the range to any obstacle in a camera lens’ field of view accurately and in real-time. This paper details the development of a portable full-field ranging system whose bench-top version has demonstrated sub-millimetre precision. However, this precision required non-real-time acquisition rates and expensive hardware. By iterative replacement of components, a portable, modular and inexpensive version of this full-field ranger has been constructed, capable of real-time operation with some (user-defined) trade-off with precision

Linear CCD-Based Spectrometry Using Either an ASIC or FPGA Design Methodology

At room temperature, high-responsivity charge-coupled devices (CCD) comprising arrays of several thousand linear photodiodes are readily available. These sensors are capable of ultraviolet to near infrared wavelengths sensing with detecting resolutions of up to 24 dots per millimeter. Their applicability in novel spectrometry applications has been demonstrated. However, the complexity of their timing, image acquisition, and processing necessitates sophisticated peripheral circuitry for viable output. In this chapter, we outline the application specifications for a versatile spectrometer that is reliant on a field programmable gate array (FPGA) automation. The sustained throughput is 1.23 gigabit per second 8-bit color readout rate. This approach is attractive because the final FPGA design may be reconfigured readily to a single, branded, application-specific integrated circuit (ASIC) to drive a wider range of linear CCDs on the market. This is advantageous for rapid development and deployment of the spectrometer instrument

Bio-Inspired Multi-Spectral Imaging Sensors and Algorithms for Image Guided Surgery

Image guided surgery (IGS) utilizes emerging imaging technologies to provide additional structural and functional information to the physician in clinical settings. This additional visual information can help physicians delineate cancerous tissue during resection as well as avoid damage to near-by healthy tissue. Near-infrared (NIR) fluorescence imaging (700 nm to 900 nm wavelengths) is a promising imaging modality for IGS, namely for the following reasons: First, tissue absorption and scattering in the NIR window is very low, which allows for deeper imaging and localization of tumor tissue in the range of several millimeters to a centimeter depending on the tissue surrounding the tumor. Second, spontaneous tissue fluorescence emission is minimal in the NIR region, allowing for high signal-to-background ratio imaging compared to visible spectrum fluorescence imaging. Third, decoupling the fluorescence signal from the visible spectrum allows for optimization of NIR fluorescence while attaining high quality color images. Fourth, there are two FDA approved fluorescent dyes in the NIR region—namely methylene blue (MB) and indocyanine green—which can help to identify tumor tissue due to passive accumulation in human subjects. The aforementioned advantages have led to the development of NIR fluorescence imaging systems for a variety of clinical applications, such as sentinel lymph node imaging, angiography, and tumor margin assessment. With these technological advances, secondary surgeries due to positive tumor margins or damage to healthy organs can be largely mitigated, reducing the emotional and financial toll on the patient. Currently, several NIR fluorescence imaging systems (NFIS) are available commercially or are undergoing clinical trials, such as FLARE, SPY, PDE, Fluobeam, and others. These systems capture multi-spectral images using complex optical equipment and are combined with real-time image processing to present an augmented view to the surgeon. The information is presented on a standard monitor above the operating bed, which requires the physician to stop the surgical procedure and look up at the monitor. The break in the surgical flow sometimes outweighs the benefits of fluorescence based IGS, especially in time-critical surgical situations. Furthermore, these instruments tend to be very bulky and have a large foot print, which significantly complicates their adoption in an already crowded operating room. In this document, I present the development of a compact and wearable goggle system capable of real-time sensing of both NIR fluorescence and color information. The imaging system is inspired by the ommatidia of the monarch butterfly, in which pixelated spectral filters are integrated with light sensitive elements. The pixelated spectral filters are fabricated via a carefully optimized nanofabrication procedure and integrated with a CMOS imaging array. The entire imaging system has been optimized for high signal-to-background fluorescence imaging using an analytical approach, and the efficacy of the system has been experimentally verified. The bio-inspired spectral imaging sensor is integrated with an FPGA for compact and real-time signal processing and a wearable goggle for easy integration in the operating room. The complete imaging system is undergoing clinical trials at Washington University in the St. Louis Medical School for imaging sentinel lymph nodes in both breast cancer patients and melanoma patients

An embedded adaptive optics real time controller

The design and realisation of a low cost, high speed control system for adaptive optics (AO) is presented. This control system is built around a field programmable gate array (FPGA). FPGA devices represent a fundamentally different approach to implementing control systems than conventional central processing units. The performance of the FPGA control system is demonstrated in a specifically constructed laboratory AO experiment where closed loop AO correction is shown. An alternative application of the control system is demonstrated in the field of optical tweezing, where it is used to study the motion dynamics of particles trapped within laser foci

A high speed Tri-Vision system for automotive applications

Purpose: Cameras are excellent ways of non-invasively monitoring the interior and exterior of vehicles. In particular, high speed stereovision and multivision systems are important for transport applications such as driver eye tracking or collision avoidance. This paper addresses the synchronisation problem which arises when multivision camera systems are used to capture the high speed motion common in such applications. Methods: An experimental, high-speed tri-vision camera system intended for real-time driver eye-blink and saccade measurement was designed, developed, implemented and tested using prototype, ultra-high dynamic range, automotive-grade image sensors specifically developed by E2V (formerly Atmel) Grenoble SA as part of the European FP6 project – sensation (advanced sensor development for attention stress, vigilance and sleep/wakefulness monitoring). Results : The developed system can sustain frame rates of 59.8 Hz at the full stereovision resolution of 1280 × 480 but this can reach 750 Hz when a 10 k pixel Region of Interest (ROI) is used, with a maximum global shutter speed of 1/48000 s and a shutter efficiency of 99.7%. The data can be reliably transmitted uncompressed over standard copper Camera-Link® cables over 5 metres. The synchronisation error between the left and right stereo images is less than 100 ps and this has been verified both electrically and optically. Synchronisation is automatically established at boot-up and maintained during resolution changes. A third camera in the set can be configured independently. The dynamic range of the 10bit sensors exceeds 123 dB with a spectral sensitivity extending well into the infra-red range. Conclusion: The system was subjected to a comprehensive testing protocol, which confirms that the salient requirements for the driver monitoring application are adequately met and in some respects, exceeded. The synchronisation technique presented may also benefit several other automotive stereovision applications including near and far-field obstacle detection and collision avoidance, road condition monitoring and others.Partially funded by the EU FP6 through the IST-507231 SENSATION project.peer-reviewe

AIDI: An adaptive image denoising FPGA-based IP-core for real-time applications

The presence of noise in images can significantly impact the performances of digital image processing and computer vision algorithms. Thus, it should be removed to improve the robustness of the entire processing flow. The noise estimation in an image is also a key factor, since, to be more effective, algorithms and denoising filters should be tuned to the actual level of noise. Moreover, the complexity of these algorithms brings a new challenge in real-time image processing applications, requiring high computing capacity. In this context, hardware acceleration is crucial, and Field Programmable Gate Arrays (FPGAs) best fit the growing demand of computational capabilities. This paper presents an Adaptive Image Denoising IP-core (AIDI) for real-time applications. The core first estimates the level of noise in the input image, then applies an adaptive Gaussian smoothing filter to remove the estimated noise. The filtering parameters are computed on-the-fly, adapting them to the level of noise in the image, and pixel by pixel, to preserve image information (e.g., edges or corners). The FPGA-based architecture is presented, highlighting its improvements w.r.t. a standard static filtering approac

Smart cmos image sensor for 3d measurement

3D measurements are concerned with extracting visual information from the geometry of visible surfaces and interpreting the 3D coordinate data thus obtained, to detect or track the position or reconstruct the profile of an object, often in real time. These systems necessitate image sensors with high accuracy of position estimation and high frame rate of data processing for handling large volumes of data. A standard imager cannot address the requirements of fast image acquisition and processing, which are the two figures of merit for 3D measurements. Hence, dedicated VLSI imager architectures are indispensable for designing these high performance sensors. CMOS imaging technology provides potential to integrate image processing algorithms on the focal plane of the device, resulting in smart image sensors, capable of achieving better processing features in handling massive image data. The objective of this thesis is to present a new architecture of smart CMOS image sensor for real time 3D measurement using the sheet-beam projection methods based on active triangulation. Proposing the vision sensor as an ensemble of linear sensor arrays, all working in parallel and processing the entire image in slices, the complexity of the image-processing task shifts from O (N 2 ) to O (N). Inherent also in the design is the high level of parallelism to achieve massive parallel processing at high frame rate, required in 3D computation problems. This work demonstrates a prototype of the smart linear sensor incorporating full testability features to test and debug both at device and system levels. The salient features of this work are the asynchronous position to pulse stream conversion, multiple images binarization, high parallelism and modular architecture resulting in frame rate and sub-pixel resolution suitable for real time 3D measurements

Wave Front Sensing and Correction Using Spatial Modulation and Digitally Enhanced Heterodyne Interferometry

This thesis is about light. Specifically it explores a new way sensing the spatial distribution of amplitude and phase across the wavefront of a propagating laser. It uses spatial light modulators to tag spatially distinct regions of the beam, a single diode to collect the resulting light and digitally enhanced heterodyne interferometry to decode the phase and amplitude information across the wavefront. It also demonstrates how using these methods can be used to maximise the transmission of light through a cavity and shows how minor aberrations in the beam can be corrected in real time. Finally it demonstrate the preferential transmission of higher order modes. Wavefront sensing is becoming increasingly important as the demands on modern interferometers increase. Land based systems such as the Laser Interferometer Gravitational-Wave Observatory (LIGO) use it to maximise the amount of power in the arm cavities during operation and reduce noise, while space based missions such as the Laser Interferometer Space Antenna (LISA) will use it to align distant partner satellites and ensure that the maximum amount of signal is exchanged. Conventionally wavefront sensing is accomplished using either Hartmann Sensors or multi-element diodes. These are well proven and very effective techniques but bring with them a number of well understood limitations. Critically, while they can map a wavefront in detail, they are strictly sensors and can do nothing to correct it. Our new technique is based on a single-element photo-diode and the spatial modulation of the local oscillator beam. We encode orthogonal codes spatially onto this light and use these to separate the phases and amplitudes of different parts of the signal beam in post processing. This technique shifts complexity from the optical hardware into deterministic digital signal processing. Notably, the use of a single analogue channel (photo-diode, connections and analogue to digital converter) avoids some low-frequency error sources. The technique can also sense the wavefront phase at many points, limited only by the number of actuators on the spatial light modulator in contrast to the standard 4 points from a quadrant photo-diode. For ground-based systems, our technique could be used to identify and eliminate higher-order modes, while, for space-based systems, it provides a measure of wavefront tilt which is less susceptible to low frequency noise. In the future it may be possible to couple the technique with an artificial intelligence engine to automate more of the beam alignment process in arrangements involving multiple cavities, preferentially select (or reject) specific higher order modes and start to reduce the burgeoning requirements for human control of these complex instruments