Neuromorphic electronics for real-time biomedical image processing

Published versio

Neuromorphic perception for greenhouse technology using event-based sensors

Event-Based Cameras (EBCs), unlike conventional cameras, feature independent pixels that asynchronously generate outputs upon detecting changes in their field of view. Short calculations are performed on each event to mimic the brain. The output is a sparse sequence of events with high temporal precision. Conventional computer vision algorithms do not leverage these properties. Thus a new paradigm has been devised. While event cameras are very efficient in representing sparse sequences of events with high temporal precision, many approaches are challenged in applications where a large amount of spatially-temporally rich information must be processed in real-time. In reality, most tasks in everyday life take place in complex and uncontrollable environments, which require sophisticated models and intelligent reasoning. Typical hard problems in real-world scenes are detecting various non-uniform objects or navigation in an unknown and complex environment. In addition, colour perception is an essential fundamental property in distinguishing objects in natural scenes. Colour is a new aspect of event-based sensors, which work fundamentally differently from standard cameras, measuring per-pixel brightness changes per colour filter asynchronously rather than measuring “absolute” brightness at a constant rate. This thesis explores neuromorphic event-based processing methods for high-noise and cluttered environments with imbalanced classes. A fully event-driven processing pipeline was developed for agricultural applications to perform fruits detection and classification to unlock the outstanding properties of event cameras. The nature of features in such data was explored, and methods to represent and detect features were demonstrated. A framework for detecting and classifying features was developed and evaluated on the N-MNIST and Dynamic Vision Sensor (DVS) gesture datasets. The same network was evaluated on laboratory recorded and real-world data with various internal variations for fruits detection such as overlap, variation in size and appearance. In addition, a method to handle highly imbalanced data was developed. We examined the characteristics of spatio-temporal patterns for each colour filter to help expand our understanding of this novel data and explored their applications in classification tasks where colours were more relevant features than shapes and appearances. The results presented in this thesis demonstrate the potential and efficacy of event- based systems by demonstrating the applicability of colour event data and the viability of event-driven classification

Estimating general motion and intensity from event cameras

Robotic vision algorithms have become widely used in many consumer products which enabled technologies such as autonomous vehicles, drones, augmented reality (AR) and virtual reality (VR) devices to name a few. These applications require vision algorithms to work in real-world environments with extreme lighting variations and fast moving objects. However, robotic vision applications rely often on standard video cameras which face severe limitations in fast-moving scenes or by bright light sources which diminish the image quality with artefacts like motion blur or over-saturation. To address these limitations, the body of work presented here investigates the use of alternative sensor devices which mimic the superior perception properties of human vision. Such silicon retinas were proposed by neuromorphic engineering, and we focus here on one such biologically inspired sensor called the event camera which offers a new camera paradigm for real-time robotic vision. The camera provides a high measurement rate, low latency, high dynamic range, and low data rate. The signal of the camera is composed of a stream of asynchronous events at microsecond resolution. Each event indicates when individual pixels registers a logarithmic intensity changes of a pre-set threshold size. Using this novel signal has proven to be very challenging in most computer vision problems since common vision methods require synchronous absolute intensity information. In this thesis, we present for the first time a method to reconstruct an image and es- timation motion from an event stream without additional sensing or prior knowledge of the scene. This method is based on coupled estimations of both motion and intensity which enables our event-based analysis, which was previously only possible with severe limitations. We also present the first machine learning algorithm for event-based unsu- pervised intensity reconstruction which does not depend on an explicit motion estimation and reveals finer image details. This learning approach does not rely on event-to-image examples, but learns from standard camera image examples which are not coupled to the event data. In experiments we show that the learned reconstruction improves upon our handcrafted approach. Finally, we combine our learned approach with motion estima- tion methods and show the improved intensity reconstruction also significantly improves the motion estimation results. We hope our work in this thesis bridges the gap between the event signal and images and that it opens event cameras to practical solutions to overcome the current limitations of frame-based cameras in robotic vision.Open Acces

Evaluation of tone-mapping algorithms for focal-plane implementation

Scenes in the real world may simultaneously contain very bright and very dark regions, caused by different illumination conditions. These scenes contain a wide range of different light intensity values. Attempting to exhibit a picture of such scene on a conventional display device, such as a computer monitor, leads to (a possibly large) loss of details in the displayed scene, since conventional display devices can only represent a limited amount of different light intensity values, which span a smaller range. To mitigate the loss of details, before it is shown on the display device, the picture of the scene must be processed by a tone-mapping algorithm, which maps the original light intensities into the light intensities representable by the display, thereby accommodating the input high dynamic range of values into a smaller range. In this work, a comparison between different tone-mapping algorithms is presented. More specifically, the performances (regarding processing time and overall quality of the processed image) from a digital version of the tone-mapping operator originally proposed by Fern´andez-Berni et al. [11] that is implemented in the focal plane of the camera and from different tone-mapping operators that are originally implemented in software are compared. Furthermore, a second digital version of the focal-plane operator, which simulates a modified version of the original hardware implementation, is considered and its performance is analyzed. The modified hardware implementation is less complex and requires less space than the original implementation and, subjectively, keeps the overall image quality approximately equal to that achieved by digital operators. Issues regarding colors of the tone-mapped images are also addressed, especially the required processing that must be performed by the focal-plane operator after the tone mapping, in order to yield images without color distortions.Cenas no mundo real podem conter uma ampla faixa de valores de diferentes intensidades luminosas. Mostrar a cena original em um aparelho de exibição convencional, tal como um monitor de computador, leva a uma (possivelmente grande) perda de detalhes na cena exibida, uma vez que esses aparelhos são capazes de representar somente uma quantidade limitada de diferentes intensidades luminosas, as quais ocupam uma faixa de valores menor. Para diminuir a perda de detalhes, antes de ser exibida em tais aparelhos, a cena deve ser processada por um algoritmo de tone mapping, o qual mapeia os valores originais de intensidade luminosa em valores que são representáveis pelo aparelho de exibição, acomodando, com isso, a alta faixa dinâmica dos valores de entrada em uma faixa de valores menor. Neste trabalho, uma comparação entre diferentes algoritmos de tone-mapping é apresentada. Mais especificamente, são comparados entre si os desempenhos (referentes a tempos de execução e qualidade geral da imagem processada) da versão digital do operador de tone mapping originalmente proposto por Fernández-Berni et al. [11] que ´e implementado no plano focal da câmera e de diferentes operadores de tone mapping que são originalmente implementados em software. Além disso, uma segunda versão digital do operador no plano focal, a qual simula uma versão modificada da implementação original em hardware, é considerada e seu desempenho é analisado. Essa versão modificada requer um hardware que é menos complexo e ocupa menos espaço que o hardware da implementação original, além de, subjetivamente, manter a qualidade geral da imagem próxima daquela alcançada por operadores digitais. Questões referentes às cores das imagens processadas também são tratadas, especialmente os processamentos que são requeridos pelo operador do plano focal após o tone mapping, de modo a gerar imagens sem distorções de cor

Towards High-Frequency Tracking and Fast Edge-Aware Optimization

This dissertation advances the state of the art for AR/VR tracking systems by increasing the tracking frequency by orders of magnitude and proposes an efficient algorithm for the problem of edge-aware optimization. AR/VR is a natural way of interacting with computers, where the physical and digital worlds coexist. We are on the cusp of a radical change in how humans perform and interact with computing. Humans are sensitive to small misalignments between the real and the virtual world, and tracking at kilo-Hertz frequencies becomes essential. Current vision-based systems fall short, as their tracking frequency is implicitly limited by the frame-rate of the camera. This thesis presents a prototype system which can track at orders of magnitude higher than the state-of-the-art methods using multiple commodity cameras. The proposed system exploits characteristics of the camera traditionally considered as flaws, namely rolling shutter and radial distortion. The experimental evaluation shows the effectiveness of the method for various degrees of motion. Furthermore, edge-aware optimization is an indispensable tool in the computer vision arsenal for accurate filtering of depth-data and image-based rendering, which is increasingly being used for content creation and geometry processing for AR/VR. As applications increasingly demand higher resolution and speed, there exists a need to develop methods that scale accordingly. This dissertation proposes such an edge-aware optimization framework which is efficient, accurate, and algorithmically scales well, all of which are much desirable traits not found jointly in the state of the art. The experiments show the effectiveness of the framework in a multitude of computer vision tasks such as computational photography and stereo.Comment: PhD thesi

Characterization and visualization of reflective properties of surfaces

Images play a vital role in several fields of natural science research, including biology, physics, astrophysics, and computer science. In the natural sciences, images are commonly used in measurements or documentation; such applications include images made with telescopes, optical microscopes, or electron microscopes. In the humanities, images also play an important role in research. In art history, for example, many different types of images, from photos of small objects to three-dimensional reconstructions of buildings, help art historians to develop theories, to discuss them with other scholars, and to document the current state of artworks, e.g. in the process of restoration. This is particularly useful if the object is not easily accessible, in which case a common solution is to work with photographs. Digital photography has simplified the process of visual representation, because digital images can be easily shared and made accessible. However, when it comes to more complex kinds of artworks like mosaics, these static and two-dimensional images are not able to reproduce the actual visual impression of the object. Similar considerations apply to a variety of other artifacts, such as early prints, books, parchments, and textiles. The challenge in the digitization of of these objects lies in their complex surface properties and reflection behavior. A promising way to solve those limitations is the use of Reflectance Transformation Imaging. RTI is a set of computational photographic methods that capture a subject’s surface shape and color, making it possible to interactively re-light the subject from any direction by means of a mathematical model. The major drawback of RTI is the limitation of the applied mathematical model. Other drawbacks are the RTI imaging workflow and the fact that display of RTI requires a particular stand-alone application. In this thesis, the author developed a data-driven scientific approach to reproduce surfaces composed of lambertian and glossy materials using the RTI technique with as few parameters as possible. This new approach has been called eRTI (enhanced Reflection Transformation Imaging). Furthermore the hardware needed to acquire RTI and eRTI has been improved, by collaborating with a local Swiss firm to develop a novel solution for image acquisition. Lastly a web-based viewer has been developed, to render eRTI images in any standard web browser, even on most mobile devices. The qualities of eRTI have been tested using a novel approach that includes a quantitative and a qualitative method. The results show agreement between the techniques

Aerospace Medicine and Biology: A cumulative index to a continuing bibliography

This publication is a cumulative index to the abstracts contained in Supplements 138 through 149 of AEROSPACE MEDICINE AND BIOLOGY: A CONTINUING BIBLIOGRAPHY. It includes three indexes -- subject, personal author, and corporate source