430 research outputs found
Event-based Vision: A Survey
Event cameras are bio-inspired sensors that differ from conventional frame
cameras: Instead of capturing images at a fixed rate, they asynchronously
measure per-pixel brightness changes, and output a stream of events that encode
the time, location and sign of the brightness changes. Event cameras offer
attractive properties compared to traditional cameras: high temporal resolution
(in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low
power consumption, and high pixel bandwidth (on the order of kHz) resulting in
reduced motion blur. Hence, event cameras have a large potential for robotics
and computer vision in challenging scenarios for traditional cameras, such as
low-latency, high speed, and high dynamic range. However, novel methods are
required to process the unconventional output of these sensors in order to
unlock their potential. This paper provides a comprehensive overview of the
emerging field of event-based vision, with a focus on the applications and the
algorithms developed to unlock the outstanding properties of event cameras. We
present event cameras from their working principle, the actual sensors that are
available and the tasks that they have been used for, from low-level vision
(feature detection and tracking, optic flow, etc.) to high-level vision
(reconstruction, segmentation, recognition). We also discuss the techniques
developed to process events, including learning-based techniques, as well as
specialized processors for these novel sensors, such as spiking neural
networks. Additionally, we highlight the challenges that remain to be tackled
and the opportunities that lie ahead in the search for a more efficient,
bio-inspired way for machines to perceive and interact with the world
DART: Distribution Aware Retinal Transform for Event-based Cameras
We introduce a generic visual descriptor, termed as distribution aware
retinal transform (DART), that encodes the structural context using log-polar
grids for event cameras. The DART descriptor is applied to four different
problems, namely object classification, tracking, detection and feature
matching: (1) The DART features are directly employed as local descriptors in a
bag-of-features classification framework and testing is carried out on four
standard event-based object datasets (N-MNIST, MNIST-DVS, CIFAR10-DVS,
NCaltech-101). (2) Extending the classification system, tracking is
demonstrated using two key novelties: (i) For overcoming the low-sample problem
for the one-shot learning of a binary classifier, statistical bootstrapping is
leveraged with online learning; (ii) To achieve tracker robustness, the scale
and rotation equivariance property of the DART descriptors is exploited for the
one-shot learning. (3) To solve the long-term object tracking problem, an
object detector is designed using the principle of cluster majority voting. The
detection scheme is then combined with the tracker to result in a high
intersection-over-union score with augmented ground truth annotations on the
publicly available event camera dataset. (4) Finally, the event context encoded
by DART greatly simplifies the feature correspondence problem, especially for
spatio-temporal slices far apart in time, which has not been explicitly tackled
in the event-based vision domain.Comment: 12 pages, revision submitted to TPAMI in Nov 201
An Event-Based Neurobiological Recognition System with Orientation Detector for Objects in Multiple Orientations
A new multiple orientation event-based neurobiological recognition system is proposed by integrating recognition and tracking function in this paper, which is used for asynchronous address-event representation (AER) image sensors. The characteristic of this system has been enriched to recognize the objects in multiple orientations with only training samples moving in a single orientation. The system extracts multi-scale and multi-orientation line features inspired by models of the primate visual cortex. An orientation detector based on modified Gaussian blob tracking algorithm is introduced for object tracking and orientation detection. The orientation detector and feature extraction block work in simultaneous mode, without any increase in categorization time. An addresses lookup table (addresses LUT) is also presented to adjust the feature maps by addresses mapping and reordering, and they are categorized in the trained spiking neural network. This recognition system is evaluated with the MNIST dataset which have played important roles in the development of computer vision, and the accuracy is increase owing to the use of both ON and OFF events. AER data acquired by a DVS are also tested on the system, such as moving digits, pokers, and vehicles. The experimental results show that the proposed system can realize event-based multi-orientation recognition.The work presented in this paper makes a number of contributions to the event-based vision processing system for multi-orientation object recognition. It develops a new tracking-recognition architecture to feedforward categorization system and an address reorder approach to classify multi-orientation objects using event-based data. It provides a new way to recognize multiple orientation objects with only samples in single orientation
Real-time Tracking Based on Neuromrophic Vision
Real-time tracking is an important problem in computer vision in which most
methods are based on the conventional cameras. Neuromorphic vision is a concept
defined by incorporating neuromorphic vision sensors such as silicon retinas in
vision processing system. With the development of the silicon technology,
asynchronous event-based silicon retinas that mimic neuro-biological
architectures has been developed in recent years. In this work, we combine the
vision tracking algorithm of computer vision with the information encoding
mechanism of event-based sensors which is inspired from the neural rate coding
mechanism. The real-time tracking of single object with the advantage of high
speed of 100 time bins per second is successfully realized. Our method
demonstrates that the computer vision methods could be used for the
neuromorphic vision processing and we can realize fast real-time tracking using
neuromorphic vision sensors compare to the conventional camera
AER Building Blocks for Multi-Layer Multi-Chip Neuromorphic Vision Systems
A 5-layer neuromorphic vision processor whose components
communicate spike events asychronously using the address-eventrepresentation
(AER) is demonstrated. The system includes a retina
chip, two convolution chips, a 2D winner-take-all chip, a delay line
chip, a learning classifier chip, and a set of PCBs for computer
interfacing and address space remappings. The components use a
mixture of analog and digital computation and will learn to classify
trajectories of a moving object. A complete experimental setup and
measurements results are shown.Unión Europea IST-2001-34124 (CAVIAR)Ministerio de Ciencia y Tecnología TIC-2003-08164-C0
End-to-End Learning of Representations for Asynchronous Event-Based Data
Event cameras are vision sensors that record asynchronous streams of
per-pixel brightness changes, referred to as "events". They have appealing
advantages over frame-based cameras for computer vision, including high
temporal resolution, high dynamic range, and no motion blur. Due to the sparse,
non-uniform spatiotemporal layout of the event signal, pattern recognition
algorithms typically aggregate events into a grid-based representation and
subsequently process it by a standard vision pipeline, e.g., Convolutional
Neural Network (CNN). In this work, we introduce a general framework to convert
event streams into grid-based representations through a sequence of
differentiable operations. Our framework comes with two main advantages: (i)
allows learning the input event representation together with the task dedicated
network in an end to end manner, and (ii) lays out a taxonomy that unifies the
majority of extant event representations in the literature and identifies novel
ones. Empirically, we show that our approach to learning the event
representation end-to-end yields an improvement of approximately 12% on optical
flow estimation and object recognition over state-of-the-art methods.Comment: To appear at ICCV 201
Neutron-Induced, Single-Event Effects on Neuromorphic Event-based Vision Sensor: A First Step Towards Space Applications
This paper studies the suitability of neuromorphic event-based vision cameras
for spaceflight, and the effects of neutron radiation on their performance.
Neuromorphic event-based vision cameras are novel sensors that implement
asynchronous, clockless data acquisition, providing information about the
change in illuminance greater than 120dB with sub-millisecond temporal
precision. These sensors have huge potential for space applications as they
provide an extremely sparse representation of visual dynamics while removing
redundant information, thereby conforming to low-resource requirements. An
event-based sensor was irradiated under wide-spectrum neutrons at Los Alamos
Neutron Science Center and its effects were classified. We found that the
sensor had very fast recovery during radiation, showing high correlation of
noise event bursts with respect to source macro-pulses. No significant
differences were observed between the number of events induced at different
angles of incidence but significant differences were found in the spatial
structure of noise events at different angles. The results show that
event-based cameras are capable of functioning in a space-like, radiative
environment with a signal-to-noise ratio of 3.355. They also show that
radiation-induced noise does not affect event-level computation. We also
introduce the Event-based Radiation-Induced Noise Simulation Environment
(Event-RINSE), a simulation environment based on the noise-modelling we
conducted and capable of injecting the effects of radiation-induced noise from
the collected data to any stream of events in order to ensure that developed
code can operate in a radiative environment. To the best of our knowledge, this
is the first time such analysis of neutron-induced noise analysis has been
performed on a neuromorphic vision sensor, and this study shows the advantage
of using such sensors for space applications
- …