17 research outputs found
Independent Motion Detection with Event-driven Cameras
Unlike standard cameras that send intensity images at a constant frame rate,
event-driven cameras asynchronously report pixel-level brightness changes,
offering low latency and high temporal resolution (both in the order of
micro-seconds). As such, they have great potential for fast and low power
vision algorithms for robots. Visual tracking, for example, is easily achieved
even for very fast stimuli, as only moving objects cause brightness changes.
However, cameras mounted on a moving robot are typically non-stationary and the
same tracking problem becomes confounded by background clutter events due to
the robot ego-motion. In this paper, we propose a method for segmenting the
motion of an independently moving object for event-driven cameras. Our method
detects and tracks corners in the event stream and learns the statistics of
their motion as a function of the robot's joint velocities when no
independently moving objects are present. During robot operation, independently
moving objects are identified by discrepancies between the predicted corner
velocities from ego-motion and the measured corner velocities. We validate the
algorithm on data collected from the neuromorphic iCub robot. We achieve a
precision of ~ 90 % and show that the method is robust to changes in speed of
both the head and the target.Comment: 7 pages, 6 figure
Asynchronous, Photometric Feature Tracking using Events and Frames
We present a method that leverages the complementarity of event cameras and
standard cameras to track visual features with low-latency. Event cameras are
novel sensors that output pixel-level brightness changes, called "events". They
offer significant advantages over standard cameras, namely a very high dynamic
range, no motion blur, and a latency in the order of microseconds. However,
because the same scene pattern can produce different events depending on the
motion direction, establishing event correspondences across time is
challenging. By contrast, standard cameras provide intensity measurements
(frames) that do not depend on motion direction. Our method extracts features
on frames and subsequently tracks them asynchronously using events, thereby
exploiting the best of both types of data: the frames provide a photometric
representation that does not depend on motion direction and the events provide
low-latency updates. In contrast to previous works, which are based on
heuristics, this is the first principled method that uses raw intensity
measurements directly, based on a generative event model within a
maximum-likelihood framework. As a result, our method produces feature tracks
that are both more accurate (subpixel accuracy) and longer than the state of
the art, across a wide variety of scenes.Comment: 22 pages, 15 figures, Video: https://youtu.be/A7UfeUnG6c
Speed Invariant Time Surface for Learning to Detect Corner Points with Event-Based Cameras
We propose a learning approach to corner detection for event-based cameras
that is stable even under fast and abrupt motions. Event-based cameras offer
high temporal resolution, power efficiency, and high dynamic range. However,
the properties of event-based data are very different compared to standard
intensity images, and simple extensions of corner detection methods designed
for these images do not perform well on event-based data. We first introduce an
efficient way to compute a time surface that is invariant to the speed of the
objects. We then show that we can train a Random Forest to recognize events
generated by a moving corner from our time surface. Random Forests are also
extremely efficient, and therefore a good choice to deal with the high capture
frequency of event-based cameras ---our implementation processes up to 1.6Mev/s
on a single CPU. Thanks to our time surface formulation and this learning
approach, our method is significantly more robust to abrupt changes of
direction of the corners compared to previous ones. Our method also naturally
assigns a confidence score for the corners, which can be useful for
postprocessing. Moreover, we introduce a high-resolution dataset suitable for
quantitative evaluation and comparison of corner detection methods for
event-based cameras. We call our approach SILC, for Speed Invariant Learned
Corners, and compare it to the state-of-the-art with extensive experiments,
showing better performance.Comment: 8 pages, 7 figures, accepted at CVPR 201
Event Blob Tracking: An Asynchronous Real-Time Algorithm
Event-based cameras have become increasingly popular for tracking fast-moving
objects due to their high temporal resolution, low latency, and high dynamic
range. In this paper, we propose a novel algorithm for tracking event blobs
using raw events asynchronously in real time. We introduce the concept of an
event blob as a spatio-temporal likelihood of event occurrence where the
conditional spatial likelihood is blob-like. Many real-world objects generate
event blob data, for example, flickering LEDs such as car headlights or any
small foreground object moving against a static or slowly varying background.
The proposed algorithm uses a nearest neighbour classifier with a dynamic
threshold criteria for data association coupled with a Kalman filter to track
the event blob state. Our algorithm achieves highly accurate tracking and event
blob shape estimation even under challenging lighting conditions and high-speed
motions. The microsecond time resolution achieved means that the filter output
can be used to derive secondary information such as time-to-contact or range
estimation, that will enable applications to real-world problems such as
collision avoidance in autonomous driving.Comment: 17 pages, 8 figures, preprint versio
Perception understanding action : adding understanding to the perception action cycle with spiking segmentation
Traditionally the Perception Action cycle is the first stage of building an autonomous robotic system and a practical way to implement a low latency reactive system within a low Size, Weight and Power (SWaP) package. However, within complex scenarios, this method can lack contextual understanding about the scene, such as object recognition-based tracking or system attention. Object detection, identification and tracking along with semantic segmentation and attention are all modern computer vision tasks in which Convolutional Neural Networks (CNN) have shown significant success, although such networks often have a large computational overhead and power requirements, which are not ideal in smaller robotics tasks. Furthermore, cloud computing and massively parallel processing like in Graphic Processing Units (GPUs) are outside the specification of many tasks due to their respective latency and SWaP constraints. In response to this, Spiking Convolutional Neural Networks (SCNNs) look to provide the feature extraction benefits of CNNs, while maintaining low latency and power overhead thanks to their asynchronous spiking event-based processing. A novel Neuromorphic Perception Understanding Action (PUA) system is presented, that aims to combine the feature extraction benefits of CNNs with low latency processing of SCNNs. The PUA utilizes a Neuromorphic Vision Sensor for Perception that facilitates asynchronous processing within a Spiking fully Convolutional Neural Network (SpikeCNN) to provide semantic segmentation and Understanding of the scene. The output is fed to a spiking control system providing Actions. With this approach, the aim is to bring features of deep learning into the lower levels of autonomous robotics, while maintaining a biologically plausible STDP rule throughout the learned encoding part of the network. The network will be shown to provide a more robust and predictable management of spiking activity with an improved thresholding response. The reported experiments show that this system can deliver robust results of over 96 and 81% for accuracy and Intersection over Union, ensuring such a system can be successfully used within object recognition, classification and tracking problem. This demonstrates that the attention of the system can be tracked accurately, while the asynchronous processing means the controller can give precise track updates with minimal latency
Event-based feature extraction using adaptive selection thresholds
Unsupervised feature extraction algorithms form one of the most important building blocks in machine learning systems. These algorithms are often adapted to the event-based domain to perform online learning in neuromorphic hardware. However, not designed for the purpose, such algorithms typically require significant simplification during implementation to meet hardware constraints, creating trade offs with performance. Furthermore, conventional feature extraction algorithms are not designed to generate useful intermediary signals which are valuable only in the context of neuromorphic hardware limitations. In this work a novel event-based feature extraction method is proposed that focuses on these issues. The algorithm operates via simple adaptive selection thresholds which allow a simpler implementation of network homeostasis than previous works by trading off a small amount of information loss in the form of missed events that fall outside the selection thresholds. The behavior of the selection thresholds and the output of the network as a whole are shown to provide uniquely useful signals indicating network weight convergence without the need to access network weights. A novel heuristic method for network size selection is proposed which makes use of noise events and their feature representations. The use of selection thresholds is shown to produce network activation patterns that predict classification accuracy allowing rapid evaluation and optimization of system parameters without the need to run back-end classifiers. The feature extraction method is tested on both the N-MNIST (Neuromorphic-MNIST) benchmarking dataset and a dataset of airplanes passing through the field of view. Multiple configurations with different classifiers are tested with the results quantifying the resultant performance gains at each processing stage