135 research outputs found
Block-Matching Optical Flow for Dynamic Vision Sensor- Algorithm and FPGA Implementation
Rapid and low power computation of optical flow (OF) is potentially useful in
robotics. The dynamic vision sensor (DVS) event camera produces quick and
sparse output, and has high dynamic range, but conventional OF algorithms are
frame-based and cannot be directly used with event-based cameras. Previous DVS
OF methods do not work well with dense textured input and are designed for
implementation in logic circuits. This paper proposes a new block-matching
based DVS OF algorithm which is inspired by motion estimation methods used for
MPEG video compression. The algorithm was implemented both in software and on
FPGA. For each event, it computes the motion direction as one of 9 directions.
The speed of the motion is set by the sample interval. Results show that the
Average Angular Error can be improved by 30\% compared with previous methods.
The OF can be calculated on FPGA with 50\,MHz clock in 0.2\,us per event (11
clock cycles), 20 times faster than a Java software implementation running on a
desktop PC. Sample data is shown that the method works on scenes dominated by
edges, sparse features, and dense texture.Comment: Published in ISCAS 201
EDFLOW: Event Driven Optical Flow Camera with Keypoint Detection and Adaptive Block Matching
Event cameras such as the Dynamic Vision Sensor (DVS) are useful because of their low latency, sparse output, and high dynamic range. In this paper, we propose a DVS+FPGA camera platform and use it to demonstrate the hardware implementation of event-based corner keypoint detection and adaptive block-matching optical flow. To adapt sample rate dynamically, events are accumulated in event slices using the area event count slice exposure method. The area event count is feedback controlled by the average optical flow matching distance. Corners are detected by streaks of accumulated events on event slice rings of radius 3 and 4 pixels. Corner detection takes about 6 clock cycles (16 MHz event rate at the 100MHz clock frequency) At the corners, flow vectors are computed in 100 clock cycles (1 MHz event rate). The multiscale block match size is 25x25 pixels and the flow vectors span up to 30-pixel match distance. The FPGA processes the sum-of-absolute distance block matching at 123 GOp/s, the equivalent of 1230 Op/clock cycle. EDFLOW is several times more accurate on MVSEC drone and driving optical flow benchmarking sequences than the previous best DVS FPGA optical flow implementation, and achieves similar accuracy to the CNN-based EV-Flownet, although it burns about 100 times less power. The EDFLOW design and benchmarking videos are available at https://sites.google.com/view/edflow21/home
Unraveling the paradox of intensity-dependent DVS pixel noise
Dynamic vision sensor (DVS) event camera output is affected by noise,
particularly in dim lighting conditions. A theory explaining how photon and
electron noise affect DVS output events has so far not been developed.
Moreover, there is no clear understanding of how DVS parameters and operating
conditions affect noise. There is an apparent paradox between the real noise
data observed from the DVS output and the reported noise measurements of the
logarithmic photoreceptor. While measurements of the logarithmic photoreceptor
predict that the photoreceptor is approximately a first-order system with RMS
noise voltage independent of the photocurrent, DVS output shows higher noise
event rates at low light intensity. This paper unravels this paradox by showing
how the DVS photoreceptor is a second-order system, and the assumption that it
is first-order is generally not reasonable. As we show, at higher
photocurrents, the photoreceptor amplifier dominates the frequency response,
causing a drop in RMS noise voltage and noise event rate. We bring light to the
noise performance of the DVS photoreceptor by presenting a theoretical
explanation supported by both transistor-level simulation results and chip
measurements.Comment: Presented in 2021 International Image Sensor Workshop (IISW
DDD17: End-To-End DAVIS Driving Dataset
Event cameras, such as dynamic vision sensors (DVS), and dynamic and
active-pixel vision sensors (DAVIS) can supplement other autonomous driving
sensors by providing a concurrent stream of standard active pixel sensor (APS)
images and DVS temporal contrast events. The APS stream is a sequence of
standard grayscale global-shutter image sensor frames. The DVS events represent
brightness changes occurring at a particular moment, with a jitter of about a
millisecond under most lighting conditions. They have a dynamic range of >120
dB and effective frame rates >1 kHz at data rates comparable to 30 fps
(frames/second) image sensors. To overcome some of the limitations of current
image acquisition technology, we investigate in this work the use of the
combined DVS and APS streams in end-to-end driving applications. The dataset
DDD17 accompanying this paper is the first open dataset of annotated DAVIS
driving recordings. DDD17 has over 12 h of a 346x260 pixel DAVIS sensor
recording highway and city driving in daytime, evening, night, dry and wet
weather conditions, along with vehicle speed, GPS position, driver steering,
throttle, and brake captured from the car's on-board diagnostics interface. As
an example application, we performed a preliminary end-to-end learning study of
using a convolutional neural network that is trained to predict the
instantaneous steering angle from DVS and APS visual data.Comment: Presented at the ICML 2017 Workshop on Machine Learning for
Autonomous Vehicle
The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
New vision sensors, such as the Dynamic and Active-pixel Vision sensor
(DAVIS), incorporate a conventional global-shutter camera and an event-based
sensor in the same pixel array. These sensors have great potential for
high-speed robotics and computer vision because they allow us to combine the
benefits of conventional cameras with those of event-based sensors: low
latency, high temporal resolution, and very high dynamic range. However, new
algorithms are required to exploit the sensor characteristics and cope with its
unconventional output, which consists of a stream of asynchronous brightness
changes (called "events") and synchronous grayscale frames. For this purpose,
we present and release a collection of datasets captured with a DAVIS in a
variety of synthetic and real environments, which we hope will motivate
research on new algorithms for high-speed and high-dynamic-range robotics and
computer-vision applications. In addition to global-shutter intensity images
and asynchronous events, we provide inertial measurements and ground-truth
camera poses from a motion-capture system. The latter allows comparing the pose
accuracy of ego-motion estimation algorithms quantitatively. All the data are
released both as standard text files and binary files (i.e., rosbag). This
paper provides an overview of the available data and describes a simulator that
we release open-source to create synthetic event-camera data.Comment: 7 pages, 4 figures, 3 table
Delta Networks for Optimized Recurrent Network Computation
Many neural networks exhibit stability in their activation patterns over time
in response to inputs from sensors operating under real-world conditions. By
capitalizing on this property of natural signals, we propose a Recurrent Neural
Network (RNN) architecture called a delta network in which each neuron
transmits its value only when the change in its activation exceeds a threshold.
The execution of RNNs as delta networks is attractive because their states must
be stored and fetched at every timestep, unlike in convolutional neural
networks (CNNs). We show that a naive run-time delta network implementation
offers modest improvements on the number of memory accesses and computes, but
optimized training techniques confer higher accuracy at higher speedup. With
these optimizations, we demonstrate a 9X reduction in cost with negligible loss
of accuracy for the TIDIGITS audio digit recognition benchmark. Similarly, on
the large Wall Street Journal speech recognition benchmark even existing
networks can be greatly accelerated as delta networks, and a 5.7x improvement
with negligible loss of accuracy can be obtained through training. Finally, on
an end-to-end CNN trained for steering angle prediction in a driving dataset,
the RNN cost can be reduced by a substantial 100X
Shining light on the DVS pixel: A tutorial and discussion about biasing and optimization
The operation of the DVS event camera is controlled by the user through
adjusting different bias parameters. These biases affect the response of the
camera by controlling - among other parameters - the bandwidth, sensitivity,
and maximum firing rate of the pixels. Besides determining the response of the
camera to input signals, biases significantly impact its noise performance.
Bias optimization is a multivariate process depending on the task and the
scene, to which the user's knowledge about pixel design and non-idealities can
be of great importance.
In this paper, we go step-by-step along the signal pathway of the DVS pixel,
shining light on its low-level operation and non-idealities, comparing pixel
level measurements with array level measurements, and discussing and how
biasing and illumination affect the pixel's behavior. With the results and
discussion presented, we aim to help DVS users achieve more hardware-aware
camera utilization and modelling.Comment: Accepted at 2023 IEEE/CVF Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW); 4th International Workshop on Event-Based
Visio
Exploiting Alternating DVS Shot Noise Event Pair Statistics to Reduce Background Activity Rates
Dynamic Vision Sensors (DVS) record âeventsâ corresponding to pixel-level brightness changes, resulting in dataefficient representation of a dynamic visual scene. As DVS expand into increasingly diverse applications, non-ideal behaviors in their output under extreme sensing conditions are important to consider. Under low illumination (below â10 lux) their output begins to be dominated by shot noise events (SNEs) which increase the data output and obscure true signal. SNE rates can be controlled to some degree by tuning circuit parameters to reduce sensitivity or temporal response bandwidth at the cost of signal loss. Alternatively, an improved understanding of SNE statistics can be leveraged to develop novel techniques for minimizing uninformative sensor output. We first explain a fundamental observation about sequential pairing of opposite polarity SNEs based on pixel circuit logic and validate our theory using DVS recordings and simulations. Finally, we derive a practical result from this new understanding and demonstrate two novel biasing techniques shown to reduce SNEs by 50% and 80% respectively while still retaining sensitivity and/or temporal resolution
- âŠ