20 research outputs found
CED: Color Event Camera Dataset
Event cameras are novel, bio-inspired visual sensors, whose pixels output
asynchronous and independent timestamped spikes at local intensity changes,
called 'events'. Event cameras offer advantages over conventional frame-based
cameras in terms of latency, high dynamic range (HDR) and temporal resolution.
Until recently, event cameras have been limited to outputting events in the
intensity channel, however, recent advances have resulted in the development of
color event cameras, such as the Color-DAVIS346. In this work, we present and
release the first Color Event Camera Dataset (CED), containing 50 minutes of
footage with both color frames and events. CED features a wide variety of
indoor and outdoor scenes, which we hope will help drive forward event-based
vision research. We also present an extension of the event camera simulator
ESIM that enables simulation of color events. Finally, we present an evaluation
of three state-of-the-art image reconstruction methods that can be used to
convert the Color-DAVIS346 into a continuous-time, HDR, color video camera to
visualise the event stream, and for use in downstream vision applications.Comment: Conference on Computer Vision and Pattern Recognition Workshop
Advances on CMOS image sensors
This paper offers an introduction to the technological advances of image sensors designed using
complementary metal–oxide–semiconductor (CMOS) processes along the last decades. We review
some of those technological advances and examine potential disruptive growth directions for CMOS
image sensors and proposed ways to achieve them. Those advances include breakthroughs on
image quality such as resolution, capture speed, light sensitivity and color detection and advances on
the computational imaging. The current trend is to push the innovation efforts even further as the
market requires higher resolution, higher speed, lower power consumption and, mainly, lower cost
sensors. Although CMOS image sensors are currently used in several different applications from
consumer to defense to medical diagnosis, product differentiation is becoming both a requirement and
a difficult goal for any image sensor manufacturer. The unique properties of CMOS process allows the
integration of several signal processing techniques and are driving the impressive advancement of the
computational imaging. With this paper, we offer a very comprehensive review of methods,
techniques, designs and fabrication of CMOS image sensors that have impacted or might will impact
the images sensor applications and markets
Event-based, 6-DOF Camera Tracking from Photometric Depth Maps
Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. These features, along with a very low power consumption, make event cameras an ideal complement to standard cameras for VR/AR and video game applications. With these applications in mind, this paper tackles the problem of accurate, low-latency tracking of an event camera from an existing photometric depth map (i.e., intensity plus depth information) built via classic dense reconstruction pipelines. Our approach tracks the 6-DOF pose of the event camera upon the arrival of each event, thus virtually eliminating latency. We successfully evaluate the method in both indoor and outdoor scenes and show that—because of the technological advantages of the event camera—our pipeline works in scenes characterized by high-speed motion, which are still unaccessible to standard cameras
Physical Characteristics, Sensors and Applications of 2D/3DIntegrated CMOS Photodiodes
Two-dimensional photodiodes are reversely biased at a reasonable voltage whereas 3D photodiodes are likely operated at the Geiger mode. How to design integrated 2D and 3D photodiodes is investigated in terms of quantum efficiency, dark current, crosstalk, response time and so on. Beyond photodiodes, a charge supply mechanism provides a proper charge for a high dynamic range of 2D sensing, and a feedback pull-down mechanism expedites the response time of 3D sensing for time-of-flight applications. Particularly, rapid parallel reading at a 3D mode is developed by a bus-sharing mechanism. Using the TSMC 0.35ÎĽm 2P4M technology, a 2D/3D-integrated image sensor including P-diffusion_N-well_P-substrate photodiodes, pixel circuits, correlated double sampling circuits, sense amplifiers, a multi-channel time-to-digital converter, column/row decoders, bus-sharing connections/decoders, readout circuits and so on was implemented with a die size of 12mmĂ—12mm. The proposed 2D/3D-integrated image sensor can perceive a 352Ă—288-pixel 2D image and an 88Ă—72-pixel 3D image with a dynamic range up to 100dB and a depth resolution of around 4cm, respectively. Therefore, our image sensor can effectively capture gray-level and depth information of a scene at the same location without additional alignment and post-processing. Finally, the currently available 2D and 3D image sensors are discussed and presented
EDFLOW: Event Driven Optical Flow Camera with Keypoint Detection and Adaptive Block Matching
Event cameras such as the Dynamic Vision Sensor (DVS) are useful because of their low latency, sparse output, and high dynamic range. In this paper, we propose a DVS+FPGA camera platform and use it to demonstrate the hardware implementation of event-based corner keypoint detection and adaptive block-matching optical flow. To adapt sample rate dynamically, events are accumulated in event slices using the area event count slice exposure method. The area event count is feedback controlled by the average optical flow matching distance. Corners are detected by streaks of accumulated events on event slice rings of radius 3 and 4 pixels. Corner detection takes about 6 clock cycles (16 MHz event rate at the 100MHz clock frequency) At the corners, flow vectors are computed in 100 clock cycles (1 MHz event rate). The multiscale block match size is 25x25 pixels and the flow vectors span up to 30-pixel match distance. The FPGA processes the sum-of-absolute distance block matching at 123 GOp/s, the equivalent of 1230 Op/clock cycle. EDFLOW is several times more accurate on MVSEC drone and driving optical flow benchmarking sequences than the previous best DVS FPGA optical flow implementation, and achieves similar accuracy to the CNN-based EV-Flownet, although it burns about 100 times less power. The EDFLOW design and benchmarking videos are available at https://sites.google.com/view/edflow21/home
Active Perception with Dynamic Vision Sensors. Minimum Saccades with Optimum Recognition
Vision processing with Dynamic Vision Sensors
(DVS) is becoming increasingly popular. This type of bio-inspired
vision sensor does not record static scenes. DVS pixel activity
relies on changes in light intensity. In this paper, we introduce
a platform for object recognition with a DVS in which the
sensor is installed on a moving pan-tilt unit in closed-loop with
a recognition neural network. This neural network is trained
to recognize objects observed by a DVS while the pan-tilt unit
is moved to emulate micro-saccades. We show that performing
more saccades in different directions can result in having more
information about the object and therefore more accurate object
recognition is possible. However, in high performance and low latency
platforms, performing additional saccades adds additional
latency and power consumption. Here we show that the number
of saccades can be reduced while keeping the same recognition
accuracy by performing intelligent saccadic movements, in a
closed action-perception smart loop. We propose an algorithm
for smart saccadic movement decisions that can reduce the
number of necessary saccades to half, on average, for a predefined
accuracy on the N-MNIST dataset. Additionally, we show that
by replacing this control algorithm with an Artificial Neural
Network that learns to control the saccades, we can also reduce
to half the average number of saccades needed for N-MNIST
recognition.EU H2020 grant 644096 ECOMODEEU H2020 grant 687299 NEURAM3Ministry of Economy and Competitivity (Spain) / European Regional Development Fund TEC2015-63884-C2-1-P (COGNET
Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age
Simultaneous Localization and Mapping (SLAM)consists in the concurrent
construction of a model of the environment (the map), and the estimation of the
state of the robot moving within it. The SLAM community has made astonishing
progress over the last 30 years, enabling large-scale real-world applications,
and witnessing a steady transition of this technology to industry. We survey
the current state of SLAM. We start by presenting what is now the de-facto
standard formulation for SLAM. We then review related work, covering a broad
set of topics including robustness and scalability in long-term mapping, metric
and semantic representations for mapping, theoretical performance guarantees,
active SLAM and exploration, and other new frontiers. This paper simultaneously
serves as a position paper and tutorial to those who are users of SLAM. By
looking at the published research with a critical eye, we delineate open
challenges and new research issues, that still deserve careful scientific
investigation. The paper also contains the authors' take on two questions that
often animate discussions during robotics conferences: Do robots need SLAM? and
Is SLAM solved
Event-based Vision: A Survey
Event cameras are bio-inspired sensors that differ from conventional frame
cameras: Instead of capturing images at a fixed rate, they asynchronously
measure per-pixel brightness changes, and output a stream of events that encode
the time, location and sign of the brightness changes. Event cameras offer
attractive properties compared to traditional cameras: high temporal resolution
(in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low
power consumption, and high pixel bandwidth (on the order of kHz) resulting in
reduced motion blur. Hence, event cameras have a large potential for robotics
and computer vision in challenging scenarios for traditional cameras, such as
low-latency, high speed, and high dynamic range. However, novel methods are
required to process the unconventional output of these sensors in order to
unlock their potential. This paper provides a comprehensive overview of the
emerging field of event-based vision, with a focus on the applications and the
algorithms developed to unlock the outstanding properties of event cameras. We
present event cameras from their working principle, the actual sensors that are
available and the tasks that they have been used for, from low-level vision
(feature detection and tracking, optic flow, etc.) to high-level vision
(reconstruction, segmentation, recognition). We also discuss the techniques
developed to process events, including learning-based techniques, as well as
specialized processors for these novel sensors, such as spiking neural
networks. Additionally, we highlight the challenges that remain to be tackled
and the opportunities that lie ahead in the search for a more efficient,
bio-inspired way for machines to perceive and interact with the world