154 research outputs found
Seeing through events: Real-time moving object sonification for visually impaired people using event-based camera
Scene sonification is a powerful technique to help Visually Impaired People (VIP) understand their surroundings. Existing methods usually perform sonification on the entire images of the surrounding scene acquired by a standard camera or on the priori static obstacles acquired by image processing algorithms on the RGB image of the surrounding scene. However, if all the information in the scene are delivered to VIP simultaneously, it will cause information redundancy. In fact, biological vision is more sensitive to moving objects in the scene than static objects, which is also the original intention of the event-based camera. In this paper, we propose a real-time sonification framework to help VIP understand the moving objects in the scene. First, we capture the events in the scene using an event-based camera and cluster them into multiple moving objects without relying on any prior knowledge. Then, sonification based on MIDI is enabled on these objects synchronously. Finally, we conduct comprehensive experiments on the scene video with sonification audio attended by 20 VIP and 20 Sighted People (SP). The results show that our method allows both participants to clearly distinguish the number, size, motion speed, and motion trajectories of multiple objects. The results show that our method is more comfortable to hear than existing methods in terms of aesthetics
Panoramic Annular Localizer: Tackling the Variation Challenges of Outdoor Localization Using Panoramic Annular Images and Active Deep Descriptors
Visual localization is an attractive problem that estimates the camera
localization from database images based on the query image. It is a crucial
task for various applications, such as autonomous vehicles, assistive
navigation and augmented reality. The challenging issues of the task lie in
various appearance variations between query and database images, including
illumination variations, dynamic object variations and viewpoint variations. In
order to tackle those challenges, Panoramic Annular Localizer into which
panoramic annular lens and robust deep image descriptors are incorporated is
proposed in this paper. The panoramic annular images captured by the single
camera are processed and fed into the NetVLAD network to form the active deep
descriptor, and sequential matching is utilized to generate the localization
result. The experiments carried on the public datasets and in the field
illustrate the validation of the proposed system.Comment: Accepted by ITSC 201
Towards Anytime Optical Flow Estimation with Event Cameras
Event cameras are capable of responding to log-brightness changes in
microseconds. Its characteristic of producing responses only to the changing
region is particularly suitable for optical flow estimation. In contrast to the
super low-latency response speed of event cameras, existing datasets collected
via event cameras, however, only provide limited frame rate optical flow ground
truth, (e.g., at 10Hz), greatly restricting the potential of event-driven
optical flow. To address this challenge, we put forward a high-frame-rate,
low-latency event representation Unified Voxel Grid, sequentially fed into the
network bin by bin. We then propose EVA-Flow, an EVent-based Anytime Flow
estimation network to produce high-frame-rate event optical flow with only
low-frame-rate optical flow ground truth for supervision. The key component of
our EVA-Flow is the stacked Spatiotemporal Motion Refinement (SMR) module,
which predicts temporally-dense optical flow and enhances the accuracy via
spatial-temporal motion refinement. The time-dense feature warping utilized in
the SMR module provides implicit supervision for the intermediate optical flow.
Additionally, we introduce the Rectified Flow Warp Loss (RFWL) for the
unsupervised evaluation of intermediate optical flow in the absence of ground
truth. This is, to the best of our knowledge, the first work focusing on
anytime optical flow estimation via event cameras. A comprehensive variety of
experiments on MVSEC, DESC, and our EVA-FlowSet demonstrates that EVA-Flow
achieves competitive performance, super-low-latency (5ms), fastest inference
(9.2ms), time-dense motion estimation (200Hz), and strong generalization. Our
code will be available at https://github.com/Yaozhuwa/EVA-Flow.Comment: Code will be available at https://github.com/Yaozhuwa/EVA-Flo
FlowLens: Seeing Beyond the FoV via Flow-guided Clip-Recurrent Transformer
Limited by hardware cost and system size, camera's Field-of-View (FoV) is not
always satisfactory. However, from a spatio-temporal perspective, information
beyond the camera's physical FoV is off-the-shelf and can actually be obtained
"for free" from the past. In this paper, we propose a novel task termed
Beyond-FoV Estimation, aiming to exploit past visual cues and bidirectional
break through the physical FoV of a camera. We put forward a FlowLens
architecture to expand the FoV by achieving feature propagation explicitly by
optical flow and implicitly by a novel clip-recurrent transformer, which has
two appealing features: 1) FlowLens comprises a newly proposed Clip-Recurrent
Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global
information accumulated in the temporal dimension. 2) A multi-branch Mix Fusion
Feed Forward Network (MixF3N) is integrated to enhance the spatially-precise
flow of local features. To foster training and evaluation, we establish
KITTI360-EX, a dataset for outer- and inner FoV expansion. Extensive
experiments on both video inpainting and beyond-FoV estimation tasks show that
FlowLens achieves state-of-the-art performance. Code will be made publicly
available at https://github.com/MasterHow/FlowLens.Comment: Code will be made publicly available at
https://github.com/MasterHow/FlowLen
- …