67 research outputs found
Event-guided Multi-patch Network with Self-supervision for Non-uniform Motion Deblurring
Contemporary deep learning multi-scale deblurring models suffer from many
issues: 1) They perform poorly on non-uniformly blurred images/videos; 2)
Simply increasing the model depth with finer-scale levels cannot improve
deblurring; 3) Individual RGB frames contain a limited motion information for
deblurring; 4) Previous models have a limited robustness to spatial
transformations and noise. Below, we extend the DMPHN model by several
mechanisms to address the above issues: I) We present a novel self-supervised
event-guided deep hierarchical Multi-patch Network (MPN) to deal with blurry
images and videos via fine-to-coarse hierarchical localized representations;
II) We propose a novel stacked pipeline, StackMPN, to improve the deblurring
performance under the increased network depth; III) We propose an event-guided
architecture to exploit motion cues contained in videos to tackle complex blur
in videos; IV) We propose a novel self-supervised step to expose the model to
random transformations (rotations, scale changes), and make it robust to
Gaussian noises. Our MPN achieves the state of the art on the GoPro and
VideoDeblur datasets with a 40x faster runtime compared to current multi-scale
methods. With 30ms to process an image at 1280x720 resolution, it is the first
real-time deep motion deblurring model for 720p images at 30fps. For StackMPN,
we obtain significant improvements over 1.2dB on the GoPro dataset by
increasing the network depth. Utilizing the event information and
self-supervision further boost results to 33.83dB.Comment: International Journal of Computer Vision. arXiv admin note:
substantial text overlap with arXiv:1904.0346
MC-Blur: A Comprehensive Benchmark for Image Deblurring
Blur artifacts can seriously degrade the visual quality of images, and
numerous deblurring methods have been proposed for specific scenarios. However,
in most real-world images, blur is caused by different factors, e.g., motion
and defocus. In this paper, we address how different deblurring methods perform
in the case of multiple types of blur. For in-depth performance evaluation, we
construct a new large-scale multi-cause image deblurring dataset (called
MC-Blur), including real-world and synthesized blurry images with mixed factors
of blurs. The images in the proposed MC-Blur dataset are collected using
different techniques: averaging sharp images captured by a 1000-fps high-speed
camera, convolving Ultra-High-Definition (UHD) sharp images with large-size
kernels, adding defocus to images, and real-world blurry images captured by
various camera models. Based on the MC-Blur dataset, we conduct extensive
benchmarking studies to compare SOTA methods in different scenarios, analyze
their efficiency, and investigate the built dataset's capacity. These
benchmarking results provide a comprehensive overview of the advantages and
limitations of current deblurring methods, and reveal the advances of our
dataset
Event-Based Fusion for Motion Deblurring with Cross-modal Attention
Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47dB) and on our REBlur dataset, even in extreme blurry conditions. The code and our REBlur dataset will be made publicly available
BANet: Blur-aware Attention Networks for Dynamic Scene Deblurring
Image motion blur usually results from moving objects or camera shakes. Such
blur is generally directional and non-uniform. Previous research efforts
attempt to solve non-uniform blur by using self-recurrent multi-scale or
multi-patch architectures accompanying with self-attention. However, using
self-recurrent frameworks typically leads to a longer inference time, while
inter-pixel or inter-channel self-attention may cause excessive memory usage.
This paper proposes blur-aware attention networks (BANet) that accomplish
accurate and efficient deblurring via a single forward pass. Our BANet utilizes
region-based self-attention with multi-kernel strip pooling to disentangle blur
patterns of different degrees and with cascaded parallel dilated convolution to
aggregate multi-scale content features. Extensive experimental results on the
GoPro and HIDE benchmarks demonstrate that the proposed BANet performs
favorably against the state-of-the-art in blurred image restoration and can
provide deblurred results in real-time
Event-Based Fusion for Motion Deblurring with Cross-modal Attention
Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47dB) and on our REBlur dataset, even in extreme blurry conditions. The code and our REBlur dataset will be made publicly available
- …