577 research outputs found
Deformable Kernel Networks for Joint Image Filtering
Joint image filters are used to transfer structural details from a guidance
picture used as a prior to a target image, in tasks such as enhancing spatial
resolution and suppressing noise. Previous methods based on convolutional
neural networks (CNNs) combine nonlinear activations of spatially-invariant
kernels to estimate structural details and regress the filtering result. In
this paper, we instead learn explicitly sparse and spatially-variant kernels.
We propose a CNN architecture and its efficient implementation, called the
deformable kernel network (DKN), that outputs sets of neighbors and the
corresponding weights adaptively for each pixel. The filtering result is then
computed as a weighted average. We also propose a fast version of DKN that runs
about seventeen times faster for an image of size 640 x 480. We demonstrate the
effectiveness and flexibility of our models on the tasks of depth map
upsampling, saliency map upsampling, cross-modality image restoration, texture
removal, and semantic segmentation. In particular, we show that the weighted
averaging process with sparsely sampled 3 x 3 kernels outperforms the state of
the art by a significant margin in all cases.Comment: arXiv admin note: substantial text overlap with arXiv:1903.11286
(IJCV accepted
Learning to Upsample by Learning to Sample
We present DySample, an ultra-lightweight and effective dynamic upsampler.
While impressive performance gains have been witnessed from recent kernel-based
dynamic upsamplers such as CARAFE, FADE, and SAPA, they introduce much
workload, mostly due to the time-consuming dynamic convolution and the
additional sub-network used to generate dynamic kernels. Further, the need for
high-res feature guidance of FADE and SAPA somehow limits their application
scenarios. To address these concerns, we bypass dynamic convolution and
formulate upsampling from the perspective of point sampling, which is more
resource-efficient and can be easily implemented with the standard built-in
function in PyTorch. We first showcase a naive design, and then demonstrate how
to strengthen its upsampling behavior step by step towards our new upsampler,
DySample. Compared with former kernel-based dynamic upsamplers, DySample
requires no customized CUDA package and has much fewer parameters, FLOPs, GPU
memory, and latency. Besides the light-weight characteristics, DySample
outperforms other upsamplers across five dense prediction tasks, including
semantic segmentation, object detection, instance segmentation, panoptic
segmentation, and monocular depth estimation. Code is available at
https://github.com/tiny-smart/dysample.Comment: Accepted by ICCV 202
- …