9 research outputs found
Journey towards tiny perceptual super-resolution
Recent works in single-image perceptual super-resolution (SR) have demonstrated unprecedented performance in generating realistic textures by means of deep convolutional networks. However, these convolutional models are excessively large and expensive, hindering their effective deployment to end devices. In this work, we propose a neural architecture search (NAS) approach that integrates NAS and generative adversarial networks (GANs) with recent advances in perceptual SR and pushes the efficiency of small perceptual SR models to facilitate on-device execution. Specifically, we search over the architectures of both the generator and the discriminator sequentially, highlighting the unique challenges and key observations of searching for an SR-optimized discriminator and comparing them with existing discriminator architectures in the literature. Our tiny perceptual SR (TPSR) models outperform SRGAN and EnhanceNet on both full-reference perceptual metric (LPIPS) and distortion metric (PSNR) while being up to 26.4 × more memory efficient and 33.6 × more compute efficient respectively
Kernelized Back-Projection Networks for Blind Super Resolution
Since non-blind Super Resolution (SR) fails to super-resolve Low-Resolution
(LR) images degraded by arbitrary degradations, SR with the degradation model
is required. However, this paper reveals that non-blind SR that is trained
simply with various blur kernels exhibits comparable performance as those with
the degradation model for blind SR. This result motivates us to revisit
high-performance non-blind SR and extend it to blind SR with blur kernels. This
paper proposes two SR networks by integrating kernel estimation and SR branches
in an iterative end-to-end manner. In the first model, which is called the
Kernel Conditioned Back-Projection Network (KCBPN), the low-dimensional kernel
representations are estimated for conditioning the SR branch. In our second
model, the Kernelized BackProjection Network (KBPN), a raw kernel is estimated
and directly employed for modeling the image degradation. The estimated kernel
is employed not only for back-propagating its residual but also for
forward-propagating the residual to iterative stages. This forward-propagation
encourages these stages to learn a variety of different features in different
stages by focusing on pixels with large residuals in each stage. Experimental
results validate the effectiveness of our proposed networks for kernel
estimation and SR. We will release the code for this work.Comment: The first two authors contributed equally to this wor
Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search
Though recent years have witnessed remarkable progress in single image
super-resolution (SISR) tasks with the prosperous development of deep neural
networks (DNNs), the deep learning methods are confronted with the computation
and memory consumption issues in practice, especially for resource-limited
platforms such as mobile devices. To overcome the challenge and facilitate the
real-time deployment of SISR tasks on mobile, we combine neural architecture
search with pruning search and propose an automatic search framework that
derives sparse super-resolution (SR) models with high image quality while
satisfying the real-time inference requirement. To decrease the search cost, we
leverage the weight sharing strategy by introducing a supernet and decouple the
search problem into three stages, including supernet construction,
compiler-aware architecture and pruning search, and compiler-aware pruning
ratio search. With the proposed framework, we are the first to achieve
real-time SR inference (with only tens of milliseconds per frame) for
implementing 720p resolution with competitive image quality (in terms of PSNR
and SSIM) on mobile platforms (Samsung Galaxy S20)
Zero-Cost Proxies Meet Differentiable Architecture Search
Differentiable neural architecture search (NAS) has attracted significant
attention in recent years due to its ability to quickly discover promising
architectures of deep neural networks even in very large search spaces. Despite
its success, DARTS lacks robustness in certain cases, e.g. it may degenerate to
trivial architectures with excessive parametric-free operations such as skip
connection or random noise, leading to inferior performance. In particular,
operation selection based on the magnitude of architectural parameters was
recently proven to be fundamentally wrong showcasing the need to rethink this
aspect. On the other hand, zero-cost proxies have been recently studied in the
context of sample-based NAS showing promising results -- speeding up the search
process drastically in some cases but also failing on some of the large search
spaces typical for differentiable NAS. In this work we propose a novel
operation selection paradigm in the context of differentiable NAS which
utilises zero-cost proxies. Our perturbation-based zero-cost operation
selection (Zero-Cost-PT) improves searching time and, in many cases, accuracy
compared to the best available differentiable architecture search, regardless
of the search space size. Specifically, we are able to find comparable
architectures to DARTS-PT on the DARTS CNN search space while being over 40x
faster (total searching time 25 minutes on a single GPU)
Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
With the advent of Deep Learning (DL), Super-Resolution (SR) has also become
a thriving research area. However, despite promising results, the field still
faces challenges that require further research e.g., allowing flexible
upsampling, more effective loss functions, and better evaluation metrics. We
review the domain of SR in light of recent advances, and examine
state-of-the-art models such as diffusion (DDPM) and transformer-based SR
models. We present a critical discussion on contemporary strategies used in SR,
and identify promising yet unexplored research directions. We complement
previous surveys by incorporating the latest developments in the field such as
uncertainty-driven losses, wavelet networks, neural architecture search, novel
normalization methods, and the latests evaluation techniques. We also include
several visualizations for the models and methods throughout each chapter in
order to facilitate a global understanding of the trends in the field. This
review is ultimately aimed at helping researchers to push the boundaries of DL
applied to SR.Comment: accepted by IEEE Transactions on Pattern Analysis and Machine
Intelligence, 202