Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration

Recent years have witnessed the remarkable performance of diffusion models in various vision tasks. However, for image restoration that aims to recover clear images with sharper details from given degraded observations, diffusion-based methods may fail to recover promising results due to inaccurate noise estimation. Moreover, simple constraining noises cannot effectively learn complex degradation information, which subsequently hinders the model capacity. To solve the above problems, we propose a coarse-to-fine diffusion Transformer (C2F-DFT) for image restoration. Specifically, our C2F-DFT contains diffusion self-attention (DFSA) and diffusion feed-forward network (DFN) within a new coarse-to-fine training scheme. The DFSA and DFN respectively capture the long-range diffusion dependencies and learn hierarchy diffusion representation to facilitate better restoration. In the coarse training stage, our C2F-DFT estimates noises and then generates the final clean image by a sampling algorithm. To further improve the restoration quality, we propose a simple yet effective fine training scheme. It first exploits the coarse-trained diffusion model with fixed steps to generate restoration results, which then would be constrained with corresponding ground-truth ones to optimize the models to remedy the unsatisfactory results affected by inaccurate noise estimation. Extensive experiments show that C2F-DFT significantly outperforms diffusion-based restoration method IR-SDE and achieves competitive performance compared with Transformer-based state-of-the-art methods on $3$ tasks, including deraining, deblurring, and real denoising.Comment: 9 pages, 8 figure

A Dive into SAM Prior in Image Restoration

The goal of image restoration (IR), a fundamental issue in computer vision, is to restore a high-quality (HQ) image from its degraded low-quality (LQ) observation. Multiple HQ solutions may correspond to an LQ input in this poorly posed problem, creating an ambiguous solution space. This motivates the investigation and incorporation of prior knowledge in order to effectively constrain the solution space and enhance the quality of the restored images. In spite of the pervasive use of hand-crafted and learned priors in IR, limited attention has been paid to the incorporation of knowledge from large-scale foundation models. In this paper, we for the first time leverage the prior knowledge of the state-of-the-art segment anything model (SAM) to boost the performance of existing IR networks in an parameter-efficient tuning manner. In particular, the choice of SAM is based on its robustness to image degradations, such that HQ semantic masks can be extracted from it. In order to leverage semantic priors and enhance restoration quality, we propose a lightweight SAM prior tuning (SPT) unit. This plug-and-play component allows us to effectively integrate semantic priors into existing IR networks, resulting in significant improvements in restoration quality. As the only trainable module in our method, the SPT unit has the potential to improve both efficiency and scalability. We demonstrate the effectiveness of the proposed method in enhancing a variety of methods across multiple tasks, such as image super-resolution and color image denoising.Comment: Technical Repor

Generative Modeling in Structural-Hankel Domain for Color Image Inpainting

In recent years, some researchers focused on using a single image to obtain a large number of samples through multi-scale features. This study intends to a brand-new idea that requires only ten or even fewer samples to construct the low-rank structural-Hankel matrices-assisted score-based generative model (SHGM) for color image inpainting task. During the prior learning process, a certain amount of internal-middle patches are firstly extracted from several images and then the structural-Hankel matrices are constructed from these patches. To better apply the score-based generative model to learn the internal statistical distribution within patches, the large-scale Hankel matrices are finally folded into the higher dimensional tensors for prior learning. During the iterative inpainting process, SHGM views the inpainting problem as a conditional generation procedure in low-rank environment. As a result, the intermediate restored image is acquired by alternatively performing the stochastic differential equation solver, alternating direction method of multipliers, and data consistency steps. Experimental results demonstrated the remarkable performance and diversity of SHGM.Comment: 11 pages, 10 figure

Mini-Workshop: Analytical and Numerical Methods in Image and Surface Processing

The workshop successfully brought together researchers from mathematical analysis, numerical mathematics, computer graphics and image processing. The focus was on variational methods in image and surface processing such as active contour models, Mumford-Shah type functionals, image and surface denoising based on geometric evolution problems in image and surface fairing, physical modeling of surfaces, the restoration of images and surfaces using higher order variational formulations

DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence

Removing soft and self shadows that lack clear boundaries from a single image is still challenging. Self shadows are shadows that are cast on the object itself. Most existing methods rely on binary shadow masks, without considering the ambiguous boundaries of soft and self shadows. In this paper, we present DeS3, a method that removes hard, soft and self shadows based on the self-tuned ViT feature similarity and color convergence. Our novel ViT similarity loss utilizes features extracted from a pre-trained Vision Transformer. This loss helps guide the reverse diffusion process towards recovering scene structures. We also introduce a color convergence loss to constrain the surface colors in the reverse inference process to avoid any color shifts. Our DeS3 is able to differentiate shadow regions from the underlying objects, as well as shadow regions from the object casting the shadow. This capability enables DeS3 to better recover the structures of objects even when they are partially occluded by shadows. Different from existing methods that rely on constraints during the training phase, we incorporate the ViT similarity and color convergence loss during the sampling stage. This enables our DeS3 model to effectively integrate its strong modeling capabilities with input-specific knowledge in a self-tuned manner. Our method outperforms state-of-the-art methods on the SRD, AISTD, LRSS, USR and UIUC datasets, removing hard, soft, and self shadows robustly. Specifically, our method outperforms the SOTA method by 20% of the RMSE of the whole image on the SRD dataset

A multigrid platform for real-time motion computation with discontinuity-preserving variational methods

Variational methods are among the most accurate techniques for estimating the optic flow. They yield dense flow fields and can be designed such that they preserve discontinuities, allow to deal with large displacements and perform well under noise or varying illumination. However, such adaptations render the minimisation of the underlying energy functional very expensive in terms of computational costs: Typically, one or more large linear or nonlinear systems of equations have to be solved in order to obtain the desired solution. Consequently, variational methods are considered to be too slow for real-time performance. In our paper we address this problem in two ways: (i) We present a numerical framework based on bidirectional multigrid methods for accelerating a broad class of variational optic flow methods with different constancy and smoothness assumptions. In particular, discontinuity-preserving regularisation strategies are thereby in the focus of our work. (ii) We show by the examples of classical as well as more advanced variational techniques that real-time performance is possible - even for very complex optic flow models with high accuracy. Experiments show frame rates up to 63 dense flow fields per second for real-world image sequences of size 160 x 120 on a standard PC. Compared to classical iterative methods this constitutes a speedup of two to four orders of magnitude

AMPA Receptor Phosphorylation and Synaptic Colocalization on Motor Neurons Drive Maladaptive Plasticity below Complete Spinal Cord Injury.

Clinical spinal cord injury (SCI) is accompanied by comorbid peripheral injury in 47% of patients. Human and animal modeling data have shown that painful peripheral injuries undermine long-term recovery of locomotion through unknown mechanisms. Peripheral nociceptive stimuli induce maladaptive synaptic plasticity in dorsal horn sensory systems through AMPA receptor (AMPAR) phosphorylation and trafficking to synapses. Here we test whether ventral horn motor neurons in rats demonstrate similar experience-dependent maladaptive plasticity below a complete SCI in vivo. Quantitative biochemistry demonstrated that intermittent nociceptive stimulation (INS) rapidly and selectively increases AMPAR subunit GluA1 serine 831 phosphorylation and localization to synapses in the injured spinal cord, while reducing synaptic GluA2. These changes predict motor dysfunction in the absence of cell death signaling, suggesting an opportunity for therapeutic reversal. Automated confocal time-course analysis of lumbar ventral horn motor neurons confirmed a time-dependent increase in synaptic GluA1 with concurrent decrease in synaptic GluA2. Optical fractionation of neuronal plasma membranes revealed GluA2 removal from extrasynaptic sites on motor neurons early after INS followed by removal from synapses 2 h later. As GluA2-lacking AMPARs are canonical calcium-permeable AMPARs (CP-AMPARs), their stimulus- and time-dependent insertion provides a therapeutic target for limiting calcium-dependent dynamic maladaptive plasticity after SCI. Confirming this, a selective CP-AMPAR antagonist protected against INS-induced maladaptive spinal plasticity, restoring adaptive motor responses on a sensorimotor spinal training task. These findings highlight the critical involvement of AMPARs in experience-dependent spinal cord plasticity after injury and provide a pharmacologically targetable synaptic mechanism by which early postinjury experience shapes motor plasticity