21 research outputs found
Towards Consistent Video Editing with Text-to-Image Diffusion Models
Existing works have advanced Text-to-Image (TTI) diffusion models for video
editing in a one-shot learning manner. Despite their low requirements of data
and computation, these methods might produce results of unsatisfied consistency
with text prompt as well as temporal sequence, limiting their applications in
the real world. In this paper, we propose to address the above issues with a
novel EI model towards \textbf{E}nhancing v\textbf{I}deo \textbf{E}diting
cons\textbf{I}stency of TTI-based frameworks. Specifically, we analyze and find
that the inconsistent problem is caused by newly added modules into TTI models
for learning temporal information. These modules lead to covariate shift in the
feature space, which harms the editing capability. Thus, we design EI to
tackle the above drawbacks with two classical modules: Shift-restricted
Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM).
First, through theoretical analysis, we demonstrate that covariate shift is
highly related to Layer Normalization, thus STAM employs a \textit{Instance
Centering} layer replacing it to preserve the distribution of temporal
features. In addition, {STAM} employs an attention layer with normalized
mapping to transform temporal features while constraining the variance shift.
As the second part, we incorporate {STAM} with a novel {FFAM}, which
efficiently leverages fine-coarse spatial information of overall frames to
further enhance temporal consistency. Extensive experiments demonstrate the
superiority of the proposed EI model for text-driven video editing
DiffBFR: Bootstrapping Diffusion Model Towards Blind Face Restoration
Blind face restoration (BFR) is important while challenging. Prior works
prefer to exploit GAN-based frameworks to tackle this task due to the balance
of quality and efficiency. However, these methods suffer from poor stability
and adaptability to long-tail distribution, failing to simultaneously retain
source identity and restore detail. We propose DiffBFR to introduce Diffusion
Probabilistic Model (DPM) for BFR to tackle the above problem, given its
superiority over GAN in aspects of avoiding training collapse and generating
long-tail distribution. DiffBFR utilizes a two-step design, that first restores
identity information from low-quality images and then enhances texture details
according to the distribution of real faces. This design is implemented with
two key components: 1) Identity Restoration Module (IRM) for preserving the
face details in results. Instead of denoising from pure Gaussian random
distribution with LQ images as the condition during the reverse process, we
propose a novel truncated sampling method which starts from LQ images with part
noise added. We theoretically prove that this change shrinks the evidence lower
bound of DPM and then restores more original details. With theoretical proof,
two cascade conditional DPMs with different input sizes are introduced to
strengthen this sampling effect and reduce training difficulty in the
high-resolution image generated directly. 2) Texture Enhancement Module (TEM)
for polishing the texture of the image. Here an unconditional DPM, a LQ-free
model, is introduced to further force the restorations to appear realistic. We
theoretically proved that this unconditional DPM trained on pure HQ images
contributes to justifying the correct distribution of inference images output
from IRM in pixel-level space. Truncated sampling with fractional time step is
utilized to polish pixel-level textures while preserving identity information
eMotions: A Large-Scale Dataset for Emotion Recognition in Short Videos
Nowadays, short videos (SVs) are essential to information acquisition and
sharing in our life. The prevailing use of SVs to spread emotions leads to the
necessity of emotion recognition in SVs. Considering the lack of SVs emotion
data, we introduce a large-scale dataset named eMotions, comprising 27,996
videos. Meanwhile, we alleviate the impact of subjectivities on labeling
quality by emphasizing better personnel allocations and multi-stage
annotations. In addition, we provide the category-balanced and test-oriented
variants through targeted data sampling. Some commonly used videos (e.g.,
facial expressions and postures) have been well studied. However, it is still
challenging to understand the emotions in SVs. Since the enhanced content
diversity brings more distinct semantic gaps and difficulties in learning
emotion-related features, and there exists information gaps caused by the
emotion incompleteness under the prevalently audio-visual co-expressions. To
tackle these problems, we present an end-to-end baseline method AV-CPNet that
employs the video transformer to better learn semantically relevant
representations. We further design the two-stage cross-modal fusion module to
complementarily model the correlations of audio-visual features. The EP-CE
Loss, incorporating three emotion polarities, is then applied to guide model
optimization. Extensive experimental results on nine datasets verify the
effectiveness of AV-CPNet. Datasets and code will be open on
https://github.com/XuecWu/eMotions
DropKey
In this paper, we focus on analyzing and improving the dropout technique for
self-attention layers of Vision Transformer, which is important while
surprisingly ignored by prior works. In particular, we conduct researches on
three core questions: First, what to drop in self-attention layers? Different
from dropping attention weights in literature, we propose to move dropout
operations forward ahead of attention matrix calculation and set the Key as the
dropout unit, yielding a novel dropout-before-softmax scheme. We theoretically
verify that this scheme helps keep both regularization and probability features
of attention weights, alleviating the overfittings problem to specific patterns
and enhancing the model to globally capture vital information; Second, how to
schedule the drop ratio in consecutive layers? In contrast to exploit a
constant drop ratio for all layers, we present a new decreasing schedule that
gradually decreases the drop ratio along the stack of self-attention layers. We
experimentally validate the proposed schedule can avoid overfittings in
low-level features and missing in high-level semantics, thus improving the
robustness and stableness of model training; Third, whether need to perform
structured dropout operation as CNN? We attempt patch-based block-version of
dropout operation and find that this useful trick for CNN is not essential for
ViT. Given exploration on the above three questions, we present the novel
DropKey method that regards Key as the drop unit and exploits decreasing
schedule for drop ratio, improving ViTs in a general way. Comprehensive
experiments demonstrate the effectiveness of DropKey for various ViT
architectures, e.g. T2T and VOLO, as well as for various vision tasks, e.g.,
image classification, object detection, human-object interaction detection and
human body shape recovery.Comment: Accepted by CVPR202