169,105 research outputs found
MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment
The visual quality of point clouds has been greatly emphasized since the
ever-increasing 3D vision applications are expected to provide cost-effective
and high-quality experiences for users. Looking back on the development of
point cloud quality assessment (PCQA) methods, the visual quality is usually
evaluated by utilizing single-modal information, i.e., either extracted from
the 2D projections or 3D point cloud. The 2D projections contain rich texture
and semantic information but are highly dependent on viewpoints, while the 3D
point clouds are more sensitive to geometry distortions and invariant to
viewpoints. Therefore, to leverage the advantages of both point cloud and
projected image modalities, we propose a novel no-reference point cloud quality
assessment (NR-PCQA) metric in a multi-modal fashion. In specific, we split the
point clouds into sub-models to represent local geometry distortions such as
point shift and down-sampling. Then we render the point clouds into 2D image
projections for texture feature extraction. To achieve the goals, the
sub-models and projected images are encoded with point-based and image-based
neural networks. Finally, symmetric cross-modal attention is employed to fuse
multi-modal quality-aware information. Experimental results show that our
approach outperforms all compared state-of-the-art methods and is far ahead of
previous NR-PCQA methods, which highlights the effectiveness of the proposed
method. The code is available at https://github.com/zzc-1998/MM-PCQA
Data-Efficient Image Quality Assessment with Attention-Panel Decoder
Blind Image Quality Assessment (BIQA) is a fundamental task in computer
vision, which however remains unresolved due to the complex distortion
conditions and diversified image contents. To confront this challenge, we in
this paper propose a novel BIQA pipeline based on the Transformer architecture,
which achieves an efficient quality-aware feature representation with much
fewer data. More specifically, we consider the traditional fine-tuning in BIQA
as an interpretation of the pre-trained model. In this way, we further
introduce a Transformer decoder to refine the perceptual information of the CLS
token from different perspectives. This enables our model to establish the
quality-aware feature manifold efficiently while attaining a strong
generalization capability. Meanwhile, inspired by the subjective evaluation
behaviors of human, we introduce a novel attention panel mechanism, which
improves the model performance and reduces the prediction uncertainty
simultaneously. The proposed BIQA method maintains a lightweight design with
only one layer of the decoder, yet extensive experiments on eight standard BIQA
datasets (both synthetic and authentic) demonstrate its superior performance to
the state-of-the-art BIQA methods, i.e., achieving the SRCC values of 0.875
(vs. 0.859 in LIVEC) and 0.980 (vs. 0.969 in LIVE).Comment: Accepted by AAAI 202
How is Gaze Influenced by Image Transformations? Dataset and Model
Data size is the bottleneck for developing deep saliency models, because
collecting eye-movement data is very time consuming and expensive. Most of
current studies on human attention and saliency modeling have used high quality
stereotype stimuli. In real world, however, captured images undergo various
types of transformations. Can we use these transformations to augment existing
saliency datasets? Here, we first create a novel saliency dataset including
fixations of 10 observers over 1900 images degraded by 19 types of
transformations. Second, by analyzing eye movements, we find that observers
look at different locations over transformed versus original images. Third, we
utilize the new data over transformed images, called data augmentation
transformation (DAT), to train deep saliency models. We find that label
preserving DATs with negligible impact on human gaze boost saliency prediction,
whereas some other DATs that severely impact human gaze degrade the
performance. These label preserving valid augmentation transformations provide
a solution to enlarge existing saliency datasets. Finally, we introduce a novel
saliency model based on generative adversarial network (dubbed GazeGAN). A
modified UNet is proposed as the generator of the GazeGAN, which combines
classic skip connections with a novel center-surround connection (CSC), in
order to leverage multi level features. We also propose a histogram loss based
on Alternative Chi Square Distance (ACS HistLoss) to refine the saliency map in
terms of luminance distribution. Extensive experiments and comparisons over 3
datasets indicate that GazeGAN achieves the best performance in terms of
popular saliency evaluation metrics, and is more robust to various
perturbations. Our code and data are available at:
https://github.com/CZHQuality/Sal-CFS-GAN
Visual Comfort Assessment for Stereoscopic Image Retargeting
In recent years, visual comfort assessment (VCA) for 3D/stereoscopic content
has aroused extensive attention. However, much less work has been done on the
perceptual evaluation of stereoscopic image retargeting. In this paper, we
first build a Stereoscopic Image Retargeting Database (SIRD), which contains
source images and retargeted images produced by four typical stereoscopic
retargeting methods. Then, the subjective experiment is conducted to assess
four aspects of visual distortion, i.e. visual comfort, image quality, depth
quality and the overall quality. Furthermore, we propose a Visual Comfort
Assessment metric for Stereoscopic Image Retargeting (VCA-SIR). Based on the
characteristics of stereoscopic retargeted images, the proposed model
introduces novel features like disparity range, boundary disparity as well as
disparity intensity distribution into the assessment model. Experimental
results demonstrate that VCA-SIR can achieve high consistency with subjective
perception
Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
We devise a cascade GAN approach to generate talking face video, which is
robust to different face shapes, view angles, facial characteristics, and noisy
audio conditions. Instead of learning a direct mapping from audio to video
frames, we propose first to transfer audio to high-level structure, i.e., the
facial landmarks, and then to generate video frames conditioned on the
landmarks. Compared to a direct audio-to-image approach, our cascade approach
avoids fitting spurious correlations between audiovisual signals that are
irrelevant to the speech content. We, humans, are sensitive to temporal
discontinuities and subtle artifacts in video. To avoid those pixel jittering
problems and to enforce the network to focus on audiovisual-correlated regions,
we propose a novel dynamically adjustable pixel-wise loss with an attention
mechanism. Furthermore, to generate a sharper image with well-synchronized
facial movements, we propose a novel regression-based discriminator structure,
which considers sequence-level information along with frame-level information.
Thoughtful experiments on several datasets and real-world samples demonstrate
significantly better results obtained by our method than the state-of-the-art
methods in both quantitative and qualitative comparisons
- …