92 research outputs found
Learn to Propagate Reliably on Noisy Affinity Graphs
Recent works have shown that exploiting unlabeled data through label
propagation can substantially reduce the labeling cost, which has been a
critical issue in developing visual recognition models. Yet, how to propagate
labels reliably, especially on a dataset with unknown outliers, remains an open
question. Conventional methods such as linear diffusion lack the capability of
handling complex graph structures and may perform poorly when the seeds are
sparse. Latest methods based on graph neural networks would face difficulties
on performance drop as they scale out to noisy graphs. To overcome these
difficulties, we propose a new framework that allows labels to be propagated
reliably on large-scale real-world data. This framework incorporates (1) a
local graph neural network to predict accurately on varying local structures
while maintaining high scalability, and (2) a confidence-based path scheduler
that identifies outliers and moves forward the propagation frontier in a
prudent way. Experiments on both ImageNet and Ms-Celeb-1M show that our
confidence guided framework can significantly improve the overall accuracies of
the propagated labels, especially when the graph is very noisy.Comment: 14 pages, 7 figures, ECCV 202
Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation
In this paper, we propose a simple yet effective approach for self-supervised
video object segmentation (VOS). Our key insight is that the inherent
structural dependencies present in DINO-pretrained Transformers can be
leveraged to establish robust spatio-temporal correspondences in videos.
Furthermore, simple clustering on this correspondence cue is sufficient to
yield competitive segmentation results. Previous self-supervised VOS techniques
majorly resort to auxiliary modalities or utilize iterative slot attention to
assist in object discovery, which restricts their general applicability and
imposes higher computational requirements. To deal with these challenges, we
develop a simplified architecture that capitalizes on the emerging objectness
from DINO-pretrained Transformers, bypassing the need for additional modalities
or slot attention. Specifically, we first introduce a single spatio-temporal
Transformer block to process the frame-wise DINO features and establish
spatio-temporal dependencies in the form of self-attention. Subsequently,
utilizing these attention maps, we implement hierarchical clustering to
generate object segmentation masks. To train the spatio-temporal block in a
fully self-supervised manner, we employ semantic and dynamic motion consistency
coupled with entropy normalization. Our method demonstrates state-of-the-art
performance across multiple unsupervised VOS benchmarks and particularly excels
in complex real-world multi-object video segmentation tasks such as
DAVIS-17-Unsupervised and YouTube-VIS-19. The code and model checkpoints will
be released at https://github.com/shvdiwnkozbw/SSL-UVOS
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
Recent advances in adversarial attacks uncover the intrinsic vulnerability of
modern deep neural networks. Since then, extensive efforts have been devoted to
enhancing the robustness of deep networks via specialized learning algorithms
and loss functions. In this work, we take an architectural perspective and
investigate the patterns of network architectures that are resilient to
adversarial attacks. To obtain the large number of networks needed for this
study, we adopt one-shot neural architecture search, training a large network
for once and then finetuning the sub-networks sampled therefrom. The sampled
architectures together with the accuracies they achieve provide a rich basis
for our study. Our "robust architecture Odyssey" reveals several valuable
observations: 1) densely connected patterns result in improved robustness; 2)
under computational budget, adding convolution operations to direct connection
edge is effective; 3) flow of solution procedure (FSP) matrix is a good
indicator of network robustness. Based on these observations, we discover a
family of robust architectures (RobNets). On various datasets, including CIFAR,
SVHN, Tiny-ImageNet, and ImageNet, RobNets exhibit superior robustness
performance to other widely used architectures. Notably, RobNets substantially
improve the robust accuracy (~5% absolute gains) under both white-box and
black-box attacks, even with fewer parameter numbers. Code is available at
https://github.com/gmh14/RobNets.Comment: CVPR 2020. First two authors contributed equall
PointLLM: Empowering Large Language Models to Understand Point Clouds
The unprecedented advancements in Large Language Models (LLMs) have created a
profound impact on natural language processing but are yet to fully embrace the
realm of 3D understanding. This paper introduces PointLLM, a preliminary effort
to fill this gap, thereby enabling LLMs to understand point clouds and offering
a new avenue beyond 2D visual data. PointLLM processes colored object point
clouds with human instructions and generates contextually appropriate
responses, illustrating its grasp of point clouds and common sense.
Specifically, it leverages a point cloud encoder with a powerful LLM to
effectively fuse geometric, appearance, and linguistic information. We collect
a novel dataset comprising 660K simple and 70K complex point-text instruction
pairs to enable a two-stage training strategy: initially aligning latent spaces
and subsequently instruction-tuning the unified model. To rigorously evaluate
our model's perceptual abilities and its generalization capabilities, we
establish two benchmarks: Generative 3D Object Classification and 3D Object
Captioning, assessed through three different methods, including human
evaluation, GPT-4/ChatGPT evaluation, and traditional metrics. Experiment
results show that PointLLM demonstrates superior performance over existing 2D
baselines. Remarkably, in human-evaluated object captioning tasks, PointLLM
outperforms human annotators in over 50% of the samples. Codes, datasets, and
benchmarks are available at https://github.com/OpenRobotLab/PointLLM .Comment: 19 pages. Empowering large language models with 3D point cloud
understanding, accompanied by a novel dataset and carefully designed
benchmarks. Project page: https://runsenxu.com/projects/PointLL
Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstruction
Neural surface reconstruction aims to reconstruct accurate 3D surfaces based
on multi-view images. Previous methods based on neural volume rendering mostly
train a fully implicit model with MLPs, which typically require hours of
training for a single scene. Recent efforts explore the explicit volumetric
representation to accelerate the optimization via memorizing significant
information with learnable voxel grids. However, existing voxel-based methods
often struggle in reconstructing fine-grained geometry, even when combined with
an SDF-based volume rendering scheme. We reveal that this is because 1) the
voxel grids tend to break the color-geometry dependency that facilitates
fine-geometry learning, and 2) the under-constrained voxel grids lack spatial
coherence and are vulnerable to local minima. In this work, we present Voxurf,
a voxel-based surface reconstruction approach that is both efficient and
accurate. Voxurf addresses the aforementioned issues via several key designs,
including 1) a two-stage training procedure that attains a coherent coarse
shape and recovers fine details successively, 2) a dual color network that
maintains color-geometry dependency, and 3) a hierarchical geometry feature to
encourage information propagation across voxels. Extensive experiments show
that Voxurf achieves high efficiency and high quality at the same time. On the
DTU benchmark, Voxurf achieves higher reconstruction quality with a 20x
training speedup compared to previous fully implicit methods
- …