15 research outputs found
Benchmarking Low-Shot Robustness to Natural Distribution Shifts
Robustness to natural distribution shifts has seen remarkable progress thanks
to recent pre-training strategies combined with better fine-tuning methods.
However, such fine-tuning assumes access to large amounts of labelled data, and
the extent to which the observations hold when the amount of training data is
not as high remains unknown. We address this gap by performing the first
in-depth study of robustness to various natural distribution shifts in
different low-shot regimes: spanning datasets, architectures, pre-trained
initializations, and state-of-the-art robustness interventions. Most
importantly, we find that there is no single model of choice that is often more
robust than others, and existing interventions can fail to improve robustness
on some datasets even if they do so in the full-shot regime. We hope that our
work will motivate the community to focus on this problem of practical
importance.Comment: 22 Pages, 18 Tables, 12 Figure
PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization
Synthetic data offers the promise of cheap and bountiful training data for
settings where lots of labeled real-world data for tasks is unavailable.
However, models trained on synthetic data significantly underperform on
real-world data. In this paper, we propose Proportional Amplitude Spectrum
Training Augmentation (PASTA), a simple and effective augmentation strategy to
improve out-of-the-box synthetic-to-real (syn-to-real) generalization
performance. PASTA involves perturbing the amplitude spectrums of the synthetic
images in the Fourier domain to generate augmented views. We design PASTA to
perturb the amplitude spectrums in a structured manner such that high-frequency
components are perturbed relatively more than the low-frequency ones. For the
tasks of semantic segmentation (GTAV to Real), object detection (Sim10K to
Real), and object recognition (VisDA-C Syn to Real), across a total of 5
syn-to-real shifts, we find that PASTA outperforms more complex
state-of-the-art generalization methods while being complementary to the same.Comment: Code: https://github.com/prithv1/PAST
LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images
We propose an automated algorithm to stress-test a trained visual model by
generating language-guided counterfactual test images (LANCE). Our method
leverages recent progress in large language modeling and text-based image
editing to augment an IID test set with a suite of diverse, realistic, and
challenging test images without altering model weights. We benchmark the
performance of a diverse set of pretrained models on our generated data and
observe significant and consistent performance drops. We further analyze model
sensitivity across different types of edits, and demonstrate its applicability
at surfacing previously unknown class-level model biases in ImageNet.Comment: Project webpage: https://virajprabhu.github.io/lance-web
Evaluating Visual Conversational Agents via Cooperative Human-AI Games
As AI continues to advance, human-AI teams are inevitable. However, progress
in AI is routinely measured in isolation, without a human in the loop. It is
crucial to benchmark progress in AI, not just in isolation, but also in terms
of how it translates to helping humans perform certain tasks, i.e., the
performance of human-AI teams.
In this work, we design a cooperative game - GuessWhich - to measure human-AI
team performance in the specific context of the AI being a visual
conversational agent. GuessWhich involves live interaction between the human
and the AI. The AI, which we call ALICE, is provided an image which is unseen
by the human. Following a brief description of the image, the human questions
ALICE about this secret image to identify it from a fixed pool of images.
We measure performance of the human-ALICE team by the number of guesses it
takes the human to correctly identify the secret image after a fixed number of
dialog rounds with ALICE. We compare performance of the human-ALICE teams for
two versions of ALICE. Our human studies suggest a counterintuitive trend -
that while AI literature shows that one version outperforms the other when
paired with an AI questioner bot, we find that this improvement in AI-AI
performance does not translate to improved human-AI performance. This suggests
a mismatch between benchmarking of AI in isolation and in the context of
human-AI teams.Comment: HCOMP 201
SkyScenes: A Synthetic Dataset for Aerial Scene Understanding
Real-world aerial scene understanding is limited by a lack of datasets that
contain densely annotated images curated under a diverse set of conditions. Due
to inherent challenges in obtaining such images in controlled real-world
settings, we present SkyScenes, a synthetic dataset of densely annotated aerial
images captured from Unmanned Aerial Vehicle (UAV) perspectives. We carefully
curate SkyScenes images from CARLA to comprehensively capture diversity across
layout (urban and rural maps), weather conditions, times of day, pitch angles
and altitudes with corresponding semantic, instance and depth annotations.
Through our experiments using SkyScenes, we show that (1) Models trained on
SkyScenes generalize well to different real-world scenarios, (2) augmenting
training on real images with SkyScenes data can improve real-world performance,
(3) controlled variations in SkyScenes can offer insights into how models
respond to changes in viewpoint conditions, and (4) incorporating additional
sensor modalities (depth) can improve aerial scene understanding