42 research outputs found
SC-MAD: Mixtures of Higher-order Networks for Data Augmentation
The myriad complex systems with multiway interactions motivate the extension
of graph-based pairwise connections to higher-order relations. In particular,
the simplicial complex has inspired generalizations of graph neural networks
(GNNs) to simplicial complex-based models. Learning on such systems requires
large amounts of data, which can be expensive or impossible to obtain. We
propose data augmentation of simplicial complexes through both linear and
nonlinear mixup mechanisms that return mixtures of existing labeled samples. In
addition to traditional pairwise mixup, we present a convex clustering mixup
approach for a data-driven relationship among several simplicial complexes. We
theoretically demonstrate that the resultant synthetic simplicial complexes
interpolate among existing data with respect to homomorphism densities. Our
method is demonstrated on both synthetic and real-world datasets for simplicial
complex classification.Comment: 5 pages, 1 figure, 1 tabl
Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?
For more than a decade, researchers have measured progress in object
recognition on ImageNet-based generalization benchmarks such as ImageNet-A, -C,
and -R. Recent advances in foundation models, trained on orders of magnitude
more data, have begun to saturate these standard benchmarks, but remain brittle
in practice. This suggests standard benchmarks, which tend to focus on
predefined or synthetic changes, may not be sufficient for measuring real world
generalization. Consequently, we propose studying generalization across
geography as a more realistic measure of progress using two datasets of objects
from households across the globe. We conduct an extensive empirical evaluation
of progress across nearly 100 vision models up to most recent foundation
models. We first identify a progress gap between standard benchmarks and
real-world, geographical shifts: progress on ImageNet results in up to 2.5x
more progress on standard generalization benchmarks than real-world
distribution shifts. Second, we study model generalization across geographies
by measuring the disparities in performance across regions, a more fine-grained
measure of real world generalization. We observe all models have large
geographic disparities, even foundation CLIP models, with differences of 7-20%
in accuracy between regions. Counter to modern intuition, we discover progress
on standard benchmarks fails to improve geographic disparities and often
exacerbates them: geographic disparities between the least performant models
and today's best models have more than tripled. Our results suggest scaling
alone is insufficient for consistent robustness to real-world distribution
shifts. Finally, we highlight in early experiments how simple last layer
retraining on more representative, curated data can complement scaling as a
promising direction of future work, reducing geographic disparity on both
benchmarks by over two-thirds
Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
Prior works have found it beneficial to combine provably noise-robust loss
functions e.g., mean absolute error (MAE) with standard categorical loss
function e.g. cross entropy (CE) to improve their learnability. Here, we
propose to use Jensen-Shannon divergence as a noise-robust loss function and
show that it interestingly interpolate between CE and MAE with a controllable
mixing parameter. Furthermore, we make a crucial observation that CE exhibit
lower consistency around noisy data points. Based on this observation, we adopt
a generalized version of the Jensen-Shannon divergence for multiple
distributions to encourage consistency around data points. Using this loss
function, we show state-of-the-art results on both synthetic (CIFAR), and
real-world (e.g., WebVision) noise with varying noise rates.Comment: Neural Information Processing Systems (NeurIPS 2021
Domain Generalization in Machine Learning Models for Wireless Communications: Concepts, State-of-the-Art, and Open Issues
Data-driven machine learning (ML) is promoted as one potential technology to
be used in next-generations wireless systems. This led to a large body of
research work that applies ML techniques to solve problems in different layers
of the wireless transmission link. However, most of these applications rely on
supervised learning which assumes that the source (training) and target (test)
data are independent and identically distributed (i.i.d). This assumption is
often violated in the real world due to domain or distribution shifts between
the source and the target data. Thus, it is important to ensure that these
algorithms generalize to out-of-distribution (OOD) data. In this context,
domain generalization (DG) tackles the OOD-related issues by learning models on
different and distinct source domains/datasets with generalization capabilities
to unseen new domains without additional finetuning. Motivated by the
importance of DG requirements for wireless applications, we present a
comprehensive overview of the recent developments in DG and the different
sources of domain shift. We also summarize the existing DG methods and review
their applications in selected wireless communication problems, and conclude
with insights and open questions
Adaptive Contextual Perception: How to Generalize to New Backgrounds and Ambiguous Objects
Biological vision systems make adaptive use of context to recognize objects
in new settings with novel contexts as well as occluded or blurry objects in
familiar settings. In this paper, we investigate how vision models adaptively
use context for out-of-distribution (OOD) generalization and leverage our
analysis results to improve model OOD generalization. First, we formulate two
distinct OOD settings where the contexts are either irrelevant
(Background-Invariance) or beneficial (Object-Disambiguation), reflecting the
diverse contextual challenges faced in biological vision. We then analyze model
performance in these two different OOD settings and demonstrate that models
that excel in one setting tend to struggle in the other. Notably, prior works
on learning causal features improve on one setting but hurt in the other. This
underscores the importance of generalizing across both OOD settings, as this
ability is crucial for both human cognition and robust AI systems. Next, to
better understand the model properties contributing to OOD generalization, we
use representational geometry analysis and our own probing methods to examine a
population of models, and we discover that those with more factorized
representations and appropriate feature weighting are more successful in
handling Background-Invariance and Object-Disambiguation tests. We further
validate these findings through causal intervention on representation
factorization and feature weighting to demonstrate their causal effect on
performance. Lastly, we propose new augmentation methods to enhance model
generalization. These methods outperform strong baselines, yielding
improvements in both in-distribution and OOD tests. In conclusion, to replicate
the generalization abilities of biological vision, computer vision models must
have factorized object vs. background representations and appropriately weight
both kinds of features.Comment: 21 pages, 12 figures. Our code is available at
https://github.com/zfying/AdaptiveContex
Towards Generalizable Deepfake Detection by Primary Region Regularization
The existing deepfake detection methods have reached a bottleneck in
generalizing to unseen forgeries and manipulation approaches. Based on the
observation that the deepfake detectors exhibit a preference for overfitting
the specific primary regions in input, this paper enhances the generalization
capability from a novel regularization perspective. This can be simply achieved
by augmenting the images through primary region removal, thereby preventing the
detector from over-relying on data bias. Our method consists of two stages,
namely the static localization for primary region maps, as well as the dynamic
exploitation of primary region masks. The proposed method can be seamlessly
integrated into different backbones without affecting their inference
efficiency. We conduct extensive experiments over three widely used deepfake
datasets - DFDC, DF-1.0, and Celeb-DF with five backbones. Our method
demonstrates an average performance improvement of 6% across different
backbones and performs competitively with several state-of-the-art baselines.Comment: 12 pages. Code and Dataset: https://github.com/xaCheng1996/PRL
Domain Generalization in Computational Pathology: Survey and Guidelines
Deep learning models have exhibited exceptional effectiveness in
Computational Pathology (CPath) by tackling intricate tasks across an array of
histology image analysis applications. Nevertheless, the presence of
out-of-distribution data (stemming from a multitude of sources such as
disparate imaging devices and diverse tissue preparation methods) can cause
\emph{domain shift} (DS). DS decreases the generalization of trained models to
unseen datasets with slightly different data distributions, prompting the need
for innovative \emph{domain generalization} (DG) solutions. Recognizing the
potential of DG methods to significantly influence diagnostic and prognostic
models in cancer studies and clinical practice, we present this survey along
with guidelines on achieving DG in CPath. We rigorously define various DS
types, systematically review and categorize existing DG approaches and
resources in CPath, and provide insights into their advantages, limitations,
and applicability. We also conduct thorough benchmarking experiments with 28
cutting-edge DG algorithms to address a complex DG problem. Our findings
suggest that careful experiment design and CPath-specific Stain Augmentation
technique can be very effective. However, there is no one-size-fits-all
solution for DG in CPath. Therefore, we establish clear guidelines for
detecting and managing DS depending on different scenarios. While most of the
concepts, guidelines, and recommendations are given for applications in CPath,
we believe that they are applicable to most medical image analysis tasks as
well.Comment: Extended Versio
Meta-Learning in Neural Networks: A Survey
The field of meta-learning, or learning-to-learn, has seen a dramatic rise in
interest in recent years. Contrary to conventional approaches to AI where tasks
are solved from scratch using a fixed learning algorithm, meta-learning aims to
improve the learning algorithm itself, given the experience of multiple
learning episodes. This paradigm provides an opportunity to tackle many
conventional challenges of deep learning, including data and computation
bottlenecks, as well as generalization. This survey describes the contemporary
meta-learning landscape. We first discuss definitions of meta-learning and
position it with respect to related fields, such as transfer learning and
hyperparameter optimization. We then propose a new taxonomy that provides a
more comprehensive breakdown of the space of meta-learning methods today. We
survey promising applications and successes of meta-learning such as few-shot
learning and reinforcement learning. Finally, we discuss outstanding challenges
and promising areas for future research