5 research outputs found
Open-Set Likelihood Maximization for Few-Shot Learning
We tackle the Few-Shot Open-Set Recognition (FSOSR) problem, i.e. classifying
instances among a set of classes for which we only have a few labeled samples,
while simultaneously detecting instances that do not belong to any known class.
We explore the popular transductive setting, which leverages the unlabelled
query instances at inference. Motivated by the observation that existing
transductive methods perform poorly in open-set scenarios, we propose a
generalization of the maximum likelihood principle, in which latent scores
down-weighing the influence of potential outliers are introduced alongside the
usual parametric model. Our formulation embeds supervision constraints from the
support set and additional penalties discouraging overconfident predictions on
the query set. We proceed with a block-coordinate descent, with the latent
scores and parametric model co-optimized alternately, thereby benefiting from
each other. We call our resulting formulation \textit{Open-Set Likelihood
Optimization} (OSLO). OSLO is interpretable and fully modular; it can be
applied on top of any pre-trained model seamlessly. Through extensive
experiments, we show that our method surpasses existing inductive and
transductive methods on both aspects of open-set recognition, namely inlier
classification and outlier detection.Comment: arXiv admin note: substantial text overlap with arXiv:2206.0923
Verification of high-level transformations with inductive refinement types
International audienceHigh-level transformation languages like Rascal include expressive features for manipulating large abstract syntax trees: first-class traversals, expressive pattern matching, backtrack-ing and generalized iterators. We present the design and implementation of an abstract interpretation tool, Rabit, for verifying inductive type and shape properties for transformations written in such languages. We describe how to perform abstract interpretation based on operational semantics, specifically focusing on the challenges arising when analyzing the expressive traversals and pattern matching. Finally, we evaluate Rabit on a series of transformations (normaliza-tion, desugaring, refactoring, code generators, type inference, etc.) showing that we can effectively verify stated properties. CCS Concepts • Software and its engineering → General programming languages; • Social and professional topics → History of programming languages
Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling
Every day, a new method is published to tackle Few-Shot Image Classification,
showing better and better performances on academic benchmarks. Nevertheless, we
observe that these current benchmarks do not accurately represent the real
industrial use cases that we encountered. In this work, through both
qualitative and quantitative studies, we expose that the widely used benchmark
tieredImageNet is strongly biased towards tasks composed of very semantically
dissimilar classes e.g. bathtub, cabbage, pizza, schipperke, and cardoon. This
makes tieredImageNet (and similar benchmarks) irrelevant to evaluate the
ability of a model to solve real-life use cases usually involving more
fine-grained classification. We mitigate this bias using semantic information
about the classes of tieredImageNet and generate an improved, balanced
benchmark. Going further, we also introduce a new benchmark for Few-Shot Image
Classification using the Danish Fungi 2020 dataset. This benchmark proposes a
wide variety of evaluation tasks with various fine-graininess. Moreover, this
benchmark includes many-way tasks (e.g. composed of 100 classes), which is a
challenging setting yet very common in industrial applications. Our experiments
bring out the correlation between the difficulty of a task and the semantic
similarity between its classes, as well as a heavy performance drop of
state-of-the-art methods on many-way few-shot classification, raising questions
about the scaling abilities of these methods. We hope that our work will
encourage the community to further question the quality of standard evaluation
processes and their relevance to real-life applications.Comment: CVPR 2022 Workshop on Vision Datasets Understandin
Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift
Few-Shot Learning (FSL) algorithms have made substantial progress in learning novel concepts with just a handful of labelled data. To classify query instances from novel classes encountered at test-time, they only require a support set composed of a few labelled samples. FSL benchmarks commonly assume that those queries come from the same distribution as instances in the support set. However, in a realistic setting, data distribution is plausibly subject to change, a situation referred to as Distribution Shift (DS). The present work addresses the new and challenging problem of Few-Shot Learning under Support/Query Shift (FSQS) i.e., when support and query instances are sampled from related but different distributions. Our contributions are the following. First, we release a testbed for FSQS, including datasets, relevant baselines and a protocol for a rigorous and reproducible evaluation. Second, we observe that well-established FSL algorithms unsurprisingly suffer from a considerable drop in accuracy when facing FSQS, stressing the significance of our study. Finally, we show that transductive algorithms can limit the inopportune effect of DS. In particular, we study both the role of Batch-Normalization and Optimal Transport (OT) in aligning distributions, bridging Unsupervised Domain Adaptation with FSL. This results in a new method that efficiently combines OT with the celebrated Prototypical Networks. We bring compelling experiments demonstrating the advantage of our method. Our work opens an exciting line of research by providing a testbed and strong baselines. Our code is available at https://github.com/ebennequin/meta-domain-shift