50 research outputs found
Minimizing Supervision in Multi-label Categorization
Multiple categories of objects are present in most images. Treating this as a
multi-class classification is not justified. We treat this as a multi-label
classification problem. In this paper, we further aim to minimize the
supervision required for providing supervision in multi-label classification.
Specifically, we investigate an effective class of approaches that associate a
weak localization with each category either in terms of the bounding box or
segmentation mask. Doing so improves the accuracy of multi-label
categorization. The approach we adopt is one of active learning, i.e.,
incrementally selecting a set of samples that need supervision based on the
current model, obtaining supervision for these samples, retraining the model
with the additional set of supervised samples and proceeding again to select
the next set of samples. A crucial concern is the choice of the set of samples.
In doing so, we provide a novel insight, and no specific measure succeeds in
obtaining a consistently improved selection criterion. We, therefore, provide a
selection criterion that consistently improves the overall baseline criterion
by choosing the top k set of samples for a varied set of criteria. Using this
criterion, we are able to show that we can retain more than 98% of the fully
supervised performance with just 20% of samples (and more than 96% using 10%)
of the dataset on PASCAL VOC 2007 and 2012. Also, our proposed approach
consistently outperforms all other baseline metrics for all benchmark datasets
and model combinations.Comment: Accepted in CVPR-W 202
Active-Learning-as-a-Service: An Efficient MLOps System for Data-Centric AI
The success of today's AI applications requires not only model training
(Model-centric) but also data engineering (Data-centric). In data-centric AI,
active learning (AL) plays a vital role, but current AL tools can not perform
AL tasks efficiently. To this end, this paper presents an efficient MLOps
system for AL, named ALaaS (Active-Learning-as-a-Service). Specifically, ALaaS
adopts a server-client architecture to support an AL pipeline and implements
stage-level parallelism for high efficiency. Meanwhile, caching and batching
techniques are employed to further accelerate the AL process. In addition to
efficiency, ALaaS ensures accessibility with the help of the design philosophy
of configuration-as-a-service. It also abstracts an AL process to several
components and provides rich APIs for advanced users to extend the system to
new scenarios. Extensive experiments show that ALaaS outperforms all other
baselines in terms of latency and throughput. Further ablation studies
demonstrate the effectiveness of our design as well as ALaaS's ease to use. Our
code is available at \url{https://github.com/MLSysOps/alaas}.Comment: 8 pages, 7 figure
CLINICAL: Targeted Active Learning for Imbalanced Medical Image Classification
Training deep learning models on medical datasets that perform well for all
classes is a challenging task. It is often the case that a suboptimal
performance is obtained on some classes due to the natural class imbalance
issue that comes with medical data. An effective way to tackle this problem is
by using targeted active learning, where we iteratively add data points to the
training data that belong to the rare classes. However, existing active
learning methods are ineffective in targeting rare classes in medical datasets.
In this work, we propose Clinical (targeted aCtive Learning for ImbalaNced
medICal imAge cLassification) a framework that uses submodular mutual
information functions as acquisition functions to mine critical data points
from rare classes. We apply our framework to a wide-array of medical imaging
datasets on a variety of real-world class imbalance scenarios - namely, binary
imbalance and long-tail imbalance. We show that Clinical outperforms the
state-of-the-art active learning methods by acquiring a diverse set of data
points that belong to the rare classes.Comment: Accepted to MICCAI 2022 MILLanD Worksho
Active Learning for Fine-Grained Sketch-Based Image Retrieval
The ability to retrieve a photo by mere free-hand sketching highlights the
immense potential of Fine-grained sketch-based image retrieval (FG-SBIR).
However, its rapid practical adoption, as well as scalability, is limited by
the expense of acquiring faithful sketches for easily available photo
counterparts. A solution to this problem is Active Learning, which could
minimise the need for labeled sketches while maximising performance. Despite
extensive studies in the field, there exists no work that utilises it for
reducing sketching effort in FG-SBIR tasks. To this end, we propose a novel
active learning sampling technique that drastically minimises the need for
drawing photo sketches. Our proposed approach tackles the trade-off between
uncertainty and diversity by utilising the relationship between the existing
photo-sketch pair to a photo that does not have its sketch and augmenting this
relation with its intermediate representations. Since our approach relies only
on the underlying data distribution, it is agnostic of the modelling approach
and hence is applicable to other cross-modal instance-level retrieval tasks as
well. With experimentation over two publicly available fine-grained SBIR
datasets ChairV2 and ShoeV2, we validate our approach and reveal its
superiority over adapted baselines.Comment: Accepted at BMVC 202
Towards Comparable Active Learning
Active Learning has received significant attention in the field of machine
learning for its potential in selecting the most informative samples for
labeling, thereby reducing data annotation costs. However, we show that the
reported lifts in recent literature generalize poorly to other domains leading
to an inconclusive landscape in Active Learning research. Furthermore, we
highlight overlooked problems for reproducing AL experiments that can lead to
unfair comparisons and increased variance in the results. This paper addresses
these issues by providing an Active Learning framework for a fair comparison of
algorithms across different tasks and domains, as well as a fast and performant
oracle algorithm for evaluation. To the best of our knowledge, we propose the
first AL benchmark that tests algorithms in 3 major domains: Tabular, Image,
and Text. We report empirical results for 6 widely used algorithms on 7
real-world and 2 synthetic datasets and aggregate them into a domain-specific
ranking of AL algorithms