130 research outputs found
Harvesting Information from Captions for Weakly Supervised Semantic Segmentation
Since acquiring pixel-wise annotations for training convolutional neural
networks for semantic image segmentation is time-consuming, weakly supervised
approaches that only require class tags have been proposed. In this work, we
propose another form of supervision, namely image captions as they can be found
on the Internet. These captions have two advantages. They do not require
additional curation as it is the case for the clean class tags used by current
weakly supervised approaches and they provide textual context for the classes
present in an image. To leverage such textual context, we deploy a multi-modal
network that learns a joint embedding of the visual representation of the image
and the textual representation of the caption. The network estimates text
activation maps (TAMs) for class names as well as compound concepts, i.e.
combinations of nouns and their attributes. The TAMs of compound concepts
describing classes of interest substantially improve the quality of the
estimated class activation maps which are then used to train a network for
semantic segmentation. We evaluate our method on the COCO dataset where it
achieves state of the art results for weakly supervised image segmentation
Complex Event Recognition from Images with Few Training Examples
We propose to leverage concept-level representations for complex event
recognition in photographs given limited training examples. We introduce a
novel framework to discover event concept attributes from the web and use that
to extract semantic features from images and classify them into social event
categories with few training examples. Discovered concepts include a variety of
objects, scenes, actions and event sub-types, leading to a discriminative and
compact representation for event images. Web images are obtained for each
discovered event concept and we use (pretrained) CNN features to train concept
classifiers. Extensive experiments on challenging event datasets demonstrate
that our proposed method outperforms several baselines using deep CNN features
directly in classifying images into events with limited training examples. We
also demonstrate that our method achieves the best overall accuracy on a
dataset with unseen event categories using a single training example.Comment: Accepted to Winter Applications of Computer Vision (WACV'17
Movie/Script: Alignment and Parsing of Video and Text Transcription
Movies and TV are a rich source of diverse and complex video of people, objects, actions and locales “in the wild”. Harvesting automatically labeled sequences of actions from video would enable creation of large-scale and highly-varied datasets. To enable such collection, we focus on the task of recovering scene structure in movies and TV series for object tracking and action retrieval. We present a weakly supervised algorithm that uses the screenplay and closed captions to parse a movie into a hierarchy of shots and scenes. Scene boundaries in the movie are aligned with screenplay scene labels and shots are reordered into a sequence of long continuous tracks or threads which allow for more accurate tracking of people, actions and objects. Scene segmentation, alignment, and shot threading are formulated as inference in a unified generative model and a novel hierarchical dynamic programming algorithm that can handle alignment and jump-limited reorderings in linear time is presented. We present quantitative and qualitative results on movie alignment and parsing, and use the recovered structure to improve character naming and retrieval of common actions in several episodes of popular TV series
Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser
Audio-visual learning has been a major pillar of multi-modal machine
learning, where the community mostly focused on its modality-aligned setting,
i.e., the audio and visual modality are both assumed to signal the prediction
target. With the Look, Listen, and Parse dataset (LLP), we investigate the
under-explored unaligned setting, where the goal is to recognize audio and
visual events in a video with only weak labels observed. Such weak video-level
labels only tell what events happen without knowing the modality they are
perceived (audio, visual, or both). To enhance learning in this challenging
setting, we incorporate large-scale contrastively pre-trained models as the
modality teachers. A simple, effective, and generic method, termed Visual-Audio
Label Elaboration (VALOR), is innovated to harvest modality labels for the
training events. Empirical studies show that the harvested labels significantly
improve an attentional baseline by 8.0 in average F-score (Type@AV).
Surprisingly, we found that modality-independent teachers outperform their
modality-fused counterparts since they are noise-proof from the other
potentially unaligned modality. Moreover, our best model achieves the new
state-of-the-art on all metrics of LLP by a substantial margin (+5.4 F-score
for Type@AV). VALOR is further generalized to Audio-Visual Event Localization
and achieves the new state-of-the-art as well. Code is available at:
https://github.com/Franklin905/VALOR
Text-Only Training for Image Captioning using Noise-Injected CLIP
We consider the task of image-captioning using only the CLIP model and
additional text data at training time, and no additional captioned images. Our
approach relies on the fact that CLIP is trained to make visual and textual
embeddings similar. Therefore, we only need to learn how to translate CLIP
textual embeddings back into text, and we can learn how to do this by learning
a decoder for the frozen CLIP text encoder using only text. We argue that this
intuition is "almost correct" because of a gap between the embedding spaces,
and propose to rectify this via noise injection during training. We demonstrate
the effectiveness of our approach by showing SOTA zero-shot image captioning
across four benchmarks, including style transfer. Code, data, and models are
available on GitHub.Comment: Will be presented at EMNLP 2022. GitHub:
https://github.com/DavidHuji/CapDe
Foundation Models in Smart Agriculture: Basics, Opportunities, and Challenges
The past decade has witnessed the rapid development of ML and DL
methodologies in agricultural systems, showcased by great successes in variety
of agricultural applications. However, these conventional ML/DL models have
certain limitations: They heavily rely on large, costly-to-acquire labeled
datasets for training, require specialized expertise for development and
maintenance, and are mostly tailored for specific tasks, thus lacking
generalizability. Recently, foundation models have demonstrated remarkable
successes in language and vision tasks across various domains. These models are
trained on a vast amount of data from multiple domains and modalities. Once
trained, they can accomplish versatile tasks with just minor fine-tuning and
minimal task-specific labeled data. Despite their proven effectiveness and huge
potential, there has been little exploration of applying FMs to agriculture
fields. Therefore, this study aims to explore the potential of FMs in the field
of smart agriculture. In particular, we present conceptual tools and technical
background to facilitate the understanding of the problem space and uncover new
research directions in this field. To this end, we first review recent FMs in
the general computer science domain and categorize them into four categories:
language FMs, vision FMs, multimodal FMs, and reinforcement learning FMs.
Subsequently, we outline the process of developing agriculture FMs and discuss
their potential applications in smart agriculture. We also discuss the unique
challenges associated with developing AFMs, including model training,
validation, and deployment. Through this study, we contribute to the
advancement of AI in agriculture by introducing AFMs as a promising paradigm
that can significantly mitigate the reliance on extensive labeled datasets and
enhance the efficiency, effectiveness, and generalization of agricultural AI
systems.Comment: 16 pages, 2 figure
Fine Art Pattern Extraction and Recognition
This is a reprint of articles from the Special Issue published online in the open access journal Journal of Imaging (ISSN 2313-433X) (available at: https://www.mdpi.com/journal/jimaging/special issues/faper2020)
Visual Question Answering: A Survey of Methods and Datasets
Visual Question Answering (VQA) is a challenging task that has received
increasing attention from both the computer vision and the natural language
processing communities. Given an image and a question in natural language, it
requires reasoning over visual elements of the image and general knowledge to
infer the correct answer. In the first part of this survey, we examine the
state of the art by comparing modern approaches to the problem. We classify
methods by their mechanism to connect the visual and textual modalities. In
particular, we examine the common approach of combining convolutional and
recurrent neural networks to map images and questions to a common feature
space. We also discuss memory-augmented and modular architectures that
interface with structured knowledge bases. In the second part of this survey,
we review the datasets available for training and evaluating VQA systems. The
various datatsets contain questions at different levels of complexity, which
require different capabilities and types of reasoning. We examine in depth the
question/answer pairs from the Visual Genome project, and evaluate the
relevance of the structured annotations of images with scene graphs for VQA.
Finally, we discuss promising future directions for the field, in particular
the connection to structured knowledge bases and the use of natural language
processing models.Comment: 25 page
- …