44,791 research outputs found
Neural approaches to spoken content embedding
Comparing spoken segments is a central operation to speech processing.
Traditional approaches in this area have favored frame-level dynamic
programming algorithms, such as dynamic time warping, because they require no
supervision, but they are limited in performance and efficiency. As an
alternative, acoustic word embeddings -- fixed-dimensional vector
representations of variable-length spoken word segments -- have begun to be
considered for such tasks as well. However, the current space of such
discriminative embedding models, training approaches, and their application to
real-world downstream tasks is limited. We start by considering ``single-view"
training losses where the goal is to learn an acoustic word embedding model
that separates same-word and different-word spoken segment pairs. Then, we
consider ``multi-view" contrastive losses. In this setting, acoustic word
embeddings are learned jointly with embeddings of character sequences to
generate acoustically grounded embeddings of written words, or acoustically
grounded word embeddings.
In this thesis, we contribute new discriminative acoustic word embedding
(AWE) and acoustically grounded word embedding (AGWE) approaches based on
recurrent neural networks (RNNs). We improve model training in terms of both
efficiency and performance. We take these developments beyond English to
several low-resource languages and show that multilingual training improves
performance when labeled data is limited. We apply our embedding models, both
monolingual and multilingual, to the downstream tasks of query-by-example
speech search and automatic speech recognition. Finally, we show how our
embedding approaches compare with and complement more recent self-supervised
speech models.Comment: PhD thesi
ATM: Action Temporality Modeling for Video Question Answering
Despite significant progress in video question answering (VideoQA), existing
methods fall short of questions that require causal/temporal reasoning across
frames. This can be attributed to imprecise motion representations. We
introduce Action Temporality Modeling (ATM) for temporality reasoning via
three-fold uniqueness: (1) rethinking the optical flow and realizing that
optical flow is effective in capturing the long horizon temporality reasoning;
(2) training the visual-text embedding by contrastive learning in an
action-centric manner, leading to better action representations in both vision
and text modalities; and (3) preventing the model from answering the question
given the shuffled video in the fine-tuning stage, to avoid spurious
correlation between appearance and motion and hence ensure faithful temporality
reasoning. In the experiments, we show that ATM outperforms previous approaches
in terms of the accuracy on multiple VideoQAs and exhibits better true
temporality reasoning ability
CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines
Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective.
The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines.
From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research
- …