17,610 research outputs found
Learning Robust Visual-Semantic Embedding for Generalizable Person Re-identification
Generalizable person re-identification (Re-ID) is a very hot research topic
in machine learning and computer vision, which plays a significant role in
realistic scenarios due to its various applications in public security and
video surveillance. However, previous methods mainly focus on the visual
representation learning, while neglect to explore the potential of semantic
features during training, which easily leads to poor generalization capability
when adapted to the new domain. In this paper, we propose a Multi-Modal
Equivalent Transformer called MMET for more robust visual-semantic embedding
learning on visual, textual and visual-textual tasks respectively. To further
enhance the robust feature learning in the context of transformer, a dynamic
masking mechanism called Masked Multimodal Modeling strategy (MMM) is
introduced to mask both the image patches and the text tokens, which can
jointly works on multimodal or unimodal data and significantly boost the
performance of generalizable person Re-ID. Extensive experiments on benchmark
datasets demonstrate the competitive performance of our method over previous
approaches. We hope this method could advance the research towards
visual-semantic representation learning. Our source code is also publicly
available at https://github.com/JeremyXSC/MMET
The Metaverse: Survey, Trends, Novel Pipeline Ecosystem & Future Directions
The Metaverse offers a second world beyond reality, where boundaries are
non-existent, and possibilities are endless through engagement and immersive
experiences using the virtual reality (VR) technology. Many disciplines can
benefit from the advancement of the Metaverse when accurately developed,
including the fields of technology, gaming, education, art, and culture.
Nevertheless, developing the Metaverse environment to its full potential is an
ambiguous task that needs proper guidance and directions. Existing surveys on
the Metaverse focus only on a specific aspect and discipline of the Metaverse
and lack a holistic view of the entire process. To this end, a more holistic,
multi-disciplinary, in-depth, and academic and industry-oriented review is
required to provide a thorough study of the Metaverse development pipeline. To
address these issues, we present in this survey a novel multi-layered pipeline
ecosystem composed of (1) the Metaverse computing, networking, communications
and hardware infrastructure, (2) environment digitization, and (3) user
interactions. For every layer, we discuss the components that detail the steps
of its development. Also, for each of these components, we examine the impact
of a set of enabling technologies and empowering domains (e.g., Artificial
Intelligence, Security & Privacy, Blockchain, Business, Ethics, and Social) on
its advancement. In addition, we explain the importance of these technologies
to support decentralization, interoperability, user experiences, interactions,
and monetization. Our presented study highlights the existing challenges for
each component, followed by research directions and potential solutions. To the
best of our knowledge, this survey is the most comprehensive and allows users,
scholars, and entrepreneurs to get an in-depth understanding of the Metaverse
ecosystem to find their opportunities and potentials for contribution
Sign Language Translation from Instructional Videos
The advances in automatic sign language translation (SLT) to spoken languages
have been mostly benchmarked with datasets of limited size and restricted
domains. Our work advances the state of the art by providing the first baseline
results on How2Sign, a large and broad dataset.
We train a Transformer over I3D video features, using the reduced BLEU as a
reference metric for validation, instead of the widely used BLEU score. We
report a result of 8.03 on the BLEU score, and publish the first open-source
implementation of its kind to promote further advances.Comment: Paper accepted at WiCV @CVPR2
Audio-Visual Automatic Speech Recognition Towards Education for Disabilities
Education is a fundamental right that enriches everyone’s life. However, physically challenged people often debar from the general and advanced education system. Audio-Visual Automatic Speech Recognition (AV-ASR) based system is useful to improve the education of physically challenged people by providing hands-free computing. They can communicate to the learning system through AV-ASR. However, it is challenging to trace the lip correctly for visual modality. Thus, this paper addresses the appearance-based visual feature along with the co-occurrence statistical measure for visual speech recognition. Local Binary Pattern-Three Orthogonal Planes (LBP-TOP) and Grey-Level Co-occurrence Matrix (GLCM) is proposed for visual speech information. The experimental results show that the proposed system achieves 76.60 % accuracy for visual speech and 96.00 % accuracy for audio speech recognition
Open Set Classification of GAN-based Image Manipulations via a ViT-based Hybrid Architecture
Classification of AI-manipulated content is receiving great attention, for
distinguishing different types of manipulations. Most of the methods developed
so far fail in the open-set scenario, that is when the algorithm used for the
manipulation is not represented by the training set. In this paper, we focus on
the classification of synthetic face generation and manipulation in open-set
scenarios, and propose a method for classification with a rejection option. The
proposed method combines the use of Vision Transformers (ViT) with a hybrid
approach for simultaneous classification and localization. Feature map
correlation is exploited by the ViT module, while a localization branch is
employed as an attention mechanism to force the model to learn per-class
discriminative features associated with the forgery when the manipulation is
performed locally in the image. Rejection is performed by considering several
strategies and analyzing the model output layers. The effectiveness of the
proposed method is assessed for the task of classification of facial attribute
editing and GAN attribution
Anuário científico da Escola Superior de Tecnologia da Saúde de Lisboa - 2021
É com grande prazer que apresentamos a mais recente edição (a 11.ª) do Anuário Científico da Escola Superior de Tecnologia da Saúde de Lisboa. Como instituição de ensino superior, temos o compromisso de promover e incentivar a pesquisa científica em todas as áreas do conhecimento que contemplam a nossa missão. Esta publicação tem como objetivo divulgar toda a produção científica desenvolvida pelos Professores, Investigadores, Estudantes e Pessoal não Docente da ESTeSL durante 2021. Este Anuário é, assim, o reflexo do trabalho árduo e dedicado da nossa comunidade, que se empenhou na produção de conteúdo científico de elevada qualidade e partilhada com a Sociedade na forma de livros, capítulos de livros, artigos publicados em revistas nacionais e internacionais, resumos de comunicações orais e pósteres, bem como resultado dos trabalhos de 1º e 2º ciclo. Com isto, o conteúdo desta publicação abrange uma ampla variedade de tópicos, desde temas mais fundamentais até estudos de aplicação prática em contextos específicos de Saúde, refletindo desta forma a pluralidade e diversidade de áreas que definem, e tornam única, a ESTeSL. Acreditamos que a investigação e pesquisa científica é um eixo fundamental para o desenvolvimento da sociedade e é por isso que incentivamos os nossos estudantes a envolverem-se em atividades de pesquisa e prática baseada na evidência desde o início dos seus estudos na ESTeSL. Esta publicação é um exemplo do sucesso desses esforços, sendo a maior de sempre, o que faz com que estejamos muito orgulhosos em partilhar os resultados e descobertas dos nossos investigadores com a comunidade científica e o público em geral. Esperamos que este Anuário inspire e motive outros estudantes, profissionais de saúde, professores e outros colaboradores a continuarem a explorar novas ideias e contribuir para o avanço da ciência e da tecnologia no corpo de conhecimento próprio das áreas que compõe a ESTeSL. Agradecemos a todos os envolvidos na produção deste anuário e desejamos uma leitura inspiradora e agradável.info:eu-repo/semantics/publishedVersio
Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training
Text-to-motion generation is an emerging and challenging problem, which aims
to synthesize motion with the same semantics as the input text. However, due to
the lack of diverse labeled training data, most approaches either limit to
specific types of text annotations or require online optimizations to cater to
the texts during inference at the cost of efficiency and stability. In this
paper, we investigate offline open-vocabulary text-to-motion generation in a
zero-shot learning manner that neither requires paired training data nor extra
online optimization to adapt for unseen texts. Inspired by the prompt learning
in NLP, we pretrain a motion generator that learns to reconstruct the full
motion from the masked motion. During inference, instead of changing the motion
generator, our method reformulates the input text into a masked motion as the
prompt for the motion generator to ``reconstruct'' the motion. In constructing
the prompt, the unmasked poses of the prompt are synthesized by a text-to-pose
generator. To supervise the optimization of the text-to-pose generator, we
propose the first text-pose alignment model for measuring the alignment between
texts and 3D poses. And to prevent the pose generator from overfitting to
limited training texts, we further propose a novel wordless training mechanism
that optimizes the text-to-pose generator without any training texts. The
comprehensive experimental results show that our method obtains a significant
improvement against the baseline methods. The code is available at
https://github.com/junfanlin/oohmg
Information Extraction from Documents: Question Answering vs Token Classification in real-world setups
Research in Document Intelligence and especially in Document Key Information
Extraction (DocKIE) has been mainly solved as Token Classification problem.
Recent breakthroughs in both natural language processing (NLP) and computer
vision helped building document-focused pre-training methods, leveraging a
multimodal understanding of the document text, layout and image modalities.
However, these breakthroughs also led to the emergence of a new DocKIE subtask
of extractive document Question Answering (DocQA), as part of the Machine
Reading Comprehension (MRC) research field. In this work, we compare the
Question Answering approach with the classical token classification approach
for document key information extraction. We designed experiments to benchmark
five different experimental setups : raw performances, robustness to noisy
environment, capacity to extract long entities, fine-tuning speed on Few-Shot
Learning and finally Zero-Shot Learning. Our research showed that when dealing
with clean and relatively short entities, it is still best to use token
classification-based approach, while the QA approach could be a good
alternative for noisy environment or long entities use-cases
Deep Unrestricted Document Image Rectification
In recent years, tremendous efforts have been made on document image
rectification, but existing advanced algorithms are limited to processing
restricted document images, i.e., the input images must incorporate a complete
document. Once the captured image merely involves a local text region, its
rectification quality is degraded and unsatisfactory. Our previously proposed
DocTr, a transformer-assisted network for document image rectification, also
suffers from this limitation. In this work, we present DocTr++, a novel unified
framework for document image rectification, without any restrictions on the
input distorted images. Our major technical improvements can be concluded in
three aspects. Firstly, we upgrade the original architecture by adopting a
hierarchical encoder-decoder structure for multi-scale representation
extraction and parsing. Secondly, we reformulate the pixel-wise mapping
relationship between the unrestricted distorted document images and the
distortion-free counterparts. The obtained data is used to train our DocTr++
for unrestricted document image rectification. Thirdly, we contribute a
real-world test set and metrics applicable for evaluating the rectification
quality. To our best knowledge, this is the first learning-based method for the
rectification of unrestricted document images. Extensive experiments are
conducted, and the results demonstrate the effectiveness and superiority of our
method. We hope our DocTr++ will serve as a strong baseline for generic
document image rectification, prompting the further advancement and application
of learning-based algorithms. The source code and the proposed dataset are
publicly available at https://github.com/fh2019ustc/DocTr-Plus
SViTT: Temporal Learning of Sparse Video-Text Transformers
Do video-text transformers learn to model temporal relationships across
frames? Despite their immense capacity and the abundance of multimodal training
data, recent work has revealed the strong tendency of video-text models towards
frame-based spatial representations, while temporal reasoning remains largely
unsolved. In this work, we identify several key challenges in temporal learning
of video-text transformers: the spatiotemporal trade-off from limited network
size; the curse of dimensionality for multi-frame modeling; and the diminishing
returns of semantic information by extending clip length. Guided by these
findings, we propose SViTT, a sparse video-text architecture that performs
multi-frame reasoning with significantly lower cost than naive transformers
with dense attention. Analogous to graph-based networks, SViTT employs two
forms of sparsity: edge sparsity that limits the query-key communications
between tokens in self-attention, and node sparsity that discards uninformative
visual tokens. Trained with a curriculum which increases model sparsity with
the clip length, SViTT outperforms dense transformer baselines on multiple
video-text retrieval and question answering benchmarks, with a fraction of
computational cost. Project page: http://svcl.ucsd.edu/projects/svitt.Comment: CVPR 202
- …