Museums for all: translation and interpreting for multimodal spaces as a tool for universal accessibility

Audiovisual Translation (AVT) has a scientific responsibility to develop analytical methodologies for the textual phenomenon of multimodality, and for the translation strategies associated with it. At the same time, it should aim to provide studies of universal accessibility with a powerful tool for facilitating access to knowledge. This article offers some reflections on the theoretical foundations of AVT and considers how these are projected in the creation of new professional profiles, with specific application to universal accessibility in the museums.La Traducción Audiovisual (TAV) tiene la responsabilidad científica de desarrollar metodologías de análisis para el fenómeno textual de la multimodalidad así como para sus estrategias de traducción, a la vez que ha de proporcionar a los estudios en accesibilidad universal una poderosa herramienta de acceso al conocimiento. Este artículo ofrece reflexiones en torno a los fundamentos teóricos de la TAV y a la proyección de estos en nuevos perfiles profesionales; todo ello aplicado a la accesibilidad museística universal.This article is the English version of “Museos para todos. La traducción e interpretación para entornos multimodales como herramienta de accesibilidad universal” by Catalina Jiménez Hurtado, Claudia Seibel & Silvia Soler Gallego. It was not published on the print version of MonTI for reasons of space. The online version of MonTI does not suffer from these limitations, and this is our way of promoting plurilingualism.AMATRA Project (P07-SEJ/2660)

Multimedia Information Retrieval nelle biblioteche

The paper aims to introduce libraries to the view that operating within the terms of traditional Information Retrieval (IR), only through textual language, is limitative, and that considering broader criteria, as those of Multimedia Information Retrieval (MIR), is necessary. The paper stresses the story of MIR fundamental principles, from early years of questioning on documentation to today’s theories on semantic means. New issues for a LIS methodology of processing and searching multimedia documents are theoretically argued, introducing MIR as a holistic whole composed by content-based and semantic information retrieval methodologies. MIR offers a better information searching way: every kind of digital document can be analyzed and retrieved through the elements of language appropriate to its own nature. MIR approach directly handles the concrete content of documents, also considering semantic aspects. Paper conclusions remark the organic integration of the revolutionary contentual conception of information processing with an improved semantics conception, gathering and composing advantages of both systems for accessing to information.L'articolo vuole introdurre le biblioteche alla prospettiva che operare entro i termini dell'Information Retrieval (IR) tradizionale mediante il solo uso del linguaggio testuale è limitativo, e che prendere in considerazione i criteri più ampi del Multimedia Information Retrieval (MIR) è invece necessario. L'articolo illustra la storia dei principi fondamentali del MIR, a partire dai primi anni di dibattito sulla documentazione fino alle teorie odierne sui significati semantici. Vengono dibattute nuovi argomentazioni teoriche per una metodologia LIS di trattamento e ricerca di documenti multimediali, proponendo il MIR come un tutto olistico composto da metolodogie di information retrieval semantico e basato sul contenuto. Il MIR offre modalità di ricerca migliori: ogni tipologia di documento digitale può essere analizzata e recuperata attraverso elementi del linguaggio appropriato alla sua specifica natura. L'approccio del MIR si basa sulla gestione diretta del contenuto dei documenti, considerando anche gli aspetti semantici. Le conclusioni dell'articolo rimarcano l'integrazione organica della rivoluzione della concezione di tipo contenutistico del trattamento dell'informazione con una concezione semantica migliorata, raccogliendo e componendo i vantaggi di entrambi i sistemi per l'accesso all'informazione

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning

Developing artificial learning systems that can understand and generate natural language has been one of the long-standing goals of artificial intelligence. Recent decades have witnessed an impressive progress on both of these problems, giving rise to a new family of approaches. Especially, the advances in deep learning over the past couple of years have led to neural approaches to natural language generation (NLG). These methods combine generative language learning techniques with neural-networks based frameworks. With a wide range of applications in natural language processing, neural NLG (NNLG) is a new and fast growing field of research. In this state-of-the-art report, we investigate the recent developments and applications of NNLG in its full extent from a multidimensional view, covering critical perspectives such as multimodality, multilinguality, controllability and learning strategies. We summarize the fundamental building blocks of NNLG approaches from these aspects and provide detailed reviews of commonly used preprocessing steps and basic neural architectures. This report also focuses on the seminal applications of these NNLG models such as machine translation, description generation, automatic speech recognition, abstractive summarization, text simplification, question answering and generation, and dialogue generation. Finally, we conclude with a thorough discussion of the described frameworks by pointing out some open research directions.This work has been partially supported by the European Commission ICT COST Action “Multi-task, Multilingual, Multi-modal Language Generation” (CA18231). AE was supported by BAGEP 2021 Award of the Science Academy. EE was supported in part by TUBA GEBIP 2018 Award. BP is in in part funded by Independent Research Fund Denmark (DFF) grant 9063-00077B. IC has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 838188. EL is partly funded by Generalitat Valenciana and the Spanish Government throught projects PROMETEU/2018/089 and RTI2018-094649-B-I00, respectively. SMI is partly funded by UNIRI project uniri-drustv-18-20. GB is partly supported by the Ministry of Innovation and the National Research, Development and Innovation Office within the framework of the Hungarian Artificial Intelligence National Laboratory Programme. COT is partially funded by the Romanian Ministry of European Investments and Projects through the Competitiveness Operational Program (POC) project “HOLOTRAIN” (grant no. 29/221 ap2/07.04.2020, SMIS code: 129077) and by the German Academic Exchange Service (DAAD) through the project “AWAKEN: content-Aware and netWork-Aware faKE News mitigation” (grant no. 91809005). ESA is partially funded by the German Academic Exchange Service (DAAD) through the project “Deep-Learning Anomaly Detection for Human and Automated Users Behavior” (grant no. 91809358)

Integrated analysis of audiovisual signals and external information sources for event detection in team sports video

Ph.DDOCTOR OF PHILOSOPH

Understanding, Categorizing and Predicting Semantic Image-Text Relations

Two modalities are often used to convey information in a complementary and beneficial manner, e.g., in online news, videos, educational resources, or scientific publications. The automatic understanding of semantic correlations between text and associated images as well as their interplay has a great potential for enhanced multimodal web search and recommender systems. However, automatic understanding of multimodal information is still an unsolved research problem. Recent approaches such as image captioning focus on precisely describing visual content and translating it to text, but typically address neither semantic interpretations nor the specific role or purpose of an image-text constellation. In this paper, we go beyond previous work and investigate, inspired by research in visual communication, useful semantic image-text relations for multimodal information retrieval. We derive a categorization of eight semantic image-text classes (e.g., "illustration" or "anchorage") and show how they can systematically be characterized by a set of three metrics: cross-modal mutual information, semantic correlation, and the status relation of image and text. Furthermore, we present a deep learning system to predict these classes by utilizing multimodal embeddings. To obtain a sufficiently large amount of training data, we have automatically collected and augmented data from a variety of data sets and web resources, which enables future research on this topic. Experimental results on a demanding test set demonstrate the feasibility of the approach.Comment: 8 pages, 8 Figures, 5 table

Multimedia Annotation Interoperability Framework

Multimedia systems typically contain digital documents of mixed media types, which are indexed on the basis of strongly divergent metadata standards. This severely hamplers the inter-operation of such systems. Therefore, machine understanding of metadata comming from different applications is a basic requirement for the inter-operation of distributed Multimedia systems. In this document, we present how interoperability among metadata, vocabularies/ontologies and services is enhanced using Semantic Web technologies. In addition, it provides guidelines for semantic interoperability, illustrated by use cases. Finally, it presents an overview of the most commonly used metadata standards and tools, and provides the general research direction for semantic interoperability using Semantic Web technologies

Spectators’ aesthetic experiences of sound and movement in dance performance

In this paper we present a study of spectators’ aesthetic experiences of sound and movement in live dance performance. A multidisciplinary team comprising a choreographer, neuroscientists and qualitative researchers investigated the effects of different sound scores on dance spectators. What would be the impact of auditory stimulation on kinesthetic experience and/or aesthetic appreciation of the dance? What would be the effect of removing music altogether, so that spectators watched dance while hearing only the performers’ breathing and footfalls? We investigated audience experience through qualitative research, using post-performance focus groups, while a separately conducted functional brain imaging (fMRI) study measured the synchrony in brain activity across spectators when they watched dance with sound or breathing only. When audiences watched dance accompanied by music the fMRI data revealed evidence of greater intersubject synchronisation in a brain region consistent with complex auditory processing. The audience research found that some spectators derived pleasure from finding convergences between two complex stimuli (dance and music). The removal of music and the resulting audibility of the performers’ breathing had a significant impact on spectators’ aesthetic experience. The fMRI analysis showed increased synchronisation among observers, suggesting greater influence of the body when interpreting the dance stimuli. The audience research found evidence of similar corporeally focused experience. The paper discusses possible connections between the findings of our different approaches, and considers the implications of this study for interdisciplinary research collaborations between arts and sciences