Issues in designing novel applications for multimedia technologies

Emerging computational multimedia tools and techniques promise powerful ways to organise, search and browse our ever-increasing multimedia contents by automating annotation and indexing, augmenting meta-data, understanding media contents, linking related pieces of information amongst them, and providing intriguing visualisation and exploration front-ends. Identifying real-world scenarios and designing interactive applications that leverage these developing multimedia technology is certainly an important research topic in itself but poses a number of challenges. In this talk, I will discuss and highlight some of these challenges in designing these novel applications by reflecting on my own design practice with a number of design examples

ELAN development, keeping pace with communities' needs

ELAN is a versatile multimedia annotation tool that is being developed at the Max Planck Institute for Psycholinguistics. About a decade ago it emerged out of a number of corpus tools and utilities and it has been extended ever since. This paper focuses on the efforts made to ensure that the application keeps up with the growing needs of that era in linguistics and multimodality research; growing needs in terms of length and resolution of recordings, the number of recordings made and transcribed and the number of levels of annotation per transcription

Annotation Studio: multimedia text annotation for students

Annotation Studio will be a web-based application that actively engages students in interpreting literary texts and other humanities documents. While strengthening students' new media literacies, this open source web application will develop traditional humanistic skills including close reading, textual analysis, persuasive writing, and critical thinking. Initial features will include: 1) easy-to-use annotation tools that facilitate linking and comparing primary texts with multi-media source, variation, and adaptation documents; 2) sharable collections of multimedia materials prepared by faculty and student users; 3) multiple filtering and display mechanisms for texts, written annotations, and multimedia annotations; 4) collaboration functionality; and 5) multimedia composition tools. Products of the start-up phase will include a working prototype, feedback from students and instructors, and a white paper summarizing lessons learned

Multimedia search without visual analysis: the value of linguistic and contextual information

This paper addresses the focus of this special issue by analyzing the potential contribution of linguistic content and other non-image aspects to the processing of audiovisual data. It summarizes the various ways in which linguistic content analysis contributes to enhancing the semantic annotation of multimedia content, and, as a consequence, to improving the effectiveness of conceptual media access tools. A number of techniques are presented, including the time-alignment of textual resources, audio and speech processing, content reduction and reasoning tools, and the exploitation of surface features

Automated speech and audio analysis for semantic access to multimedia

The deployment and integration of audio processing tools can enhance the semantic annotation of multimedia content, and as a consequence, improve the effectiveness of conceptual access tools. This paper overviews the various ways in which automatic speech and audio analysis can contribute to increased granularity of automatically extracted metadata. A number of techniques will be presented, including the alignment of speech and text resources, large vocabulary speech recognition, key word spotting and speaker classification. The applicability of techniques will be discussed from a media crossing perspective. The added value of the techniques and their potential contribution to the content value chain will be illustrated by the description of two (complementary) demonstrators for browsing broadcast news archives

The VIA Annotation Software for Images, Audio and Video

In this paper, we introduce a simple and standalone manual annotation tool for images, audio and video: the VGG Image Annotator (VIA). This is a light weight, standalone and offline software package that does not require any installation or setup and runs solely in a web browser. The VIA software allows human annotators to define and describe spatial regions in images or video frames, and temporal segments in audio or video. These manual annotations can be exported to plain text data formats such as JSON and CSV and therefore are amenable to further processing by other software tools. VIA also supports collaborative annotation of a large dataset by a group of human annotators. The BSD open source license of this software allows it to be used in any academic project or commercial application.Comment: to appear in Proceedings of the 27th ACM International Conference on Multimedia (MM '19), October 21-25, 2019, Nice, France. ACM, New York, NY, USA, 4 page