Content vs. context for multimedia semantics: the case of SenseCam image structuring

Much of the current work on determining multimedia semantics from multimedia artifacts is based around using either context, or using content. When leveraged thoroughly these can independently provide content description which is used in building content-based applications. However, there are few cases where multimedia semantics are determined based on an integrated analysis of content and context. In this keynote talk we present one such example system in which we use an integrated combination of the two to automatically structure large collections of images taken by a SenseCam, a device from Microsoft Research which passively records a person’s daily activities. This paper describes the post-processing we perform on SenseCam images in order to present a structured, organised visualisation of the highlights of each of the wearer’s days

Towards a syntax for multimedia semantics

This article describes the current state of the art on representing the fouressential conceptual facets of a multimedia unit, namely the form and substance of content and the form and substance of its expression, and points to the still un solved problems regarding the syntax for media semantics. We first provide a brief overview of the general features of the MPEG-7 standard and its different parts. This serves as a description of the state of the art in content description for audio-visual media. We then analyse the ability of one of these parts for its capability to define structures for describing media semantics. We describe the problems of two currently conflicting MPEG-7 representations of expression-based media semantic, which should be equivalent. We then discuss high-level aspects of media semantics, namely the general problems of an ontology for media semantics. Finally, we talk about the problems of applying the theoretical conceptsto real applications

Context-based multimedia semantics modelling and representation

The evolution of the World Wide Web, increase in processing power, and more network bandwidth have contributed to the proliferation of digital multimedia data. Since multimedia data has become a critical resource in many organisations, there is an increasing need to gain efficient access to data, in order to share, extract knowledge, and ultimately use the knowledge to inform business decisions. Existing methods for multimedia semantic understanding are limited to the computable low-level features; which raises the question of how to identify and represent the high-level semantic knowledge in multimedia resources.In order to bridge the semantic gap between multimedia low-level features and high-level human perception, this thesis seeks to identify the possible contextual dimensions in multimedia resources to help in semantic understanding and organisation. This thesis investigates the use of contextual knowledge to organise and represent the semantics of multimedia data aimed at efficient and effective multimedia content-based semantic retrieval.A mixed methods research approach incorporating both Design Science Research and Formal Methods for investigation and evaluation was adopted. A critical review of current approaches for multimedia semantic retrieval was undertaken and various shortcomings identified. The objectives for a solution were defined which led to the design, development, and formalisation of a context-based model for multimedia semantic understanding and organisation. The model relies on the identification of different contextual dimensions in multimedia resources to aggregate meaning and facilitate semantic representation, knowledge sharing and reuse. A prototype system for multimedia annotation, CONMAN was built to demonstrate aspects of the model and validate the research hypothesis, H₁.Towards providing richer and clearer semantic representation of multimedia content, the original contributions of this thesis to Information Science include: (a) a novel framework and formalised model for organising and representing the semantics of heterogeneous visual data; and (b) a novel S-Space model that is aimed at visual information semantic organisation and discovery, and forms the foundations for automatic video semantic understanding

Semantic web technologies for video surveillance metadata

Video surveillance systems are growing in size and complexity. Such systems typically consist of integrated modules of different vendors to cope with the increasing demands on network and storage capacity, intelligent video analytics, picture quality, and enhanced visual interfaces. Within a surveillance system, relevant information (like technical details on the video sequences, or analysis results of the monitored environment) is described using metadata standards. However, different modules typically use different standards, resulting in metadata interoperability problems. In this paper, we introduce the application of Semantic Web Technologies to overcome such problems. We present a semantic, layered metadata model and integrate it within a video surveillance system. Besides dealing with the metadata interoperability problem, the advantages of using Semantic Web Technologies and the inherent rule support are shown. A practical use case scenario is presented to illustrate the benefits of our novel approach

Linking the Semantics Ecosystem with Semantics Derivation Rules for Multimedia Content

Multimedia content exhibits multiple semantics that is influenced by different factors like time, contextual use, and personal background. With the semantics ecosystem, we find an elegant and high-level description of the different factors that influence the semantics of multimedia content. On the other hand, semantics derivation rules are a concrete means to extract and to derive semantics of multimedia content while authoring it. These rules are directly applicable in concrete applications and domains. Thus, there is a gap between the high-level ecosystem and the concrete semantics derivation rules. In this position paper, we propose the use of an ontology-based description of events to combine the high-level description of the semantics ecosystem with the concrete method of semantics derivation for page-based multimedia presentations

Multimedia Markup Tools for OpenKnowledge

OpenKnowledge is a peer-to-peer system for sharing knowledge and is driven by interaction models that give the necessary context for mapping of ontological knowledge fragments necessary for the interaction to take place. The OpenKnowledge system is agnostic to any specific data formats that are used in the interactions, relying on ontology mapping techniques for shimming the messages. The potentially large search space for matching ontologies is reduced by the shared context of the interaction. In this paper we investigate what this means for multimedia data on the OpenKnowledge network by discussing how an existing application that provides multimedia annotation (the Semantic Logger) can be migrated into the OpenKnowledge domain