Semantic multimedia modelling & interpretation for annotation

The emergence of multimedia enabled devices, particularly the incorporation of cameras in mobile phones, and the accelerated revolutions in the low cost storage devices, boosts the multimedia data production rate drastically. Witnessing such an iniquitousness of digital images and videos, the research community has been projecting the issue of its significant utilization and management. Stored in monumental multimedia corpora, digital data need to be retrieved and organized in an intelligent way, leaning on the rich semantics involved. The utilization of these image and video collections demands proficient image and video annotation and retrieval techniques. Recently, the multimedia research community is progressively veering its emphasis to the personalization of these media. The main impediment in the image and video analysis is the semantic gap, which is the discrepancy among a user’s high-level interpretation of an image and the video and the low level computational interpretation of it. Content-based image and video annotation systems are remarkably susceptible to the semantic gap due to their reliance on low-level visual features for delineating semantically rich image and video contents. However, the fact is that the visual similarity is not semantic similarity, so there is a demand to break through this dilemma through an alternative way. The semantic gap can be narrowed by counting high-level and user-generated information in the annotation. High-level descriptions of images and or videos are more proficient of capturing the semantic meaning of multimedia content, but it is not always applicable to collect this information. It is commonly agreed that the problem of high level semantic annotation of multimedia is still far from being answered. This dissertation puts forward approaches for intelligent multimedia semantic extraction for high level annotation. This dissertation intends to bridge the gap between the visual features and semantics. It proposes a framework for annotation enhancement and refinement for the object/concept annotated images and videos datasets. The entire theme is to first purify the datasets from noisy keyword and then expand the concepts lexically and commonsensical to fill the vocabulary and lexical gap to achieve high level semantics for the corpus. This dissertation also explored a novel approach for high level semantic (HLS) propagation through the images corpora. The HLS propagation takes the advantages of the semantic intensity (SI), which is the concept dominancy factor in the image and annotation based semantic similarity of the images. As we are aware of the fact that the image is the combination of various concepts and among the list of concepts some of them are more dominant then the other, while semantic similarity of the images are based on the SI and concept semantic similarity among the pair of images. Moreover, the HLS exploits the clustering techniques to group similar images, where a single effort of the human experts to assign high level semantic to a randomly selected image and propagate to other images through clustering. The investigation has been made on the LabelMe image and LabelMe video dataset. Experiments exhibit that the proposed approaches perform a noticeable improvement towards bridging the semantic gap and reveal that our proposed system outperforms the traditional systems

Um Modelo para a visualização de conhecimento baseado em imagens semânticas

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico. Programa de Pós-Graduação em Engenharia e Gestão do ConhecimentoOs avanços no processamento e gerenciamento eletrônico de documentos têm gerado um acúmulo grande de conhecimento que tem excedido o que os usuários comuns podem perceber. Uma quantidade considerável de conhecimento encontra-se explicitado em diversos documentos armazenados em repositórios digitais. Em muitos casos, a possibilidade de acessar de forma eficiente e reutilizar este conhecimento é limitada. Como resultado disto, a maioria do conhecimento não é suficientemente explorado nem compartilhado, e conseqüentemente é esquecido em um tempo relativamente curto. As tecnologias emergentes de visualização e o sistema perceptual humano podem ser explorados para melhorar o acesso a grandes espaços de informação facilitando a detecção de padrões. Por outro lado, o uso de elementos visuais que contenham representações do mundo real que a priori são conhecidos pelo grupo-alvo e que fazem parte da sua visão de mundo, permite que o conhecimento apresentado por meio destas representações possa facilmente ser relacionados com o conhecimento prévio dos indivíduos, facilitando assim a aprendizagem. Apesar das representações visuais terem sido usadas como suporte para a disseminação do conhecimento, não têm sido propostos modelos que integrem os métodos e técnicas da engenharia do conhecimento com o uso das imagens como meio para recuperar e visualizar conhecimento. Neste trabalho apresenta-se um modelo que visa facilitar a visualização do conhecimento armazenado em repositórios digitais usando imagens semânticas. O usuário, através das imagens semânticas, pode recuperar e visualizar o conhecimento relacionado às entidades representadas nas regiões das imagens. As imagens semânticas são representações visuais do mundo real as quais são conhecidas previamente pelo grupo alvo e possuem mecanismos que permitem identificar os conceitos do domínio representados em cada região. O modelo proposto apóia-se no framework para visualização do conhecimento proposto por Burkhard e descreve as interações dos usuários com as imagens. Um protótipo foi desenvolvido para demonstrar a viabilidade do modelo usando imagens no domínio da anatomia, a Foundational Model of Anatomy e a Unified Medical Language System como conhecimento do domínio e o banco de dados da Scientific Electronic Library Online como repositório de documento.Advances in processing and electronic document management have generated a great accumulation of knowledge that is beyond what ordinary users can understand. A considerable amount of knowledge is explained in various documents stored in digital repositories. In many cases, the ability to eficiently access and reuse this knowledge is limited. As a result, most knowledge is not exploited or shared, and therefore it is forgotten in a relatively short time. The emerging technologies of visualization and the human perceptual system can be exploited to improve access to large information spaces facilitating the patterns detection. Moreover, the use of visual elements that contain representations of the real world that are known a priori by the target group and that are part of his world view, allows that the knowledge presented by these representations can be easily related to their prior knowledge, thereby facilitating learning. Despite visual representations have been used to support knowledge dissemination, no models have been proposed to integrate knowledge engineering methods and techniques with the use of images as a medium to retrieve and display knowledge. This work presents a model that aims to facilitate the visualization of the knowledge stored in digital repositories using semantic images. Through the semantic images, the user can retrieve and visualize the knowledge related to the entities represented in the image regions. The semantic images are visual representations of the real world which are known in advance by the target group and have mechanisms to identify domain concepts represented in each region. The proposed model is based on the framework for visualization of knowledge proposed by Burkhard and describes the interactions of users with the images. A prototype was eveloped to demonstrate the feasibility of the model using archetypes in the field of anatomy, using the Foundational Model of Anatomy and the Unifiled Medical Language System as knowledge domain and the database of the Scientific Electronic Library Online as a document repository