Modeling, classifying and annotating weakly annotated images using bayesian network

International audienceIn this paper, we propose a probabilistic graphical model to represent weakly annotated images. We consider an image as weakly annotated if the number of keywords defined for it is less than the maximum number defined in the ground truth. This model is used to classify images and automatically extend existing annotations to new images by taking into account semantic relations between keywords. The proposed method has been evaluated in visual-textual classification and automatic annotation of images. The visualtextual classification is performed by using both visual and textual information. The experimental results, obtained from a database of more than 30000 images, show an improvement by 50.5% in terms of recognition rate against only visual information classification. Taking into account semantic relations between keywords improves the recognition rate by 10.5%. Moreover, the proposed model can be used to extend existing annotations to weakly annotated images, by computing distributions of missing keywords. Semantic relations improve the mean rate of good annotations by 6.9%. Finally, the proposed method is competitive with a state-of-art model

Modélisation, classification et annotation d'images partiellement annotées avec un réseau Bayésien

National audienceDans cet article, nous proposons un modèle graphique probabiliste pour représenter des images partiellement annotées. Nous considérons une image comme partiellement annotée si elle ne possède pas le nombre maximal de mots-clés disponibles pour une image dans la vérité-terrain. Ce modèle est utilisé pour classifier des images et étendre automatiquement les annotations existantes à de nouvelles images, en prenant en compte les éventuelles relations sémantiques entre mots-clés. La méthode proposée a été évaluée en classification visuo-textuelle et en extension automatique d'annotations. La classification visuo-textuelle correspond à la classification effectuée en utilisant à la fois l'information visuelle et l'information textuelle, quand elle est disponible. Les résultats expérimentaux, obtenus à partir d'une base de plus de 30000 images, montrent une amélioration de 50.5% en moyenne, en terme de taux de reconnaissance, par rapport à la classification basée sur l'information visuelle seule. La prise en compte des éventuelles relations sémantiques entre mots-clés améliore le taux de reconnaissance de 10.5% en moyenne et le taux de bonnes annotations de 6.9% en moyenne. Enfin, la méthode proposée s'est montrée compétitive, expérimentalement, avec des classificateurs de l'état de l'art

Hierarchical Image Automatic Annotation Based on Discriminative and Generative Models

图像自动标注是模式识别与计算机视觉等领域中重要而又具有挑战性的问题.针对现有模型存在数据利用率低与易受正负样本不平衡影响等问题,提出了基于判别模型与生成模型的新型层叠图像自动标注模型.该模型第一层利用判别模型对未标注图像进行主题标注,获得相应的相关图像集;第二层利用提出的面向关键词的方法建立图像与关键词之间的联系,并使用提出的迭代算法分别对语义关键词与相关图像进行扩展;最后利用生成模型与扩展的相关图像集对未标注图像进行详细标注.该模型综合了判别模型与生成模型的优点,通过利用较少的相关训练图像来获得更好的标注结果.在COrEl 5k图像库上进行的实验验证了该模型的有效性.Image automatic annotation is a significant and challenging problem in pattern recognition and computer vision.Aiming at the problems that the existing models have low utilization and they are affected by unbalanced positive and negative samples,a hierarchical image annotation model is proposed.In the first layer,discriminative model is used to assign topic annotations to unlabeled images,and then the corresponding relevant image sets are obtained.In the second layer,a keywords-oriented method is proposed to establish links between images and keywords,and then the proposed iterative algorithm is used to expand semantic words and relevant image sets.Finally,a generative model is used to assign detailed annotations to unlabeled images on expanded relevant image sets.Hierarchical model uses less relevant training images to obtain better annotation results.Experimental results on Corel 5K datasets verify the effectiveness of proposed hierarchical image annotation model.国家自然科学基金项目(No.60873179;60803078);高等学校博士学科点专项科研基金项目(No.20090121110032);深圳市科技计划基础研究项目(No.JC200903180630A)资

Semantic multimedia modelling & interpretation for annotation

The emergence of multimedia enabled devices, particularly the incorporation of cameras in mobile phones, and the accelerated revolutions in the low cost storage devices, boosts the multimedia data production rate drastically. Witnessing such an iniquitousness of digital images and videos, the research community has been projecting the issue of its significant utilization and management. Stored in monumental multimedia corpora, digital data need to be retrieved and organized in an intelligent way, leaning on the rich semantics involved. The utilization of these image and video collections demands proficient image and video annotation and retrieval techniques. Recently, the multimedia research community is progressively veering its emphasis to the personalization of these media. The main impediment in the image and video analysis is the semantic gap, which is the discrepancy among a user’s high-level interpretation of an image and the video and the low level computational interpretation of it. Content-based image and video annotation systems are remarkably susceptible to the semantic gap due to their reliance on low-level visual features for delineating semantically rich image and video contents. However, the fact is that the visual similarity is not semantic similarity, so there is a demand to break through this dilemma through an alternative way. The semantic gap can be narrowed by counting high-level and user-generated information in the annotation. High-level descriptions of images and or videos are more proficient of capturing the semantic meaning of multimedia content, but it is not always applicable to collect this information. It is commonly agreed that the problem of high level semantic annotation of multimedia is still far from being answered. This dissertation puts forward approaches for intelligent multimedia semantic extraction for high level annotation. This dissertation intends to bridge the gap between the visual features and semantics. It proposes a framework for annotation enhancement and refinement for the object/concept annotated images and videos datasets. The entire theme is to first purify the datasets from noisy keyword and then expand the concepts lexically and commonsensical to fill the vocabulary and lexical gap to achieve high level semantics for the corpus. This dissertation also explored a novel approach for high level semantic (HLS) propagation through the images corpora. The HLS propagation takes the advantages of the semantic intensity (SI), which is the concept dominancy factor in the image and annotation based semantic similarity of the images. As we are aware of the fact that the image is the combination of various concepts and among the list of concepts some of them are more dominant then the other, while semantic similarity of the images are based on the SI and concept semantic similarity among the pair of images. Moreover, the HLS exploits the clustering techniques to group similar images, where a single effort of the human experts to assign high level semantic to a randomly selected image and propagate to other images through clustering. The investigation has been made on the LabelMe image and LabelMe video dataset. Experiments exhibit that the proposed approaches perform a noticeable improvement towards bridging the semantic gap and reveal that our proposed system outperforms the traditional systems

Um Modelo para a visualização de conhecimento baseado em imagens semânticas

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico. Programa de Pós-Graduação em Engenharia e Gestão do ConhecimentoOs avanços no processamento e gerenciamento eletrônico de documentos têm gerado um acúmulo grande de conhecimento que tem excedido o que os usuários comuns podem perceber. Uma quantidade considerável de conhecimento encontra-se explicitado em diversos documentos armazenados em repositórios digitais. Em muitos casos, a possibilidade de acessar de forma eficiente e reutilizar este conhecimento é limitada. Como resultado disto, a maioria do conhecimento não é suficientemente explorado nem compartilhado, e conseqüentemente é esquecido em um tempo relativamente curto. As tecnologias emergentes de visualização e o sistema perceptual humano podem ser explorados para melhorar o acesso a grandes espaços de informação facilitando a detecção de padrões. Por outro lado, o uso de elementos visuais que contenham representações do mundo real que a priori são conhecidos pelo grupo-alvo e que fazem parte da sua visão de mundo, permite que o conhecimento apresentado por meio destas representações possa facilmente ser relacionados com o conhecimento prévio dos indivíduos, facilitando assim a aprendizagem. Apesar das representações visuais terem sido usadas como suporte para a disseminação do conhecimento, não têm sido propostos modelos que integrem os métodos e técnicas da engenharia do conhecimento com o uso das imagens como meio para recuperar e visualizar conhecimento. Neste trabalho apresenta-se um modelo que visa facilitar a visualização do conhecimento armazenado em repositórios digitais usando imagens semânticas. O usuário, através das imagens semânticas, pode recuperar e visualizar o conhecimento relacionado às entidades representadas nas regiões das imagens. As imagens semânticas são representações visuais do mundo real as quais são conhecidas previamente pelo grupo alvo e possuem mecanismos que permitem identificar os conceitos do domínio representados em cada região. O modelo proposto apóia-se no framework para visualização do conhecimento proposto por Burkhard e descreve as interações dos usuários com as imagens. Um protótipo foi desenvolvido para demonstrar a viabilidade do modelo usando imagens no domínio da anatomia, a Foundational Model of Anatomy e a Unified Medical Language System como conhecimento do domínio e o banco de dados da Scientific Electronic Library Online como repositório de documento.Advances in processing and electronic document management have generated a great accumulation of knowledge that is beyond what ordinary users can understand. A considerable amount of knowledge is explained in various documents stored in digital repositories. In many cases, the ability to eficiently access and reuse this knowledge is limited. As a result, most knowledge is not exploited or shared, and therefore it is forgotten in a relatively short time. The emerging technologies of visualization and the human perceptual system can be exploited to improve access to large information spaces facilitating the patterns detection. Moreover, the use of visual elements that contain representations of the real world that are known a priori by the target group and that are part of his world view, allows that the knowledge presented by these representations can be easily related to their prior knowledge, thereby facilitating learning. Despite visual representations have been used to support knowledge dissemination, no models have been proposed to integrate knowledge engineering methods and techniques with the use of images as a medium to retrieve and display knowledge. This work presents a model that aims to facilitate the visualization of the knowledge stored in digital repositories using semantic images. Through the semantic images, the user can retrieve and visualize the knowledge related to the entities represented in the image regions. The semantic images are visual representations of the real world which are known in advance by the target group and have mechanisms to identify domain concepts represented in each region. The proposed model is based on the framework for visualization of knowledge proposed by Burkhard and describes the interactions of users with the images. A prototype was eveloped to demonstrate the feasibility of the model using archetypes in the field of anatomy, using the Foundational Model of Anatomy and the Unifiled Medical Language System as knowledge domain and the database of the Scientific Electronic Library Online as a document repository