Tools for image annotation Using context-awareness, NFC and image clustering

Annotation of images is crucial for enabling keyword based imagesearch. However, the enormous amount of available digital photos makesmanual annotation impractical, and requires methods for automaticimage annotation. This paper describes two complementary approachesto automatic annotation of images depicting some public attraction. TheLoTagr system provides annotation information for already captured,geo-positioned images, by selecting nearby, previously tagged imagesfrom a source image collection, and subsequently collect the mostfrequently used tags from these images. The NfcAnnotate systemenables annotation at image capture time, by using NFC (Near FieldCommunication) and NFC information tags provided at the site ofthe attraction. NfcAnnotate enables clustering of topically relatedimages, which makes it possible annotate a set of images in oneannotation operation. In cases when NFC information tags are notavailable, NfcAnnotate image clustering can be combined with LoTagrto conveniently annotate every image in the cluster in a single operation

การวิเคราะห์เหมืองดัชนีถ้อยคำ จากข้อมูลระบุตำแหน่งเชิงพื้นที่ผ่านสื่อสังคมออนไลน์

งานวิจัยนี้มีวัตถุประสงค์เพื่อวิเคราะห์เหมืองดัชนีถ้อยคำจากข้อมูลระบุตำแหน่งเชิงพื้นที่ผ่านสื่อสังคมออนไลน์ แอปพลิเคชันฟลิกค์เกอร์ (flickr) ในประเทศไทย ข้อมูลที่ได้จากการระบุตำแหน่ง (geo-tagged) มาจากการที่ผู้ใช้งานแบ่งปันข้อมูล รูปภาพหรือแสดงความคิดเห็นต่างๆ ที่มีความสัมพันธ์กับสถานที่ งานวิจัยนี้นำเทคนิคเหมืองดัชนีข้อความ (tags mining) โดยการนำข้อมูลผ่านกระบวนการสกัดความรู้ มาวิเคราะห์เพื่อค้นหารูปแบบหรือความสัมพันธ์ของข้อมูล การดำเนินการวิจัยแบ่งออกเป็น 3 ขั้นตอนหลักคือ 1) ดึงข้อมูลระบุตำแหน่งและสร้างคลังฐานข้อมูล 2) ประมวลผลข้อความ 3) จัดกลุ่มแบบลำดับขั้นและหากฏของความสัมพันธ์ระหว่างข้อความและวิเคราะห์ความหนาแน่นเชิงพื้นที่ ผลการวิจัยที่ได้จากการจัดกลุ่มและการหากฏของความสัมพันธ์ดัชนีข้อความและการวิเคราะห์ความหนาแน่นที่อธิบายถึงลักษณะการกระจายเชิงพื้นที่ของการระบุตำแหน่ง flickr สรุปได้ว่า ดัชนีถ้อยคำ flickr ที่ความคล้ายคลึงกันมากในแต่ละกลุ่มจะมีค่าความเชื่อมั่นสูง ซึ่งสามารถแบ่งกลุ่มของข้อความออกเป็น 3 ประเภทที่สำคัญ ได้แก่ ประเภทสถานที่ท่องเที่ยวเชิงธรรมชาติ สถานที่ท่องเที่ยวเชิงวัฒนธรรม และกิจกรรมพิเศษอื่นๆ ซึ่งเชื่อมโยงกับลักษณะการใช้งานสื่อสังคมออนไลน์ และแสดงให้เห็นถึงปัจจัยต่างๆ ที่มีผลต่อพฤติกรรมและการตอบสนองของผู้ใช้งาน ได้แก่ ความนิยมของสถานที่ท่องเที่ยว ตามลักษณะภูมิประเทศและฤดูกาล ซึ่งสามารถนำไปใช้ประโยชน์ในการวางแผนและพัฒนาในด้านต่างๆ การเข้าถึงพื้นที่ โครงสร้างพื้นฐาน การให้บริการเครือข่ายอินเทอร์เน็ต คมนาคมขนส่ง และกิจกรรมต่างๆ ที่มีความสำคัญต่อสถานที่นั้นๆ คำสำคัญ: เหมืองดัชนีข้อความ การระบุตำแหน่ง ภูมิสารสนเทศศาสตร

Potential Indirect Relationships in Productive Networks

Productive Networks, such as Social Networks Services, organize evidence about human behavior. This evidence is independent of the network content type, and may support the discovery of new relationships between users and content, or with other users. These indirect relationships are important for recommendation systems, and systems where potential relationships between users and content (e.g., locations) is relevant, such as with the emergency management domain, where the discovery of relationships between users and locations on productive networks may enable the identification of population density variations, increasing the accuracy of emergency alerts. This thesis presents a Productive Networks model, which enables the development of a methodology for indirect relationships discovery, using the metadata on the network, and avoiding the computational cost of content analysis. We designed and conducted a set of experiments to evaluate our proposals. Our results are twofold: firstly, the productive network model is sufficiently robust to represent a wide range of networks; secondly, the indirect relationship discovery methodology successfully identifies relevant relationships between users and content. We also present applications of the model and methodology in several contexts

Explorando a localização e orientação de fotografias pessoais para descoberta de pontos de interesse baseada em agrupamento.

A descoberta de conhecimento a partir de grandes repositórios online de fotografias tem sido uma área de pesquisa bastante ativa nos últimos anos. Isso se deve principalmente a três fatores: incorporação de câmeras digitais e sensores de geolocalização aos dispositivos móveis; avanços na conectividade com a Internet; e evolução das redes sociais. As fotografias armazenadas nesses repositórios possuem metadados contextuais que podem ser utilizados em aplicações de descoberta de conhecimento, tais como: detecção de pontos de interesse (POIs); geração de roteiros de viagens; e organização automática de fotografias. A maioria das abordagens para detecção de POIs parte do princípio que as áreas geográficas onde uma grande quantidade de pessoas capturou fotografias indica a existência de um ponto de interesse. Porém, em muitos casos, os POIs estão localizados a uma certa distância desse local na orientação em que a câmera estava direcionada, e não no ponto exato da captura da fotografia. A maioria das técnicas propostas na literatura não consideram o uso da orientação no processo de detecção de pontos de interesses. Dessa forma, este trabalho propõe novos algoritmos e técnicas para detecção de pontos de interesse em cidades turísticas a partir de coleções de fotografias orientadas e georreferenciadas explorando de diversas formas a orientação geográfica. Esta pesquisa comprovou a importância do uso da orientação nos novos algoritmos voltados para detecção de pontos de interesses. Os experimentos, utilizando uma base de dados real de grandes cidades, demonstraram que os algoritmos considerando a orientação conseguem, em alguns cenários, superar os que não a consideram. Também foram propostas novas métricas de avaliação e uma ferramenta para auxiliar as atividades de descoberta de conhecimento baseada em grandes massas de fotografias.The knowledge discovery from huge photo repositories has been a very active area of research in the last years. This is due to three facts: the incorporation of digital cameras and geolocation sensors in mobile devices; the advances in Internet connectivity; and the evolution of social networks. The photos stored on those repositories have contextual metadata. Those metadata could be used for many applications of knowledge discovering, such as: Point of Interest (POI) detection; generating of tourist guides; and automatic photo organization. Most approaches for POI detection assume that geographic areas with high density of photos indicate the existence of a point of interest in that area. However, in many cases, the POIs are located in a certain distance of that position in direction where camera was aiming, and not in the exact point of photo shooting. Most of related work do not consider the use of orientation in the process of POI detection. In this way, we propose a set of algorithms and techniques for POI discovery in touristic cities using geotagged and oriented photos collection exploring the geographic orientation in different ways. This research has proven the importance of the usage of orientation in the new algorithms for POI detection. In the experiments with collections related to big cities, the algorithms considering orientation, in several scenarios, have beaten those that do not consider. Also, new metrics of evaluation have been proposed and a new framework to assist all the tasks for knowledge discovery based on huge photo collections.Cape

Suchbasierte automatische Bildannotation anhand geokodierter Community-Fotos

In the Web 2.0 era, platforms for sharing and collaboratively annotating images with keywords, called tags, became very popular. Tags are a powerful means for organizing and retrieving photos. However, manual tagging is time consuming. Recently, the sheer amount of user-tagged photos available on the Web encouraged researchers to explore new techniques for automatic image annotation. The idea is to annotate an unlabeled image by propagating the labels of community photos that are visually similar to it. Most recently, an ever increasing amount of community photos is also associated with location information, i.e., geotagged. In this thesis, we aim at exploiting the location context and propose an approach for automatically annotating geotagged photos. Our objective is to address the main limitations of state-of-the-art approaches in terms of the quality of the produced tags and the speed of the complete annotation process. To achieve these goals, we, first, deal with the problem of collecting images with the associated metadata from online repositories. Accordingly, we introduce a strategy for data crawling that takes advantage of location information and the social relationships among the contributors of the photos. To improve the quality of the collected user-tags, we present a method for resolving their ambiguity based on tag relatedness information. In this respect, we propose an approach for representing tags as probability distributions based on the algorithm of Laplacian score feature selection. Furthermore, we propose a new metric for calculating the distance between tag probability distributions by extending Jensen-Shannon Divergence to account for statistical fluctuations. To efficiently identify the visual neighbors, the thesis introduces two extensions to the state-of-the-art image matching algorithm, known as Speeded Up Robust Features (SURF). To speed up the matching, we present a solution for reducing the number of compared SURF descriptors based on classification techniques, while the accuracy of SURF is improved through an efficient method for iterative image matching. Furthermore, we propose a statistical model for ranking the mined annotations according to their relevance to the target image. This is achieved by combining multi-modal information in a statistical framework based on Bayes' rule. Finally, the effectiveness of each of mentioned contributions as well as the complete automatic annotation process are evaluated experimentally.Seit der Einführung von Web 2.0 steigt die Popularität von Plattformen, auf denen Bilder geteilt und durch die Gemeinschaft mit Schlagwörtern, sogenannten Tags, annotiert werden. Mit Tags lassen sich Fotos leichter organisieren und auffinden. Manuelles Taggen ist allerdings sehr zeitintensiv. Animiert von der schieren Menge an im Web zugänglichen, von Usern getaggten Fotos, erforschen Wissenschaftler derzeit neue Techniken der automatischen Bildannotation. Dahinter steht die Idee, ein noch nicht beschriftetes Bild auf der Grundlage visuell ähnlicher, bereits beschrifteter Community-Fotos zu annotieren. Unlängst wurde eine immer größere Menge an Community-Fotos mit geographischen Koordinaten versehen (geottagged). Die Arbeit macht sich diesen geographischen Kontext zunutze und präsentiert einen Ansatz zur automatischen Annotation geogetaggter Fotos. Ziel ist es, die wesentlichen Grenzen der bisher bekannten Ansätze in Hinsicht auf die Qualität der produzierten Tags und die Geschwindigkeit des gesamten Annotationsprozesses aufzuzeigen. Um dieses Ziel zu erreichen, wurden zunächst Bilder mit entsprechenden Metadaten aus den Online-Quellen gesammelt. Darauf basierend, wird eine Strategie zur Datensammlung eingeführt, die sich sowohl der geographischen Informationen als auch der sozialen Verbindungen zwischen denjenigen, die die Fotos zur Verfügung stellen, bedient. Um die Qualität der gesammelten User-Tags zu verbessern, wird eine Methode zur Auflösung ihrer Ambiguität vorgestellt, die auf der Information der Tag-Ähnlichkeiten basiert. In diesem Zusammenhang wird ein Ansatz zur Darstellung von Tags als Wahrscheinlichkeitsverteilungen vorgeschlagen, der auf den Algorithmus der sogenannten Laplacian Score (LS) aufbaut. Des Weiteren wird eine Erweiterung der Jensen-Shannon-Divergence (JSD) vorgestellt, die statistische Fluktuationen berücksichtigt. Zur effizienten Identifikation der visuellen Nachbarn werden in der Arbeit zwei Erweiterungen des Speeded Up Robust Features (SURF)-Algorithmus vorgestellt. Zur Beschleunigung des Abgleichs wird eine Lösung auf der Basis von Klassifikationstechniken präsentiert, die die Anzahl der miteinander verglichenen SURF-Deskriptoren minimiert, während die SURF-Genauigkeit durch eine effiziente Methode des schrittweisen Bildabgleichs verbessert wird. Des Weiteren wird ein statistisches Modell basierend auf der Baye'schen Regel vorgeschlagen, um die erlangten Annotationen entsprechend ihrer Relevanz in Bezug auf das Zielbild zu ranken. Schließlich wird die Effizienz jedes einzelnen, erwähnten Beitrags experimentell evaluiert. Darüber hinaus wird die Performanz des vorgeschlagenen automatischen Annotationsansatzes durch umfassende experimentelle Studien als Ganzes demonstriert

Annotation d'images via leur contexte spatio-temporel et les métadonnées du Web

En Recherche d'Information (RI), les documents sont classiquement indexés en fonction de leur contenu, qu'il soit textuel ou multimédia. Les moteurs de recherche s'appuyant sur ces index sont aujourd'hui des outils performants, répandus et indispensables. Ils visent à fournir des réponses pertinentes selon le besoin de l'utilisateur, sous forme de textes, images, sons, vidéos, etc. Nos travaux de thèse s'inscrivent dans le contexte des documents de type image. Plus précisément, nous nous sommes intéressés aux systèmes d'annotation automatique d'images qui permettent d'associer automatiquement des mots-clés à des images afin de pouvoir ensuite les rechercher par requête textuelle. Ce type d'annotation cherche à combler les lacunes des approches d'annotation manuelle et semi-automatique. Celles-ci ne sont plus envisageables dans le contexte actuel qui permet à chacun de prendre de nombreuses photos à faible coût (en lien avec la démocratisation des appareils photo numériques et l'intégration de capteurs numériques dans les téléphones mobiles). Parmi les différents types de collections d'images existantes (par exemple, médicales, satellitaires) dans le cadre de cette thèse nous nous sommes intéressés aux collections d'images de type paysage (c.-à-d. des images qui illustrent des points d'intérêt touristiques) pour lesquelles nous avons identifié des défis, tels que l'identification des nouveaux descripteurs pour les décrire et de nouveaux modèles pour fusionner ces derniers, l'identification des sources d'information pertinentes et le passage à l'échelle. Nos contributions portent sur trois principaux volets. En premier lieu, nous nous sommes attachés à exploiter différents descripteurs qui peuvent influencer la description des images de type paysage : le descripteur de spatialisation (caractérisé par la latitude et la longitude des images), le descripteur de temporalité (caractérisé par la date et l'heure de la prise de vue) et le descripteur de thématique (caractérisé par les tags issus des plate formes de partage d'images). Ensuite, nous avons proposé des approches pour modéliser ces descripteurs au regard de statistiques de tags liées à leur fréquence et rareté et sur des similarités spatiale et temporelle. Deuxièmement, nous avons proposé un nouveau processus d'annotation d'images qui vise à identifier les mots-clés qui décrivent le mieux les images-requêtes données en entrée d'un système d'annotation par un utilisateur. Pour ce faire, pour chaque image-requête nous avons mis en œuvre des filtres spatial, temporel et spatio-temporel afin d'identifier les images similaires ainsi que leurs tags associés. Ensuite, nous avons fédéré les différents descripteurs dans un modèle probabiliste afin de déterminer les termes qui décrivent le mieux chaque image-requête. Enfin, le fait que les contributions présentées ci-dessus s'appuient uniquement sur des informations issues des plateformes de partage d'images (c.-à-d. des informations subjectives) a suscité la question suivante : les informations issues du Web peuvent-elles fournir des termes objectifs pour enrichir les descriptions initiales des images. À cet effet, nous avons proposé une approche basée sur les techniques d'expansion de requêtes du domaine de la RI. Elle porte essentiellement sur l'étude de l'impact des différents algorithmes d'expansion, ainsi que sur l'agrégation des résultats fournis par le meilleur algorithme et les résultats fournis par le processus d'annotation d'images. Vu qu'il n'existe pas de cadre d'évaluation standard d'annotation automatique d'images, plus particulièrement adapté aux collections d'images de type paysage, nous avons proposé des cadres d'évaluation appropriés afin de valider nos contributions. En particulier, les différentes approches proposées sont évaluées au regard de la modélisation des descripteur de spatialisation, de temporalité et de thématique. De plus, nous avons validé le processus d'annotation d'images, et nous avons montré qu'il surpasse en qualité deux approches d'annotation d'images de la littérature. Nous avons comparé également l'approche d'enrichissement avec le processus d'annotation d'image pour souligner son efficacité et l'apport des informations issues du Web. Ces expérimentations ont nécessité le prototypage du logiciel AnnoTaGT, qui offre aux utilisateurs un cadre technique pour l'annotation automatique d'images.The documents processed by Information Retrieval (IR) systems are typically indexed according to their contents: Text or multimedia. Search engines based on these indexes aim to provide relevant answers to users' needs in the form of texts, images, sounds, videos, and so on. Our work is related to "image" documents. We are specifically interested in automatic image annotation systems that automatically associate keywords to images. Keywords are subsequently used for search purposes via textual queries. The automatic image annotation task intends to overcome the issues of manual and semi-automatic annotation tasks, as they are no longer feasible in nowadays' context (i.e., the development of digital technologies and the advent of devices, such as smartphones, allowing anyone to take images with a minimal cost). Among the different types of existing image collections (e.g., medical, satellite) in our work we are interested in landscape image collections for which we identified the following challenges: What are the most discriminant features for this type of images ? How to model and how to merge these features ? What are the sources of information that should be considered ? How to manage scalability issues ? The proposed contribution is threefold. First, we use different factors that influence the description of landscape images: The spatial factor (i.e., latitude and longitude of images), the temporal factor (i.e., the time when the images were taken), and the thematic factor (i.e., tags crowdsourced and contributed to image sharing platforms). We propose various techniques to model these factors based on tag frequency, as well as spatial and temporal similarities. The choice of these factors is based on the following assumptions: A tag is all the more relevant for a query-image as it is associated with images located in its close geographical area ; A tag is all the more relevant for a query-image as it is associated with images captured close in time to it ; sourcing concept). Second, we introduce a new image annotation process that recommends the terms that best describe a given query-image provided by a user. For each query-image we rely on spatial, temporal, and spatio-temporal filters to identify similar images along with their tags. Then, the different factors are merged through a probabilistic model to boost the terms best describing each query-image. Third, the contributions presented above are only based on information extracted from image photo sharing platforms (i.e., subjective information). This raised the following research question: Can the information extracted from the Web provide objective terms useful to enrich the initial description of images? We tackle this question by introducing an approach relying on query expansion techniques developed in IR. As there is no standard evaluation protocol for the automatic image annotation task tailored to landscape images, we designed various evaluation protocols to validate our contributions. We first evaluated the approaches defined to model the spatial, temporal, and thematic factors. Then, we validated the annotation image process and we showed that it yields significant improvement over two state-of-the-art baselines. Finally, we assessed the effectiveness of tag expansion through Web sources and showed its contribution to the image annotation process. These experiments are complemented by the image annotation prototype AnnoTaGT, which provides users with an operational framework for automatic image annotation

Geo-based automatic image annotation

A huge number of user-tagged images are daily uploaded to the web. Recently, a growing number of those images are also geotagged. These provide new opportunities for solutions to automatically tag images so that efficient image management and retrieval can be achieved. In this paper an automatic image annotation approach is proposed. It is based on a statistical model that combines two different kinds of information: high level information represented by user tags of images captured in the same location as a new unlabeled image (input image); and low level information represented by the visual similarity between the input image and the collection of geographically similar images. To maximize the number of images that are visually similar to the input image, an iterative visual matching approach is proposed and evaluated. The results show that a significant recall improvement can be achieved with an increasing number of iterations. The quality of the recommended tags has also been evaluated and an overall good performance has been observed