Understanding user intention in image retrieval: generalization selection using multiple concept hierarchies

Image retrieval is the technique that helps Users to find and retrieve desired images from a huge image database. The user has firstly to formulate a query that expresses his/her needs.  This query may appear in textual form as in semantic retrieval (SR), in visual example form as in query by visual example (QBVE), or as a combination of these two forms named query by semantic example (QBSE). The focus of this paper lies in the techniques of analysing queries composed of multiple semantic examples. This is a very challenging task due to the different interpretations that can be drawn from the same query. To solve such a problem, we introduce a model based on Bayesian generalization. In cognitive science, Bayesian generalization, which is the base of most works in literature, is a method that tries to find, in one hierarchy of concepts, the parent concept of a given set of concepts. In addition and instead of using one single concept hierarchy, we propose a generalization so it can be used with multiple hierarchies where each one has a different semantic context and contains several abstraction levels. Our method consists in finding the optimal generalization by, firstly, determining the appropriate concept hierarchy, and then determining the appropriate level of generalization. Experimental evaluations demonstrate that our method, which uses multiple hierarchies, yields better results than those using only one single hierarchy

Distance Metric Learning from Uncertain Side Information for Automated Photo Tagging

Automated photo tagging is an important technique for many intelligent multimedia information systems, for example, smart photo management system and intelligent digital media library. To attack the challenge, several machine learning techniques have been developed and applied for automated photo tagging. For example, supervised learning techniques have been applied to automated photo tagging by training statistical classifiers from a collection of manually labeled examples. Although the existing approaches work well for small testbeds with relatively small number of annotation words, due to the long-standing challenge of object recognition, they often perform poorly in large-scale problems. Another limitation of the existing approaches is that they require a set of high-quality labeled data, which is not only expensive to collect but also time consuming. In this article, we investigate a social image based annotation scheme by exploiting implicit side information that is available for a large number of social photos from the social web sites. The key challenge of our intelligent annotation scheme is how to learn an effective distance metric based on implicit side information (visual or textual) of social photos. To this end, we present a novel “Probabilistic Distance Metric Learning” (PDML) framework, which can learn optimized metrics by effectively exploiting the implicit side information vastly available on the social web. We apply the proposed technique to photo annotation tasks based on a large social image testbed with over 1 million tagged photos crawled from a social photo sharing portal. Encouraging results show that the proposed technique is effective and promising for social photo based annotation tasks. </jats:p

Non-parametric Kernel Ranking Approach for Social Image Retrieval

National Research Foundation (NRF) Singapore; Ministry of Education, Singapore under its Academic Research Funding Tier

A Two-View Learning Approach for Image Tag Ranking

Singapore Ministry of Education Academic Research Fund Tier

Mobile app tagging

National Research Foundation (NRF) Singapore under IDM Futures Funding Initiative; Ministry of Education, Singapore under its Academic Research Funding Tier

Community discovery from social media by low-rank matrix recovery

The pervasive usage and reach of social media have attracted a surge of attention in the multimedia research community. Community discovery from social media has therefore become an important yet challenging issue. However, due to the subjective generating process, the explicitly observed communities (e.g., group-user and user-user relationship) are often noisy and incomplete in nature. This paper presents a novel approach to discovering communities from social media, including the group membership and user friend structure, by exploring a low-rank matrix recovery technique. In particular, we take Flickr as one exemplary social media platform. We first model the observed indicator matrix of the Flickr community as a summation of a low-rank true matrix and a sparse error matrix. We then formulate an optimization problem by regularizing the true matrix to coincide with the available rich context and content (i.e., photos and their associated tags). An iterative algorithm is developed to recover the true community indicator matrix. The proposed approach leads to a variety of social applications, including community visualization, interest group refinement, friend suggestion, and influential user identification. The evaluations on a large-scale testbed, consisting of 4,919 Flickr users, 1,467 interest groups, and over five million photos, show that our approach opens a new yet effective perspective to solve social network problems with sparse learning technique. Despite being focused on Flickr, our technique can be applied in any other social media community. </jats:p

Distance metric learning from uncertain side information with application to automated photo tagging

Automated photo tagging is essential to make massive unlabeled photos searchable by text search engines. Conventional image annotation approaches, though working reasonably well on small testbeds, are either computationally expensive or inaccurate when dealing with large-scale photo tagging. Recently, with the popularity of social networking websites, we observe a massive number of user-tagged images, referred to as “social images”, that are available on the web. Unlike traditional web images, social images often contain tags and other user-generated content, which offer a new opportunity to resolve some long-standing challenges in multimedia. In this work, we aim to address the challenge of large-scale automated photo tagging by exploring the social images. We present a retrieval based approach for automated photo tagging. To tag a test image, the proposed approach first retrieves k social images that share the largest visual similarity with the test image. The tags of the test image are then derived based on the tagging of the similar images. Due to the well-known semantic gap issue, a regular Euclidean distance-based retrieval method often fails to find semantically relevant images. To address the challenge of semantic gap, we propose a novel probabilistic distance metric learning scheme that (1) automatically derives constraints from the uncertain side information, and (2) efficiently learns a distance metric from the derived constraints. We apply the proposed technique to automated photo tagging tasks based on a social image testbed with over 200,000 images crawled from Flickr. Encouraging results show that the proposed technique is effective and promising for automated photo tagging