64 research outputs found
Low-Rank Hypergraph Hashing for Large-Scale Remote Sensing Image Retrieval
[EN] As remote sensing (RS) images increase dramatically, the demand for remote sensing image retrieval (RSIR) is growing, and has received more and more attention. The characteristics of RS images, e.g., large volume, diversity and high complexity, make RSIR more challenging in terms of speed and accuracy. To reduce the retrieval complexity of RSIR, a hashing technique has been widely used for RSIR, mapping high-dimensional data into a low-dimensional Hamming space while preserving the similarity structure of data. In order to improve hashing performance, we propose a new hash learning method, named low-rank hypergraph hashing (LHH), to accomplish for the large-scale RSIR task. First, LHH employs a l(2-1) norm to constrain the projection matrix to reduce the noise and redundancy among features. In addition, low-rankness is also imposed on the projection matrix to exploit its global structure. Second, LHH uses hypergraphs to capture the high-order relationship among data, and is very suitable to explore the complex structure of RS images. Finally, an iterative algorithm is developed to generate high-quality hash codes and efficiently solve the proposed optimization problem with a theoretical convergence guarantee. Extensive experiments are conducted on three RS image datasets and one natural image dataset that are publicly available. The experimental results demonstrate that the proposed LHH outperforms the existing hashing learning in RSIR tasks.This research was supported in part by the Natural Science Foundation of China under Grant 61673220.Kong, J.; Sun, Q.; Mukherjee, M.; Lloret, J. (2020). Low-Rank Hypergraph Hashing for Large-Scale Remote Sensing Image Retrieval. Remote Sensing. 12(7):1-19. https://doi.org/10.3390/rs1207116411912
Attribute Learning for Image/Video Understanding
PhDFor the past decade computer vision research has achieved increasing success in visual recognition
including object detection and video classification. Nevertheless, these achievements still
cannot meet the urgent needs of image and video understanding. The recently rapid development
of social media sharing has created a huge demand for automatic media classification and annotation
techniques. In particular, these types of media data usually contain very complex social
activities of a group of people (e.g. YouTube video of a wedding reception) and are captured
by consumer devices with poor visual quality. Thus it is extremely challenging to automatically
understand such a high number of complex image and video categories, especially when these
categories have never been seen before.
One way to understand categories with no or few examples is by transfer learning which
transfers knowledge across related domains, tasks, or distributions. In particular, recently lifelong
learning has become popular which aims at transferring information to tasks without any
observed data. In computer vision, transfer learning often takes the form of attribute learning.
The key underpinning idea of attribute learning is to exploit transfer learning via an intermediatelevel
semantic representations – attributes. The semantic attributes are most commonly used as a
semantically meaningful bridge between low feature data and higher level class concepts, since
they can be used both descriptively (e.g., ’has legs’) and discriminatively (e.g., ’cats have it but
dogs do not’). Previous works propose many different attribute learning models for image and
video understanding. However, there are several intrinsic limitations and problems that exist in
previous attribute learning work. Such limitations discussed in this thesis include limitations of
user-defined attributes, projection domain-shift problems, prototype sparsity problems, inability
to combine multiple semantic representations and noisy annotations of relative attributes. To
tackle these limitations, this thesis explores attribute learning on image and video understanding
from the following three aspects.
Firstly to break the limitations of user-defined attributes, a framework for learning latent
attributes is present for automatic classification and annotation of unstructured group social activity
in videos, which enables the tasks of attribute learning for understanding complex multimedia
data with sparse and incomplete labels. We investigate the learning of latent attributes
for content-based understanding, which aims to model and predict classes and tags relevant to
objects, sounds and events – anything likely to be used by humans to describe or search for
media. Secondly, we propose the framework of transductive multi-view embedding hypergraph
label propagation and solve three inherent limitations of most previous attribute learning work,
i.e., the projection domain shift problems, the prototype sparsity problems and the inability to
combine multiple semantic representations. We explore the manifold structure of the data distributions
of different views projected onto the same embedding space via label propagation on
a graph. Thirdly a novel framework for robust learning is presented to effectively learn relative
attributes from the extremely noisy and sparse annotations. Relative attributes are increasingly
learned from pairwise comparisons collected via crowdsourcing tools which are more economic
and scalable than the conventional laboratory based data annotation. However, a major challenge
for taking a crowdsourcing strategy is the detection and pruning of outliers. We thus propose
a principled way to identify annotation outliers by formulating the relative attribute prediction
task as a unified robust learning to rank problem, tackling both the outlier detection and relative
attribute prediction tasks jointly.
In summary, this thesis studies and solves the key challenges and limitations of attribute
learning in image/video understanding. We show the benefits of solving these challenges and
limitations in our approach which thus achieves better performance than previous methods
On The Effect of Hyperedge Weights On Hypergraph Learning
Hypergraph is a powerful representation in several computer vision, machine
learning and pattern recognition problems. In the last decade, many researchers
have been keen to develop different hypergraph models. In contrast, no much
attention has been paid to the design of hyperedge weights. However, many
studies on pairwise graphs show that the choice of edge weight can
significantly influence the performances of such graph algorithms. We argue
that this also applies to hypegraphs. In this paper, we empirically discuss the
influence of hyperedge weight on hypegraph learning via proposing three novel
hyperedge weights from the perspectives of geometry, multivariate statistical
analysis and linear regression. Extensive experiments on ORL, COIL20, JAFFE,
Sheffield, Scene15 and Caltech256 databases verify our hypothesis. Similar to
graph learning, several representative hyperedge weighting schemes can be
concluded by our experimental studies. Moreover, the experiments also
demonstrate that the combinations of such weighting schemes and conventional
hypergraph models can get very promising classification and clustering
performances in comparison with some recent state-of-the-art algorithms
Next Generation of Product Search and Discovery
Online shopping has become an important part of people’s daily life with the rapid development of e-commerce. In some domains such as books, electronics, and CD/DVDs, online shopping has surpassed or even replaced the traditional shopping method. Compared with traditional retailing, e-commerce is information intensive. One of the key factors to succeed in e-business is how to facilitate the consumers’ approaches to discover a product. Conventionally a product search engine based on a keyword search or category browser is provided to help users find the product information they need. The general goal of a product search system is to enable users to quickly locate information of interest and to minimize users’ efforts in search and navigation. In this process human factors play a significant role. Finding product information could be a tricky task and may require an intelligent use of search engines, and a non-trivial navigation of multilayer categories. Searching for useful product information can be frustrating for many users, especially those inexperienced users.
This dissertation focuses on developing a new visual product search system that effectively extracts the properties of unstructured products, and presents the possible items of attraction to users so that the users can quickly locate the ones they would be most likely interested in. We designed and developed a feature extraction algorithm that retains product color and local pattern features, and the experimental evaluation on the benchmark dataset demonstrated that it is robust against common geometric and photometric visual distortions. Besides, instead of ignoring product text information, we investigated and developed a ranking model learned via a unified probabilistic hypergraph that is capable of capturing correlations among product visual content and textual content. Moreover, we proposed and designed a fuzzy hierarchical co-clustering algorithm for the collaborative filtering product recommendation. Via this method, users can be automatically grouped into different interest communities based on their behaviors. Then, a customized recommendation can be performed according to these implicitly detected relations. In summary, the developed search system performs much better in a visual unstructured product search when compared with state-of-art approaches. With the comprehensive ranking scheme and the collaborative filtering recommendation module, the user’s overhead in locating the information of value is reduced, and the user’s experience of seeking for useful product information is optimized
Supervised cross-modal factor analysis for multiple modal data classification
In this paper we study the problem of learning from multiple modal data for
purpose of document classification. In this problem, each document is composed
two different modals of data, i.e., an image and a text. Cross-modal factor
analysis (CFA) has been proposed to project the two different modals of data to
a shared data space, so that the classification of a image or a text can be
performed directly in this space. A disadvantage of CFA is that it has ignored
the supervision information. In this paper, we improve CFA by incorporating the
supervision information to represent and classify both image and text modals of
documents. We project both image and text data to a shared data space by factor
analysis, and then train a class label predictor in the shared space to use the
class label information. The factor analysis parameter and the predictor
parameter are learned jointly by solving one single objective function. With
this objective function, we minimize the distance between the projections of
image and text of the same document, and the classification error of the
projection measured by hinge loss function. The objective function is optimized
by an alternate optimization strategy in an iterative algorithm. Experiments in
two different multiple modal document data sets show the advantage of the
proposed algorithm over other CFA methods
Socializing the Semantic Gap: A Comparative Survey on Image Tag Assignment, Refinement and Retrieval
Where previous reviews on content-based image retrieval emphasize on what can
be seen in an image to bridge the semantic gap, this survey considers what
people tag about an image. A comprehensive treatise of three closely linked
problems, i.e., image tag assignment, refinement, and tag-based image retrieval
is presented. While existing works vary in terms of their targeted tasks and
methodology, they rely on the key functionality of tag relevance, i.e.
estimating the relevance of a specific tag with respect to the visual content
of a given image and its social context. By analyzing what information a
specific method exploits to construct its tag relevance function and how such
information is exploited, this paper introduces a taxonomy to structure the
growing literature, understand the ingredients of the main works, clarify their
connections and difference, and recognize their merits and limitations. For a
head-to-head comparison between the state-of-the-art, a new experimental
protocol is presented, with training sets containing 10k, 100k and 1m images
and an evaluation on three test sets, contributed by various research groups.
Eleven representative works are implemented and evaluated. Putting all this
together, the survey aims to provide an overview of the past and foster
progress for the near future.Comment: to appear in ACM Computing Survey
Elastic net hypergraph learning for image clustering and semi-supervised classification
© 1992-2012 IEEE. Graph model is emerging as a very effective tool for learning the complex structures and relationships hidden in data. In general, the critical purpose of graph-oriented learning algorithms is to construct an informative graph for image clustering and classification tasks. In addition to the classical K -nearest-neighbor and r-neighborhood methods for graph construction, l1-graph and its variants are emerging methods for finding the neighboring samples of a center datum, where the corresponding ingoing edge weights are simultaneously derived by the sparse reconstruction coefficients of the remaining samples. However, the pairwise links of l1-graph are not capable of capturing the high-order relationships between the center datum and its prominent data in sparse reconstruction. Meanwhile, from the perspective of variable selection, the l1 norm sparse constraint, regarded as a LASSO model, tends to select only one datum from a group of data that are highly correlated and ignore the others. To simultaneously cope with these drawbacks, we propose a new elastic net hypergraph learning model, which consists of two steps. In the first step, the robust matrix elastic net model is constructed to find the canonically related samples in a somewhat greedy way, achieving the grouping effect by adding the l2 penalty to the l1 constraint. In the second step, hypergraph is used to represent the high order relationships between each datum and its prominent samples by regarding them as a hyperedge. Subsequently, hypergraph Laplacian matrix is constructed for further analysis. New hypergraph learning algorithms, including unsupervised clustering and multi-class semi-supervised classification, are then derived. Extensive experiments on face and handwriting databases demonstrate the effectiveness of the proposed method
- …