3,994 research outputs found
Socializing the Semantic Gap: A Comparative Survey on Image Tag Assignment, Refinement and Retrieval
Where previous reviews on content-based image retrieval emphasize on what can
be seen in an image to bridge the semantic gap, this survey considers what
people tag about an image. A comprehensive treatise of three closely linked
problems, i.e., image tag assignment, refinement, and tag-based image retrieval
is presented. While existing works vary in terms of their targeted tasks and
methodology, they rely on the key functionality of tag relevance, i.e.
estimating the relevance of a specific tag with respect to the visual content
of a given image and its social context. By analyzing what information a
specific method exploits to construct its tag relevance function and how such
information is exploited, this paper introduces a taxonomy to structure the
growing literature, understand the ingredients of the main works, clarify their
connections and difference, and recognize their merits and limitations. For a
head-to-head comparison between the state-of-the-art, a new experimental
protocol is presented, with training sets containing 10k, 100k and 1m images
and an evaluation on three test sets, contributed by various research groups.
Eleven representative works are implemented and evaluated. Putting all this
together, the survey aims to provide an overview of the past and foster
progress for the near future.Comment: to appear in ACM Computing Survey
Cultural Event Recognition with Visual ConvNets and Temporal Models
This paper presents our contribution to the ChaLearn Challenge 2015 on
Cultural Event Classification. The challenge in this task is to automatically
classify images from 50 different cultural events. Our solution is based on the
combination of visual features extracted from convolutional neural networks
with temporal information using a hierarchical classifier scheme. We extract
visual features from the last three fully connected layers of both CaffeNet
(pretrained with ImageNet) and our fine tuned version for the ChaLearn
challenge. We propose a late fusion strategy that trains a separate low-level
SVM on each of the extracted neural codes. The class predictions of the
low-level SVMs form the input to a higher level SVM, which gives the final
event scores. We achieve our best result by adding a temporal refinement step
into our classification scheme, which is applied directly to the output of each
low-level SVM. Our approach penalizes high classification scores based on
visual features when their time stamp does not match well an event-specific
temporal distribution learned from the training and validation data. Our system
achieved the second best result in the ChaLearn Challenge 2015 on Cultural
Event Classification with a mean average precision of 0.767 on the test set.Comment: Initial version of the paper accepted at the CVPR Workshop ChaLearn
Looking at People 201
A Review on Personalized Tag based Image based Search Engines
The development of social media based on Web 2.0, amounts of images and videos spring up everywhere on the Internet. This phenomenon has brought great challenges to multimedia storage, indexing and retrieval. Generally speaking, tag-based image search is more commonly used in social media than content based image retrieval and content understanding. Thanks to the low relevance and diversity performance of initial retrieval results, the ranking problem in the tag-based image retrieval has gained researchers� wide attention. We will review some of techniques proposed by different authors for image retrieval in this paper
Image Understanding by Socializing the Semantic Gap
Several technological developments like the Internet, mobile devices and Social Networks have spurred the sharing of images in unprecedented volumes, making tagging and commenting a common habit. Despite the recent progress in image analysis, the problem of Semantic Gap still hinders machines in fully understand the rich semantic of a shared photo. In this book, we tackle this problem by exploiting social network contributions. A comprehensive treatise of three linked problems on image annotation is presented, with a novel experimental protocol used to test eleven state-of-the-art methods. Three novel approaches to annotate, under stand the sentiment and predict the popularity of an image are presented. We conclude with the many challenges and opportunities ahead for the multimedia community
Image Retrieval within Augmented Reality
Die vorliegende Arbeit untersucht das Potenzial von Augmented Reality zur Verbesserung von Image Retrieval Prozessen. Herausforderungen in Design und Gebrauchstauglichkeit wurden für beide Forschungsbereiche dargelegt und genutzt, um Designziele für Konzepte zu entwerfen. Eine Taxonomie für Image Retrieval in Augmented Reality wurde basierend auf der Forschungsarbeit entworfen und eingesetzt, um verwandte Arbeiten und generelle Ideen für Interaktionsmöglichkeiten zu strukturieren. Basierend auf der Taxonomie wurden Anwendungsszenarien als weitere Anforderungen für Konzepte formuliert. Mit Hilfe der generellen Ideen und Anforderungen wurden zwei umfassende Konzepte für Image Retrieval in Augmented Reality ausgearbeitet. Eins der Konzepte wurde auf einer Microsoft HoloLens umgesetzt und in einer Nutzerstudie evaluiert. Die Studie zeigt, dass das Konzept grundsätzlich positiv aufgenommen wurde und bietet Erkenntnisse über unterschiedliches Verhalten im Raum und verschiedene Suchstrategien bei der Durchführung von Image Retrieval in der erweiterten Realität.:1 Introduction
1.1 Motivation and Problem Statement
1.1.1 Augmented Reality and Head-Mounted Displays
1.1.2 Image Retrieval
1.1.3 Image Retrieval within Augmented Reality
1.2 Thesis Structure
2 Foundations of Image Retrieval and Augmented Reality
2.1 Foundations of Image Retrieval
2.1.1 Definition of Image Retrieval
2.1.2 Classification of Image Retrieval Systems
2.1.3 Design and Usability in Image Retrieval
2.2 Foundations of Augmented Reality
2.2.1 Definition of Augmented Reality
2.2.2 Augmented Reality Design and Usability
2.3 Taxonomy for Image Retrieval within Augmented Reality
2.3.1 Session Parameters
2.3.2 Interaction Process
2.3.3 Summary of the Taxonomy
3 Concepts for Image Retrieval within Augmented Reality
3.1 Related Work
3.1.1 Natural Query Specification
3.1.2 Situated Result Visualization
3.1.3 3D Result Interaction
3.1.4 Summary of Related Work
3.2 Basic Interaction Concepts for Image Retrieval in Augmented Reality
3.2.1 Natural Query Specification
3.2.2 Situated Result Visualization
3.2.3 3D Result Interaction
3.3 Requirements for Comprehensive Concepts
3.3.1 Design Goals
3.3.2 Application Scenarios
3.4 Comprehensive Concepts
3.4.1 Tangible Query Workbench
3.4.2 Situated Photograph Queries
3.4.3 Conformance of Concept Requirements
4 Prototypic Implementation of Situated Photograph Queries
4.1 Implementation Design
4.1.1 Implementation Process
4.1.2 Structure of the Implementation
4.2 Developer and User Manual
4.2.1 Setup of the Prototype
4.2.2 Usage of the Prototype
4.3 Discussion of the Prototype
5 Evaluation of Prototype and Concept by User Study
5.1 Design of the User Study
5.1.1 Usability Testing
5.1.2 Questionnaire
5.2 Results
5.2.1 Logging of User Behavior
5.2.2 Rating through Likert Scales
5.2.3 Free Text Answers and Remarks during the Study
5.2.4 Observations during the Study
5.2.5 Discussion of Results
6 Conclusion
6.1 Summary of the Present Work
6.2 Outlook on Further WorkThe present work investigates the potential of augmented reality for improving the image retrieval process. Design and usability challenges were identified for both fields of research in order to formulate design goals for the development of concepts. A taxonomy for image retrieval within augmented reality was elaborated based on research work and used to structure related work and basic ideas for interaction. Based on the taxonomy, application scenarios were formulated as further requirements for concepts. Using the basic interaction ideas and the requirements, two comprehensive concepts for image retrieval within augmented reality were elaborated. One of the concepts was implemented using a Microsoft HoloLens and evaluated in a user study. The study showed that the concept was rated generally positive by the users and provided insight in different spatial behavior and search strategies when practicing image retrieval in augmented reality.:1 Introduction
1.1 Motivation and Problem Statement
1.1.1 Augmented Reality and Head-Mounted Displays
1.1.2 Image Retrieval
1.1.3 Image Retrieval within Augmented Reality
1.2 Thesis Structure
2 Foundations of Image Retrieval and Augmented Reality
2.1 Foundations of Image Retrieval
2.1.1 Definition of Image Retrieval
2.1.2 Classification of Image Retrieval Systems
2.1.3 Design and Usability in Image Retrieval
2.2 Foundations of Augmented Reality
2.2.1 Definition of Augmented Reality
2.2.2 Augmented Reality Design and Usability
2.3 Taxonomy for Image Retrieval within Augmented Reality
2.3.1 Session Parameters
2.3.2 Interaction Process
2.3.3 Summary of the Taxonomy
3 Concepts for Image Retrieval within Augmented Reality
3.1 Related Work
3.1.1 Natural Query Specification
3.1.2 Situated Result Visualization
3.1.3 3D Result Interaction
3.1.4 Summary of Related Work
3.2 Basic Interaction Concepts for Image Retrieval in Augmented Reality
3.2.1 Natural Query Specification
3.2.2 Situated Result Visualization
3.2.3 3D Result Interaction
3.3 Requirements for Comprehensive Concepts
3.3.1 Design Goals
3.3.2 Application Scenarios
3.4 Comprehensive Concepts
3.4.1 Tangible Query Workbench
3.4.2 Situated Photograph Queries
3.4.3 Conformance of Concept Requirements
4 Prototypic Implementation of Situated Photograph Queries
4.1 Implementation Design
4.1.1 Implementation Process
4.1.2 Structure of the Implementation
4.2 Developer and User Manual
4.2.1 Setup of the Prototype
4.2.2 Usage of the Prototype
4.3 Discussion of the Prototype
5 Evaluation of Prototype and Concept by User Study
5.1 Design of the User Study
5.1.1 Usability Testing
5.1.2 Questionnaire
5.2 Results
5.2.1 Logging of User Behavior
5.2.2 Rating through Likert Scales
5.2.3 Free Text Answers and Remarks during the Study
5.2.4 Observations during the Study
5.2.5 Discussion of Results
6 Conclusion
6.1 Summary of the Present Work
6.2 Outlook on Further Wor
- …