5,192 research outputs found
Spott : on-the-spot e-commerce for television using deep learning-based video analysis techniques
Spott is an innovative second screen mobile multimedia application which offers viewers relevant information on objects (e.g., clothing, furniture, food) they see and like on their television screens. The application enables interaction between TV audiences and brands, so producers and advertisers can offer potential consumers tailored promotions, e-shop items, and/or free samples. In line with the current views on innovation management, the technological excellence of the Spott application is coupled with iterative user involvement throughout the entire development process. This article discusses both of these aspects and how they impact each other. First, we focus on the technological building blocks that facilitate the (semi-) automatic interactive tagging process of objects in the video streams. The majority of these building blocks extensively make use of novel and state-of-the-art deep learning concepts and methodologies. We show how these deep learning based video analysis techniques facilitate video summarization, semantic keyframe clustering, and (similar) object retrieval. Secondly, we provide insights in user tests that have been performed to evaluate and optimize the application's user experience. The lessons learned from these open field tests have already been an essential input in the technology development and will further shape the future modifications to the Spott application
Recommended from our members
Hierarchical video summarisation in reference frame subspace
In this paper, a hierarchical video structure summarization approach using Laplacian Eigenmap is proposed, where a small set of reference frames is selected from the video sequence to form a reference subspace to measure the dissimilarity between two arbitrary frames. In the proposed summarization scheme, the shot-level key frames are first detected from the continuity of inter-frame dissimilarity, and the sub-shot level and scene level representative frames are then summarized by using k-mean clustering. The experiment is carried on both test videos and movies, and the results show that in comparison with a similar approach using latent semantic analysis, the proposed approach using Laplacian Eigenmap can achieve a better recall rate in keyframe detection, and gives an efficient hierarchical summarization at sub shot, shot and scene levels subsequently
Video summarization by group scoring
In this paper a new model for user-centered video summarization is presented. Involvement of more than one expert in generating the final video summary should be regarded as the main use case for this algorithm. This approach consists of three major steps. First, the video frames are scored by a group of operators. Next, these assigned scores are averaged to produce a singular value for each frame and lastly, the highest scored video frames alongside the corresponding audio and textual contents are extracted to be inserted into the summary. The effectiveness of this approach has been evaluated by comparing the video summaries generated by this system against the results from a number of automatic summarization tools that use different modalities for abstraction
An Enhanced Method For Evaluating Automatic Video Summaries
Evaluation of automatic video summaries is a challenging problem. In the past
years, some evaluation methods are presented that utilize only a single feature
like color feature to detect similarity between automatic video summaries and
ground-truth user summaries. One of the drawbacks of using a single feature is
that sometimes it gives a false similarity detection which makes the assessment
of the quality of the generated video summary less perceptual and not accurate.
In this paper, a novel method for evaluating automatic video summaries is
presented. This method is based on comparing automatic video summaries
generated by video summarization techniques with ground-truth user summaries.
The objective of this evaluation method is to quantify the quality of video
summaries, and allow comparing different video summarization techniques
utilizing both color and texture features of the video frames and using the
Bhattacharya distance as a dissimilarity measure due to its advantages. Our
Experiments show that the proposed evaluation method overcomes the drawbacks of
other methods and gives a more perceptual evaluation of the quality of the
automatic video summaries.Comment: This paper has been withdrawn by the author due to some errors and
incomplete stud
Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings
In this paper we present a novel interactive multimodal learning system,
which facilitates search and exploration in large networks of social multimedia
users. It allows the analyst to identify and select users of interest, and to
find similar users in an interactive learning setting. Our approach is based on
novel multimodal representations of users, words and concepts, which we
simultaneously learn by deploying a general-purpose neural embedding model. We
show these representations to be useful not only for categorizing users, but
also for automatically generating user and community profiles. Inspired by
traditional summarization approaches, we create the profiles by selecting
diverse and representative content from all available modalities, i.e. the
text, image and user modality. The usefulness of the approach is evaluated
using artificial actors, which simulate user behavior in a relevance feedback
scenario. Multiple experiments were conducted in order to evaluate the quality
of our multimodal representations, to compare different embedding strategies,
and to determine the importance of different modalities. We demonstrate the
capabilities of the proposed approach on two different multimedia collections
originating from the violent online extremism forum Stormfront and the
microblogging platform Twitter, which are particularly interesting due to the
high semantic level of the discussions they feature
VSCAN: An Enhanced Video Summarization using Density-based Spatial Clustering
In this paper, we present VSCAN, a novel approach for generating static video
summaries. This approach is based on a modified DBSCAN clustering algorithm to
summarize the video content utilizing both color and texture features of the
video frames. The paper also introduces an enhanced evaluation method that
depends on color and texture features. Video Summaries generated by VSCAN are
compared with summaries generated by other approaches found in the literature
and those created by users. Experimental results indicate that the video
summaries generated by VSCAN have a higher quality than those generated by
other approaches.Comment: arXiv admin note: substantial text overlap with arXiv:1401.3590 by
other authors without attributio
- …