Deliverable D1.4 Visual, text and audio information analysis for hypervideo, final release

Having extensively evaluated the performance of the technologies included in the first release of WP1 multimedia analysis tools, using content from the LinkedTV scenarios and by participating in international benchmarking activities, concrete decisions regarding the appropriateness and the importance of each individual method or combination of methods were made, which, combined with an updated list of information needs for each scenario, led to a new set of analysis requirements that had to be addressed through the release of the final set of analysis techniques of WP1. To this end, coordinated efforts on three directions, including (a) the improvement of a number of methods in terms of accuracy and time efficiency, (b) the development of new technologies and (c) the definition of synergies between methods for obtaining new types of information via multimodal processing, resulted in the final bunch of multimedia analysis methods for video hyperlinking. Moreover, the different developed analysis modules have been integrated into a web-based infrastructure, allowing the fully automatic linking of the multitude of WP1 technologies and the overall LinkedTV platform

Deliverable D2.7 Final Linked Media Layer and Evaluation

This deliverable presents the evaluation of content annotation and content enrichment systems that are part of the final tool set developed within the LinkedTV consortium. The evaluations were performed on both the Linked News and Linked Culture trial content, as well as on other content annotated for this purpose. The evaluation spans three languages: German (Linked News), Dutch (Linked Culture) and English. Selected algorithms and tools were also subject to benchmarking in two international contests: MediaEval 2014 and TAC’14. Additionally, the Microposts 2015 NEEL Challenge is being organized with the support of LinkedTV

Conceptual Feedback for Semantic Multimedia Indexing

International audienceIn this paper, we consider the problem of automat- ically detecting a large number of visual concepts in images or video shots. State of the art systems involve feature (descriptor) extraction, classification (supervised learning) and fusion when several descriptors and/or classifiers are used. Though direct multi-label approaches are considered in some works, detection scores are often computed independently for each target concept. We propose here a method that we call "conceptual feedback" for improving the overall detection performance that implicitly takes into account the relations between concepts. The vector of normalized detection scores is added to the pool of available descriptors. It is then processed just as the other descriptors for the normalization, optimization and classification steps. The resulting detection scores are finally fused with the already avail- able detection scores obtained with the original descriptors. The feedback of the global detection scores in the pool of descriptors can be iterated several times. It is also compatible with the use of the temporal context that also improves the overall performance by taking into account the local homogeneity of video contents. The method has been evaluated in the context of the TRECVID 2012 semantic indexing task involving the detection of 346 visual or multimodal concepts. Combined with temporal re-scoring, the proposed method increased the global system performance (MAP) from 0.2613 to 0.3014 (+15.3% of relative improvement) while the temporal re-scoring alone increased it only from 0.2613 to 0.2691 (+3.0%)

Extended conceptual feedback for semantic multimedia indexing

International audienceIn this paper, we consider the problem of automatically detecting a large number of visual concepts in images or video shots. State of the art systems generally involve feature (descriptor) extraction, classification (supervised learning) and fusion when several descriptors and/or classifiers are used. Though direct multi-label approaches are considered in some works, detection scores are often computed independently for each target concept. We propose a method that we call "conceptual feedback" which implicitly takes into account the relations between concepts to improve the overall concepts detection performance. A conceptual descriptor is built from the system's output scores and fed back by adding it to the pool of already available descriptors. Our proposal can be iterated several times. Moreover, we propose three extensions of our method. Firstly, a weighting of the conceptual dimensions is performed to give more importance to concepts which are more correlated to the target concept. Secondly, an explicit selection of a set of concepts that are semantically or statically related to the target concept is introduced. For video indexing, we propose a third extension which integrates the temporal dimension in the feedback process by taking into account simultaneously the conceptual and the temporal dimensions to build the high-level descriptor. Our proposals have been evaluated in the context of the TRECVid 2012 semantic indexing task involving the detection of 346 visual or multi-modal concepts. Overall, combined with temporal re-scoring, the proposed method increased the global system performance (MAP) from 0.2613 to 0.3082 (+ 17.9 % of relative improvement) while the temporal re-scoring alone increased it only from 0.2613 to 0.2691 (+ 3.0 %)