CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

K-Space at TRECVid 2007

In this paper we describe K-Space participation in TRECVid 2007. K-Space participated in two tasks, high-level feature extraction and interactive search. We present our approaches for each of these activities and provide a brief analysis of our results. Our high-level feature submission utilized multi-modal low-level features which included visual, audio and temporal elements. Specific concept detectors (such as Face detectors) developed by K-Space partners were also used. We experimented with different machine learning approaches including logistic regression and support vector machines (SVM). Finally we also experimented with both early and late fusion for feature combination. This year we also participated in interactive search, submitting 6 runs. We developed two interfaces which both utilized the same retrieval functionality. Our objective was to measure the effect of context, which was supported to different degrees in each interface, on user performance. The first of the two systems was a ‘shot’ based interface, where the results from a query were presented as a ranked list of shots. The second interface was ‘broadcast’ based, where results were presented as a ranked list of broadcasts. Both systems made use of the outputs of our high-level feature submission as well as low-level visual features

Action Recognition in Videos: from Motion Capture Labs to the Web

This paper presents a survey of human action recognition approaches based on visual data recorded from a single video camera. We propose an organizing framework which puts in evidence the evolution of the area, with techniques moving from heavily constrained motion capture scenarios towards more challenging, realistic, "in the wild" videos. The proposed organization is based on the representation used as input for the recognition task, emphasizing the hypothesis assumed and thus, the constraints imposed on the type of video that each technique is able to address. Expliciting the hypothesis and constraints makes the framework particularly useful to select a method, given an application. Another advantage of the proposed organization is that it allows categorizing newest approaches seamlessly with traditional ones, while providing an insightful perspective of the evolution of the action recognition task up to now. That perspective is the basis for the discussion in the end of the paper, where we also present the main open issues in the area.Comment: Preprint submitted to CVIU, survey paper, 46 pages, 2 figures, 4 table

Fast ConvNets Using Group-wise Brain Damage

We revisit the idea of brain damage, i.e. the pruning of the coefficients of a neural network, and suggest how brain damage can be modified and used to speedup convolutional layers. The approach uses the fact that many efficient implementations reduce generalized convolutions to matrix multiplications. The suggested brain damage process prunes the convolutional kernel tensor in a group-wise fashion by adding group-sparsity regularization to the standard training process. After such group-wise pruning, convolutions can be reduced to multiplications of thinned dense matrices, which leads to speedup. In the comparison on AlexNet, the method achieves very competitive performance

Meta-KANSEI modeling with Valence-Arousal fMRI dataset of brain

Background: Traditional KANSEI methodology is an important tool in the field of psychology to comprehend the concepts and meanings; it mainly focusses on semantic differential methods. Valence-Arousal is regarded as a reflection of the KANSEI adjectives, which is the core concept in the theory of effective dimensions for brain recognition. From previous studies, it has been found that brain fMRI datasets can contain significant information related to Valence and Arousal. Methods: In this current work, a Valence-Arousal based meta-KANSEI modeling method is proposed to improve the traditional KANSEI presentation. Functional Magnetic Resonance Imaging (fMRI) was used to acquire the response dataset of Valence-Arousal of the brain in the amygdala and orbital frontal cortex respectively. In order to validate the feasibility of the proposed modeling method, the dataset was processed under dimension reduction by using Kernel Density Estimation (KDE) based segmentation and Mean Shift (MS) clustering. Furthermore, Affective Norm English Words (ANEW) by IAPS (International Affective Picture System) were used for comparison and analysis. The data sets from fMRI and ANEW under four KANSEI adjectives of angry, happy, sad and pleasant were processed by the Fuzzy C-Means (FCM) algorithm. Finally, a defined distance based on similarity computing was adopted for these two data sets. Results: The results illustrate that the proposed model is feasible and has better stability per the normal distribution plotting of the distance. The effectiveness of the experimental methods proposed in the current work was higher than in the literature. Conclusions: mean shift can be used to cluster and central points based meta-KANSEI model combining with the advantages of a variety of existing intelligent processing methods are expected to shift the KANSEI Engineering (KE) research into the medical imaging field