Data mining for activity extraction in video data

International audienceThe exploration of large video data is a task which is now possible because of the advances made on object detection and tracking. Data mining techniques such as clustering are typically employed. Such techniques have mainly been applied for segmentation/indexation of video but knowledge extraction of the activity contained in the video has been only partially addressed. In this paper we present how video information is processed with the ultimate aim to achieve knowledge discovery of people activity in the video. First, objects of interest are detected in real time. Then, in an off-line process, we aim to perform knowledge discovery at two stages: 1) finding the main trajectory patterns of people in the video. 2) finding patterns of interaction between people and contextual objects in the scene. An agglomerative hierarchical clustering is employed at each stage. We present results obtained on real videos of the Torino metro (Italy).L'exploration de larges bases de données vidéo est une tâche qui devient possible grâce aux avancées techniques dans la détection et le suivi d'objets. Les méthodes de fouille d'information comme le clustering sont typiquement employées. Celles-ci ont été principalement appliquées pour la segmentation/indexation vidéo mais l'extraction de connaissances sur l'activité présente dans la vidéo a été seulement partiellement adressée. Dans cet article nous présentons comment ces techniques peuvent être utilisées pour traiter de l'information vidéo pour l'extraction de connaissances. Tout d'abord, les objets d'intérêt sont détectés en temps réel. Ensuite, dans un traitement supplémentaire, nous recherchons à extraire des nouvelles connaissances en deux etapes : 1) extraction des motifs caractéristiques des trajectoires des personnes dans la vidéo. 2) extraction des motifs d'interaction entre les personnes et les objets contextuels dans la scène. Dans les deux cas, nous appliquons un clustering hiérarchique agglomératif. Nous présentons des résultats obtenus sur des vidéos du metro de Turin (Italie

Data Mining on large Video Recordings

National audienceThe exploration of large video data is a task which is now possible because of the advances made on object detection and tracking. Data mining techniques such as clustering are typically employed. Such techniques have mainly been applied for segmentation/indexation of video but knowledge extraction on the activity contained in the video has been only partially addressed. In this paper we present how video information is processed with the ultimate aim to achieve knowledge discovery of people activity in the video. First, objects of interest are detected in real time. Then, in an off-line process, the information related to detected objects is set into a model format suitable for knowledge representation and discovery. We then apply two clustering processes: 1) Agglomerative hierarchical clustering to find the main trajectory patterns of people in the video 2) Relational analysis clustering, which we employ to extract spatio-temporal relations between people and contextual objects in the scene. We present results obtained on real videos of the Torino metro (Italy)

A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews

Despite the recent advances in opinion mining for written reviews, few works have tackled the problem on other sources of reviews. In light of this issue, we propose a multi-modal approach for mining fine-grained opinions from video reviews that is able to determine the aspects of the item under review that are being discussed and the sentiment orientation towards them. Our approach works at the sentence level without the need for time annotations and uses features derived from the audio, video and language transcriptions of its contents. We evaluate our approach on two datasets and show that leveraging the video and audio modalities consistently provides increased performance over text-only baselines, providing evidence these extra modalities are key in better understanding video reviews.Comment: Second Grand Challenge and Workshop on Multimodal Language ACL 202