9 research outputs found
Social relation recognition in egocentric photostreams
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.This paper proposes an approach to automatically categorize the social interactions of a user wearing a photo-camera (2fpm), by relying solely on what the camera is seeing. The problem is challenging due to the overwhelming complexity of social life and the extreme intra-class variability of social interactions captured under unconstrained conditions. We adopt the formalization proposed in Bugental’s social theory, that groups human relations into five social domains with related categories. Our method is a new deep learning architecture that exploits the hierarchical structure of the label space and relies on a set of social attributes estimated at frame level to provide a semantic representation of social interactions. Experimental results on the new EgoSocialRelation dataset demonstrate the effectiveness of our proposal.Peer ReviewedPostprint (author's final draft
Social Relation Recognition in Egocentric Photostreams
This paper proposes an approach to automatically categorize the social
interactions of a user wearing a photo-camera 2fpm, by relying solely on what
the camera is seeing. The problem is challenging due to the overwhelming
complexity of social life and the extreme intra-class variability of social
interactions captured under unconstrained conditions. We adopt the
formalization proposed in Bugental's social theory, that groups human relations
into five social domains with related categories. Our method is a new deep
learning architecture that exploits the hierarchical structure of the label
space and relies on a set of social attributes estimated at frame level to
provide a semantic representation of social interactions. Experimental results
on the new EgoSocialRelation dataset demonstrate the effectiveness of our
proposal.Comment: Accepted at ICIP 201
Unsupervised routine discovery in egocentric photo-streams
The routine of a person is defined by the occurrence of activities throughout
different days, and can directly affect the person's health. In this work, we
address the recognition of routine related days. To do so, we rely on
egocentric images, which are recorded by a wearable camera and allow to monitor
the life of the user from a first-person view perspective. We propose an
unsupervised model that identifies routine related days, following an outlier
detection approach. We test the proposed framework over a total of 72 days in
the form of photo-streams covering around 2 weeks of the life of 5 different
camera wearers. Our model achieves an average of 76% Accuracy and 68% Weighted
F-Score for all the users. Thus, we show that our framework is able to
recognise routine related days and opens the door to the understanding of the
behaviour of people
Egocentric video description based on temporally-linked sequences
Egocentric vision consists in acquiring images along the day from a first person point-of-view using wearable cameras. The automatic analysis of this information allows to discover daily patterns for improving the quality of life of the user. A natural topic that arises in egocentric vision is storytelling, that is, how to understand and tell the story relying behind the pictures. In this paper, we tackle storytelling as an egocentric sequences description problem. We propose a novel methodology that exploits information from temporally neighboring events, matching precisely the nature of egocentric sequences. Furthermore, we present a new method for multimodal data fusion consisting on a multi-input attention recurrent network. We also release the EDUB-SegDesc dataset. This is the first dataset for egocentric image sequences description, consisting of 1339 events with 3991 descriptions, from 55 days acquired by 11 people. Finally, we prove that our proposal outperforms classical attentional encoder-decoder methods for video description
Egocentric video description based on temporally-linked sequences
[EN] Egocentric vision consists in acquiring images along the day from a first person point-of-view using wearable cameras. The automatic analysis of this information allows to discover daily patterns for improving the quality of life of the user. A natural topic that arises in egocentric vision is storytelling, that is, how to understand and tell the story relying behind the pictures.
In this paper, we tackle storytelling as an egocentric sequences description problem. We propose a novel methodology that exploits information from temporally neighboring events, matching precisely the nature of egocentric sequences. Furthermore, we present a new method for multimodal data fusion consisting on a multi-input attention recurrent network. We also release the EDUB-SegDesc dataset. This is the first dataset for egocentric image sequences description, consisting of 1339 events with 3991 descriptions, from 55¿days acquired by 11 people. Finally, we prove that our proposal outperforms classical attentional encoder-decoder methods for video description.This work was partially founded by TIN2015-66951-C2, SGR 1219, CERCA, Grant 20141510 (Marato TV3), PrometeoII/2014/030 and R-MIPRCV network (TIN2014-54728-REDC). Petia Radeva is partially founded by ICREA Academia'2014. Marc Bolanos is partially founded by an FPU fellowship. We gratefully acknowledge the support of NVIDIA Corporation with the donation of a Titan X GPU used for this research. The funders had no role in the study design, data collection, analysis, and preparation of the manuscript.Bolaños, M.; Peris-Abril, Á.; Casacuberta Nolla, F.; Soler, S.; Radeva, P. (2018). Egocentric video description based on temporally-linked sequences. Journal of Visual Communication and Image Representation. 50:205-216. https://doi.org/10.1016/j.jvcir.2017.11.022S2052165