Going Deeper into First-Person Activity Recognition

We bring together ideas from recent work on feature design for egocentric action recognition under one framework by exploring the use of deep convolutional neural networks (CNN). Recent work has shown that features such as hand appearance, object attributes, local hand motion and camera ego-motion are important for characterizing first-person actions. To integrate these ideas under one framework, we propose a twin stream network architecture, where one stream analyzes appearance information and the other stream analyzes motion information. Our appearance stream encodes prior knowledge of the egocentric paradigm by explicitly training the network to segment hands and localize objects. By visualizing certain neuron activation of our network, we show that our proposed architecture naturally learns features that capture object attributes and hand-object configurations. Our extensive experiments on benchmark egocentric action datasets show that our deep architecture enables recognition rates that significantly outperform state-of-the-art techniques -- an average $6.6\%$ increase in accuracy over all datasets. Furthermore, by learning to recognize objects, actions and activities jointly, the performance of individual recognition tasks also increase by $30\%$ (actions) and $14\%$ (objects). We also include the results of extensive ablative analysis to highlight the importance of network design decisions.

The multisensory body revealed through its cast shadows

One key issue when conceiving the body as a multisensory object is how the cognitive system integrates visible instances of the self and other bodies with one\u2019s own somatosensory processing, to achieve self-recognition and body ownership. Recent research has strongly suggested that shadows cast by our own body have a special status for cognitive processing, directing attention to the body in a fast and highly specific manner. The aim of the present article is to review the most recent scientific contributions addressing how body shadows affect both sensory/perceptual and attentional processes. The review examines three main points: (1) body shadows as a special window to investigate the construction of multisensory body perception; (2) experimental paradigms and related findings; (3) open questions and future trajectories. The reviewed literature suggests that shadows cast by one\u2019s own body promote binding between personal and extrapersonal space and elicit automatic orienting of attention toward the bodypart casting the shadow. Future research should address whether the effects exerted by body shadows are similar to those observed when observers are exposed to other visual instances of their body. The results will further clarify the processes underlying the merging of vision and somatosensation when creating body representations

Can't touch this: the first-person perspective provides privileged access to predictions of sensory action outcomes.

RCUK Open Access funded. ESRC ES/J019178/1Previous studies have shown that viewing others in pain activates cortical somatosensory processing areas and facilitates the detection of tactile targets. It has been suggested that such shared representations have evolved to enable us to better understand the actions and intentions of others. If this is the case, the effects of observing others in pain should be obtained from a range of viewing perspectives. Therefore, the current study examined the behavioral effects of observed grasps of painful and nonpainful objects from both a first- and third-person perspective. In the first-person perspective, a participant was faster to detect a tactile target delivered to their own hand when viewing painful grasping actions, compared with all nonpainful actions. However, this effect was not revealed in the third-person perspective. The combination of action and object information to predict the painful consequences of another person's actions when viewed from the first-person perspective, but not the third-person perspective, argues against a mechanism ostensibly evolved to understand the actions of others

What Can I Do Around Here? Deep Functional Scene Understanding for Cognitive Robots

For robots that have the capability to interact with the physical environment through their end effectors, understanding the surrounding scenes is not merely a task of image classification or object recognition. To perform actual tasks, it is critical for the robot to have a functional understanding of the visual scene. Here, we address the problem of localizing and recognition of functional areas from an arbitrary indoor scene, formulated as a two-stage deep learning based detection pipeline. A new scene functionality testing-bed, which is complied from two publicly available indoor scene datasets, is used for evaluation. Our method is evaluated quantitatively on the new dataset, demonstrating the ability to perform efficient recognition of functional areas from arbitrary indoor scenes. We also demonstrate that our detection model can be generalized onto novel indoor scenes by cross validating it with the images from two different datasets

Hitting is male, giving is female. Automatic imitation and complementarity during action observation

Is somebody going to hurt us? We draw back. The present study investigates using behavioral measures the interplay between imitative and complementary actions activated while observing female/male hands performing different actions. Female and male participants were required to discriminate the gender of biologically and artificially colored hands that displayed both individual (grasping) and social (giving and punching) actions. Biological hands evoked automatic imitation, while hands of different gender activated complementary mechanisms. Furthermore, responses reflected gender stereotypes: giving actions were more associated to females, punching actions to males. Results have implications for studies on social stereotyping, and for research on action observation, showing that the mirror neuron system resonates in both an imitative and complementary fashion

Using action understanding to understand the left inferior parietal cortex in the human brain

Published in final edited form as: Brain Res. 2014 September 25; 1582: 64–76. doi:10.1016/j.brainres.2014.07.035.Humans have a sophisticated knowledge of the actions that can be performed with objects. In an fMRI study we tried to establish whether this depends on areas that are homologous with the inferior parietal cortex (area PFG) in macaque monkeys. Cells have been described in area PFG that discharge differentially depending upon whether the observer sees an object being brought to the mouth or put in a container. In our study the observers saw videos in which the use of different objects was demonstrated in pantomime; and after viewing the videos, the subject had to pick the object that was appropriate to the pantomime. We found a cluster of activated voxels in parietal areas PFop and PFt and this cluster was greater in the left hemisphere than in the right. We suggest a mechanism that could account for this asymmetry, relate our results to handedness and suggest that they shed light on the human syndrome of apraxia. Finally, we suggest that during the evolution of the hominids, this same pantomime mechanism could have been used to ‘name’ or request objects.We thank Steve Wise for very detailed comments on a draft of this paper. We thank Rogier Mars for help with identifying the areas that were activated in parietal cortex and for comments on a draft of this paper. Finally, we thank Michael Nahhas for help with the imaging figures. This work was supported in part by the NIH grant RO1NS064100 to LMV. (RO1NS064100 - NIH)Accepted manuscrip