Detecting events and key actors in multi-person videos

Multi-person event recognition is a challenging task, often with many people active in the scene but only a small subset contributing to an actual event. In this paper, we propose a model which learns to detect events in such videos while automatically "attending" to the people responsible for the event. Our model does not use explicit annotations regarding who or where those people are during training and testing. In particular, we track people in videos and use a recurrent neural network (RNN) to represent the track features. We learn time-varying attention weights to combine these features at each time-instant. The attended features are then processed using another RNN for event detection/classification. Since most video datasets with multiple people are restricted to a small number of videos, we also collected a new basketball dataset comprising 257 basketball games with 14K event annotations corresponding to 11 event classes. Our model outperforms state-of-the-art methods for both event classification and detection on this new dataset. Additionally, we show that the attention mechanism is able to consistently localize the relevant players.Comment: Accepted for publication in CVPR'1

From pixels to affect : a study on games and player experience

Is it possible to predict the affect of a user just by observing her behavioral interaction through a video? How can we, for instance, predict a user’s arousal in games by merely looking at the screen during play? In this paper we address these questions by employing three dissimilar deep convolutional neural network architectures in our attempt to learn the underlying mapping between video streams of gameplay and the player’s arousal. We test the algorithms in an annotated dataset of 50 gameplay videos of a survival shooter game and evaluate the deep learned models’ capacity to classify high vs low arousal levels. Our key findings with the demanding leave-onevideo- out validation method reveal accuracies of over 78% on average and 98% at best. While this study focuses on games and player experience as a test domain, the findings and methodology are directly relevant to any affective computing area, introducing a general and user-agnostic approach for modeling affect.This paper is funded, in part, by the H2020 project Com N Play Science (project no: 787476).peer-reviewe

MammalNet: A Large-scale Video Benchmark for Mammal Recognition and Behavior Understanding

Monitoring animal behavior can facilitate conservation efforts by providing key insights into wildlife health, population status, and ecosystem function. Automatic recognition of animals and their behaviors is critical for capitalizing on the large unlabeled datasets generated by modern video devices and for accelerating monitoring efforts at scale. However, the development of automated recognition systems is currently hindered by a lack of appropriately labeled datasets. Existing video datasets 1) do not classify animals according to established biological taxonomies; 2) are too small to facilitate large-scale behavioral studies and are often limited to a single species; and 3) do not feature temporally localized annotations and therefore do not facilitate localization of targeted behaviors within longer video sequences. Thus, we propose MammalNet, a new large-scale animal behavior dataset with taxonomy-guided annotations of mammals and their common behaviors. MammalNet contains over 18K videos totaling 539 hours, which is ~10 times larger than the largest existing animal behavior dataset. It covers 17 orders, 69 families, and 173 mammal categories for animal categorization and captures 12 high-level animal behaviors that received focus in previous animal behavior studies. We establish three benchmarks on MammalNet: standard animal and behavior recognition, compositional low-shot animal and behavior recognition, and behavior detection. Our dataset and code have been made available at: https://mammal-net.github.io.Comment: CVPR 2023 proceedin

β1 integrin activates Rac1 in Schwann cells to generate radial lamellae during axonal sorting and myelination

Myelin is a multispiraled extension of glial membrane that surrounds axons. How glia extend a surface many-fold larger than their body is poorly understood. Schwann cells are peripheral glia and insert radial cytoplasmic extensions into bundles of axons to sort, ensheath, and myelinate them. Laminins and β1 integrins are required for axonal sorting, but the downstream signals are largely unknown. We show that Schwann cells devoid of β1 integrin migrate to and elongate on axons but cannot extend radial lamellae of cytoplasm, similar to cells with low Rac1 activation. Accordingly, active Rac1 is decreased in β1 integrin–null nerves, inhibiting Rac1 activity decreases radial lamellae in Schwann cells, and ablating Rac1 in Schwann cells of transgenic mice delays axonal sorting and impairs myelination. Finally, expressing active Rac1 in β1 integrin–null nerves improves sorting. Thus, increased activation of Rac1 by β1 integrins allows Schwann cells to switch from migration/elongation to the extension of radial membranes required for axonal sorting and myelination