25,662 research outputs found
A Hybrid Graph Network for Complex Activity Detection in Video
Interpretation and understanding of video presents a challenging computer
vision task in numerous fields - e.g. autonomous driving and sports analytics.
Existing approaches to interpreting the actions taking place within a video
clip are based upon Temporal Action Localisation (TAL), which typically
identifies short-term actions. The emerging field of Complex Activity Detection
(CompAD) extends this analysis to long-term activities, with a deeper
understanding obtained by modelling the internal structure of a complex
activity taking place within the video. We address the CompAD problem using a
hybrid graph neural network which combines attention applied to a graph
encoding the local (short-term) dynamic scene with a temporal graph modelling
the overall long-duration activity. Our approach is as follows: i) Firstly, we
propose a novel feature extraction technique which, for each video snippet,
generates spatiotemporal `tubes' for the active elements (`agents') in the
(local) scene by detecting individual objects, tracking them and then
extracting 3D features from all the agent tubes as well as the overall scene.
ii) Next, we construct a local scene graph where each node (representing either
an agent tube or the scene) is connected to all other nodes. Attention is then
applied to this graph to obtain an overall representation of the local dynamic
scene. iii) Finally, all local scene graph representations are interconnected
via a temporal graph, to estimate the complex activity class together with its
start and end time. The proposed framework outperforms all previous
state-of-the-art methods on all three datasets including ActivityNet-1.3,
Thumos-14, and ROAD.Comment: This paper is Accepted at WACV 202
- …