Article thumbnail

Speech2Action: Cross-modal Supervision for Action Recognition

By Arsha Nagrani, Chen Sun, David Ross, Rahul Sukthankar, Cordelia Schmid and Andrew Zisserman


Is it possible to guess human action from dialogue alone? In this work we investigate the link between spoken words and actions in movies. We note that movie screenplays describe actions, as well as contain the speech of characters and hence can be used to learn this correlation with no additional supervision. We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments. We then apply this model to the speech segments of a large unlabelled movie corpus (188M speech segments from 288K movies). Using the predictions of this model, we obtain weak action labels for over 800K video clips. By training on these video clips, we demonstrate superior action recognition performance on standard action recognition benchmarks, without using a single manually labelled action example.Comment: Accepted to CVPR 202

Topics: Computer Science - Computer Vision and Pattern Recognition
Year: 2020
OAI identifier:

Suggested articles

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.