Active speaker detection (ASD) in videos with multiple speakers is a
challenging task as it requires learning effective audiovisual features and
spatial-temporal correlations over long temporal windows. In this paper, we
present SPELL, a novel spatial-temporal graph learning framework that can solve
complex tasks such as ASD. To this end, each person in a video frame is first
encoded in a unique node for that frame. Nodes corresponding to a single person
across frames are connected to encode their temporal dynamics. Nodes within a
frame are also connected to encode inter-person relationships. Thus, SPELL
reduces ASD to a node classification task. Importantly, SPELL is able to reason
over long temporal contexts for all nodes without relying on computationally
expensive fully connected graph neural networks. Through extensive experiments
on the AVA-ActiveSpeaker dataset, we demonstrate that learning graph-based
representations can significantly improve the active speaker detection
performance owing to its explicit spatial and temporal structure. SPELL
outperforms all previous state-of-the-art approaches while requiring
significantly lower memory and computational resources. Our code is publicly
available at https://github.com/SRA2/SPELLComment: ECCV 2022 camera ready (Supplementary videos: on ECVA soon). This
paper supersedes arXiv:2112.0147