1 research outputs found
MAAS: Multi-modal Assignation for Active Speaker Detection
Active speaker detection requires a solid integration of multi-modal cues.
While individual modalities can approximate a solution, accurate predictions
can only be achieved by explicitly fusing the audio and visual features and
modeling their temporal progression. Despite its inherent muti-modal nature,
current methods still focus on modeling and fusing short-term audiovisual
features for individual speakers, often at frame level. In this paper we
present a novel approach to active speaker detection that directly addresses
the multi-modal nature of the problem, and provides a straightforward strategy
where independent visual features from potential speakers in the scene are
assigned to a previously detected speech event. Our experiments show that, an
small graph data structure built from a single frame, allows to approximate an
instantaneous audio-visual assignment problem. Moreover, the temporal extension
of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker
dataset with a mAP of 88.8\%