Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Many previous audio-visual voice-related works focus on speech, ignoring the singing voice in the growing number of musical video streams on the Internet. For processing diverse musical video data, voice activity detection is a necessary step. This paper attempts to detect the speech and singing voices of target performers in musical video streams using audiovisual information. To integrate information of audio and visual modalities, a multi-branch network is proposed to learn audio and image representations, and the representations are fused by attention based on semantic similarity to shape the acoustic representations through the probability of anchor vocalization. Experiments show the proposed audio-visual multi-branch network far outperforms the audio-only model in challenging acoustic environments, indicating the cross-modal information fusion based on semantic correlation is sensible and successful.Comment: Accepted by INTERSPEECH 202

Concept and key technology analysis of deep-sea walking-swimming robot

The deep-sea robot is very useful in deep sea engineering. Based on a comparison and analysis of current deep-sea robots, this paper proposes a novel concept for a deep-sea walking-swimming robot, the purpose of which is to swim extensively in the sea and walk stably on the seafloor. The overall proposal, specifications and characteristics of the deep-sea walking-swimming robot are introduced. After an analysis of its environment and function characteristics,such key techniques as the regulation of the robot's walking/swimming attitude, cooperative current anti-turbulence of multi-legs and multi-joints, path planning for low energy consumption, dynamic seal of deep-sea joints and integration and optimization of the overall design are presented, showing that it is quite different from traditional underwater and multi-foot robots. Finally, the research progress of the above-mentioned techniques is also presented

Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Many previous audio-visual voice-related works focus on speech, ignoring the singing voice in the growing number of musical video streams on the Internet. For processing diverse musical video data, voice activity detection is a necessary step. This paper attempts to detect the speech and singing voices of target performers in musical video streams using audio-visual information. To integrate information of audio and visual modalities, a multi-branch network is proposed to learn audio and image representations, and the representations are fused by attention based on semantic similarity to shape the acoustic representations through the probability of anchor vocalization. Experiments show the proposed audio-visual multi-branch network far outperforms the audio-only model in challenging acoustic environments, indicating the cross-modal information fusion based on semantic correlation is sensible and successful