1 research outputs found
Audeo: Audio Generation for a Silent Performance Video
We present a novel system that gets as an input video frames of a musician
playing the piano and generates the music for that video. Generation of music
from visual cues is a challenging problem and it is not clear whether it is an
attainable goal at all. Our main aim in this work is to explore the
plausibility of such a transformation and to identify cues and components able
to carry the association of sounds with visual events. To achieve the
transformation we built a full pipeline named `\textit{Audeo}' containing three
components. We first translate the video frames of the keyboard and the
musician hand movements into raw mechanical musical symbolic representation
Piano-Roll (Roll) for each video frame which represents the keys pressed at
each time step. We then adapt the Roll to be amenable for audio synthesis by
including temporal correlations. This step turns out to be critical for
meaningful audio generation. As a last step, we implement Midi synthesizers to
generate realistic music. \textit{Audeo} converts video to audio smoothly and
clearly with only a few setup constraints. We evaluate \textit{Audeo} on `in
the wild' piano performance videos and obtain that their generated music is of
reasonable audio quality and can be successfully recognized with high precision
by popular music identification software.Comment: Please see associated video at
https://www.youtube.com/watch?v=8rS3VgjG7_