74 research outputs found
Deep Remix: Remixing Musical Mixtures Using a Convolutional Deep Neural Network
Audio source separation is a difficult machine learning problem and
performance is measured by comparing extracted signals with the component
source signals. However, if separation is motivated by the ultimate goal of
re-mixing then complete separation is not necessary and hence separation
difficulty and separation quality are dependent on the nature of the re-mix.
Here, we use a convolutional deep neural network (DNN), trained to estimate
'ideal' binary masks for separating voice from music, to perform re-mixing of
the vocal balance by operating directly on the individual magnitude components
of the musical mixture spectrogram. Our results demonstrate that small changes
in vocal gain may be applied with very little distortion to the ultimate
re-mix. Our method may be useful for re-mixing existing mixes
Seeing Through Noise: Visually Driven Speaker Separation and Enhancement
Isolating the voice of a specific person while filtering out other voices or
background noises is challenging when video is shot in noisy environments. We
propose audio-visual methods to isolate the voice of a single speaker and
eliminate unrelated sounds. First, face motions captured in the video are used
to estimate the speaker's voice, by passing the silent video frames through a
video-to-speech neural network-based model. Then the speech predictions are
applied as a filter on the noisy input audio. This approach avoids using
mixtures of sounds in the learning process, as the number of such possible
mixtures is huge, and would inevitably bias the trained model. We evaluate our
method on two audio-visual datasets, GRID and TCD-TIMIT, and show that our
method attains significant SDR and PESQ improvements over the raw
video-to-speech predictions, and a well-known audio-only method.Comment: Supplementary video: https://www.youtube.com/watch?v=qmsyj7vAzo
Improving Source Separation via Multi-Speaker Representations
Lately there have been novel developments in deep learning towards solving
the cocktail party problem. Initial results are very promising and allow for
more research in the domain. One technique that has not yet been explored in
the neural network approach to this task is speaker adaptation. Intuitively,
information on the speakers that we are trying to separate seems fundamentally
important for the speaker separation task. However, retrieving this speaker
information is challenging since the speaker identities are not known a priori
and multiple speakers are simultaneously active. There is thus some sort of
chicken and egg problem. To tackle this, source signals and i-vectors are
estimated alternately. We show that blind multi-speaker adaptation improves the
results of the network and that (in our case) the network is not capable of
adequately retrieving this useful speaker information itself
- …