We consider the problem of online audio source separation. Existing
algorithms adopt either a sliding block approach or a stochastic gradient
approach, which is faster but less accurate. Also, they rely either on spatial
cues or on spectral cues and cannot separate certain mixtures. In this paper,
we design a general online audio source separation framework that combines both
approaches and both types of cues. The model parameters are estimated in the
Maximum Likelihood (ML) sense using a Generalised Expectation Maximisation
(GEM) algorithm with multiplicative updates. The separation performance is
evaluated as a function of the block size and the step size and compared to
that of an offline algorithm.Comment: International conference on Latente Variable Analysis and Signal
  Separation (2012

E. Vincent

M.S. Brandstein

N.Q.K. Duong

S. Makino

Y. Mori

English

arXiv

Laurent S. R. Simon

Emmanuel Vincent

Crossref

A General Framework for Online Audio Source Separation

International audienceWe consider the problem of online audio source separation. Existing algorithms adopt either a sliding block approach or a stochastic gradient approach, which is faster but less accurate. Also, they rely either on spatial cues or on spectral cues and cannot separate certain mixtures. In this paper, we design a general online audio source separation framework that combines both approaches and both types of cues. The model parameters are estimated in the Maximum Likelihood (ML) sense using a Generalised Expectation Maximisation (GEM) algorithm with multiplicative updates. The separation performance is evaluated as a function of the block size and the step size and compared to that of an offline algorithm

Simon, Laurent, S. R.

Vincent, Emmanuel

HAL-CentraleSupelec

A general framework for online audio source separation

INRIA a CCSD electronic archive server

HAL Id: hal-00655398https://hal.inria.fr/hal-00655398Submitted on 28 Dec 2011HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.A general framework for online audio source separationLaurent Simon, Emmanuel VincentTo cite this version:Laurent Simon, Emmanuel Vincent. A general framework for online audio source separation. Inter-national conference on Latent Variable Analysis and Signal Separation, Mar 2012, Tel-Aviv, Israel.￿hal-00655398￿A General Framework for Online Audio SourceSeparationLaurent S. R. Simon and Emmanuel VincentINRIA, Centre de Rennes - Bretagne AtlantiqueCampus de Beaulieu, 35042 Rennes Cedex, France{laurent.s.simon@inria.fr, emmanuel.vincent@inria.fr}Abstract. We consider the problem of online audio source separation.Existing algorithms adopt either a sliding block approach or a stochas-tic gradient approach, which is faster but less accurate. Also, they relyeither on spatial cues or on spectral cues and cannot separate certainmixtures. In this paper, we design a general online audio source separa-tion framework that combines both approaches and both types of cues.The model parameters are estimated in the Maximum Likelihood (ML)sense using a Generalised Expectation Maximisation (GEM) algorithmwith multiplicative updates. The separation performance is evaluated asa function of the block size and the step size and compared to that of anoffline algorithm.Keywords: Online audio source separation, nonnegative matrix factori-sation, sliding block, stochastic gradient.1 IntroductionAudio source separation is the process of recovering a set of audio signals froma given mixture signal. This can be addressed via established approaches suchas Independent Component Analysis (ICA), binary masking and Sparse Com-ponent Analysis (SCA) [1] or more recent approaches such as local Gaussianmodeling and Nonnegative Matrix Factorisation (NMF) [2]. Most current algo-rithms are offline algorithms which require the whole signal in order to estimatethe sources. In this paper, we focus on online audio source separation, wherebyonly the past samples of the mixture are available. This constraint arises inparticular in real-time scenarios.A few online implementations have been designed for ICA [3] [4], time-frequency masking [5], local Gaussian modeling [6], spectral continuity-basedseparation [7] and NMF [8]. However, these algorithms rely either on spatialcues [3] – [6] or on spectral cues [7, 8] alone. Such algorithms are not capableof separating mixtures where several sources have the same spatial position andseveral sources have similar spectral characteristics. For example, in pop music,the voice, the snare drum, the bass drum and the bass are often mixed to thecentre and several voices or several guitars are present.2 Laurent S. R. Simon and Emmanuel VincentIn order to address this issue, we consider the general flexible source sepa-ration framework in [9]. This framework generalises a wide range of algorithmssuch as certain forms of ICA, local Gaussian modeling and NMF, and enablesthe specification of additional constraints on the sources such as harmonicity.By jointly exploiting spatial and spectral cues, it makes it possible to robustlyseparate difficult mixtures such as above.The two main approaches for online source separation are the sliding block(also known as blockwise) approach, as used in [3] [4] [5] [7], and the stochasticgradient (also known as stepwise) approach, as used in [6] [8]. The sliding blockmethod consists in applying the offline audio source separation algorithm to ablock of M time frames. Once this block of signal has been processed, a frameis extracted for each of the J sources before sliding the processing block by oneframe. This approach is computationally intensive but accurate. The stepwisemethod offers to update the model parameters in every frame using only thelatest available frame and the model parameters estimated in the previous frame.As it uses only the latest available frame at a given time, this approach is fasterthan the sliding block approach but can be inaccurate.In this paper, we propose a general iterative online algorithm for the sourceseparation framework in [9] that combines the sliding block approach and thestepwise approach using two hyper-parameters: the block sizeM and the step sizeα. As a by-product, we provide a way of circumventing the annealing procedurein [9], which would require a large number of iterations per block. Moreover, wedetermine the best trade-off between these two approaches experimentally on aset of real-world music mixtures.The structure of the rest of the paper is as follows: the flexible framework in[9] is introduced in Section 2. Section 3 presents the online algorithm. Experi-mental results are shown in Section 4. The conclusion can be found in Section5.2 General audio source separation frameworkWe operate in the time-frequency (TF) domain by means of the Short-TimeFourier Transform (STFT). In each frequency bin f and each time frame n, themultichannel mixture signal x(f, n) can be expressed asx(f, n) =J∑j=1cj(f, n) (1)where J is the number of sources and cj(f, n) is the STFT of the spatial imageof the j-th source.2.1 ModelWe assume that cj(f, n) is a complex-valued Gaussian random vector with zeromean and covariance matrix Rcj (f, n)cj ∼ Nc(0,Rcj ) (2)A General Framework for Online Audio Source Separation 3and that Rcj (f, n) factors asRcj (f, n) = Rj(f)vj(f, n) (3)where Rj(f) is the spatial covariance matrix of the j-th source and vj(f, n) isits spectral variance.In [9], Rj(f) is expressed as Rj(f) = Aj(f)AHj (f), and Aj(f) is estimatedinstead. This results in an annealing procedure, which would translate into alarge number of iterations within each block in our context. In order to cir-cumvent the annealing, we assume that Rj(f) is full-rank and directly estimateRj(f) instead, similarly to [10].The spectral variance vj(f, n) is modeled via a form of hierarchical NMF [9].The matrix of spectral variances Vj , [vj(f, n)]f,n is first decomposed into theproduct of an excitation spectral power Vxj and a filter spectral power VfjVj = Vxj ⊙Vfj (4)where ⊙ denotes entrywise multiplication. Vxj is further decomposed into theproduct of a matrix of narrowband spectral patterns Wxj , a matrix of spectralenvelope weights Uxj , a matrix of temporal envelope weights Gxj and a matrixof time-localised temporal patterns Hxj , so thatVxj = WxjUxjGxjHxj . (5)Vfj is decomposed in a similar way.This factorisation enables the specification of various spectral or temporalconstraints over the sources. For example, harmonicity can be enforced by fixingWxj to a set of narrowband harmonic patterns.2.2 Offline EM-MU algorithmIn an offline context, the model parameters are estimated in the Maximum Like-lihood (ML) sense by a Generalised Expectation-Maximisation (GEM) algo-rithm combined with Multiplicative Updates (MU) applied to the complete data{cj(f, n)}.The log-likelihood is defined using the empirical mixture covariance matrixR̂x(f, n) [10] aslogL =∑f,n− tr(R−1x(f, n)R̂x(f, n))− log det(πRx(f, n)) (6)whereRx(f, n) =J∑j=1Rcj (f, n) (7)is the covariance of the mixture x(f, n).4 Laurent S. R. Simon and Emmanuel VincentIn the E-step, the expectation of the natural statistics is computed via [10]Ωj(f, n) = Rcj (f, n)R−1x(f, n) (8)R̂cj (f, n) = Ωj(f, n)R̂x(f, n)ΩHj (f, n) + (I−Wj(f, n))Rcj (f, n) (9)where Ωj is the Wiener filter, I is the I × I identity matrix and I is the numberof channels of the mixture.In the M-step, the model parameters are updated as [9, 10]Rj(f) =1NN∑n=11vj(f, n)R̂cj (f, n) (10)Wxj = Wxj ⊙[Ξ̂j ⊙Vxj .−2 ⊙Vfj .−1](UxjGxjHxj )TVxj .−1(UxjGxjHxj )T(11)Uxj = Uxj ⊙WxjT [Ξ̂j ⊙Vxj .−2 ⊙Vfj .−1](GxjHxj )TWxjTVxj .−1(GxjHxj )T(12)Gxj = Gxj ⊙(WxjUxj )T [Ξ̂j ⊙Vxj .−2 ⊙Vfj .−1]HxjT(WxjUxj )TVxj .−1HxjT(13)Hxj = Hxj ⊙(WxjUxjGxj )T [Ξ̂j ⊙Vxj .−2 ⊙Vfj .−1](WxjUxjGxj )TVxj .−1(14)where .p denotes entrywise raising to the power p, N is the number of timeframes in the STFT of the signal and Ξ̂j = [ξ̂j(f, n)]f,n, withξ̂j(f, n) =1Itr(R−1j (f)R̂cj (f, n)). (15)Wfj , Ufj , Gfj and Hfj are updated in a similar way.After each EM iteration, the model parameters are normalised: the mean ofRj ,Wxj , Uxj ,Gxj ,Hxj , Wfj ,Ufj and Hfj are normalised to 1 while Gfj is multipliedby the product of the normalisation factors of the other variables.The separated sources are then obtained viaĉj(f, n) = Ωj(f, n)x(f, n). (16)3 Online EM-MU algorithmWe now consider an online context where in each time frame t, the data islimited to a block of M STFT frames indexed by n with t − M + 1 ≤ n ≤ t,where M = 1 for the stepwise approach and M = N for the full offline approach.We define a step size coefficient α ∈ ]0; 1] to stabilise the parameter updates byaveraging over time. For each block, the spatial covariance matrices R(t)j (f) areinitialised to a diffuse spatial covariance spanning a part of the audio space. Thetemporal weights Gxj(t) are randomly initialised and the normalised to the meanA General Framework for Online Audio Source Separation 5spectral power of the signal. Finally, the temporal patterns Hxj(t) are initialisedto diagonal matrices. The expectation of the natural statistics is computed using(8) and (9) for t−M+1 ≤ n ≤ t, whilst the spatial covariance matrix is updatedas follows:R(t)j (f) = (1− α)R(t−1)j (f) + α(1Mt∑n=t−M+11vj(f, n)R̂cj (f, n))(17)where the superscript (t) denotes is the value of matrix for the block t.Gxj(t) and Hxj(t) are updated using (13) and (14) for t −M + 1 ≤ n ≤ t, asthey are expected to significantly vary between blocks, whereas the updates ofWxj and Uxj becomeWxj(t) = Wxj(t) ⊙Mxj(t)Cxj(t)(18)Uxj(t) = Uxj(t) ⊙Nxj(t)Dxj(t)(19)whereMxj(t) = (1− α)Mxj(t−1) + α[Ξ̂j ⊙Vxj .−2 ⊙Vfj .−1](Uxj(t)Gxj(t)Hxj(t))T (20)Cxj(t) = (1− α)Cxj(t−1) + αVxj .−1(Uxj(t)Gxj(t)Hxj(t))T (21)Nxj(t) = (1− α)Nxj(t−1) + αWxj(t)T [Ξ̂j ⊙Vxj .−2 ⊙Vfj .−1](Gxj(t)Hxj(t))T (22)Dxj(t) = (1− α)Dxj(t−1) + αWxj(t)TVxj .−1(Gxj(t)Hxj(t))T (23)where Ξ̂(t)j is computed as in (16). Mfj(t), Cfj(t), Nfj(t)and Dfj(t)are updatedin a similar way. At each block, several iterations can be performed in order toimprove the estimation of the model parameters.Although equations (17) to (19) look similar to the online update of the localGaussian model in [6] and [8], there are two crucial differences:– The framework introduced in the current paper is more general in the sensethat it uses hierarchical NMF, enabling the user to apply more specific con-straints than when using shallow NMF.– It is not limited to the sole use of the latest audio frame.4 Experimental resultsWe compared the performance of the online audio source separation frameworkto the offline framework introduced in section 2.2, as a function of the number ofEM iterations, α and M . The project aiming at remixing of recordings for soundengineers, DJs and consumers, we processed five 10 s long stereo commercialpop recordings composed of bass, drums, guitars, strings and voice. All the6 Laurent S. R. Simon and Emmanuel Vincentrecordings were recorded at 44100 Hz. The STFT was computed using half-overlapping 2048 sample sine windows. In the offline algorithm as well as in theonline algorithm, each of the modeled sources were constrained in a way similarto section V.C in [9]. In the case of an harmonic source, Wxj(t) was fixed to aset of narrowband harmonic spectral patterns and the spectral envelope weightsin Uxj(t) were updated, whereas for bass and percussive sources, Wxj(t) was afixed diagonal matrix and Uxj(t) was a fixed matrix of basis spectra learned overa corpus of bass and drum sounds.Audio samples of the separated sounds of this experiment can be found onhttp://www.irisa.fr/metiss/lssimon/LVA2012/index.html .Separation performance was evaluated using the Signal-to-Distortion Ratio(SDR), the Signal-to-Interference Ratio (SIR), the source Image to Spatial dis-tortion Ratio (ISR) and the Source-to-Artifacts Ratio (SAR) defined in [11]. Foreach set of conditions over the number of iterations, M and α, each of thesecriteria was averaged over all the mixtures and all the separated sound sources.Over all the results of this experiment, the SDR varied between -1.1 and 0.9 dB,the SIR between -4 and 1 dB, the ISR between 2.3 and 3.9 dB and the SARbetween 10 and 19 dB.Table 1. Separation performance (dB) of the offline and best online algorithms.Algorithm α M number of iterations SDR SIR ISR SARoffline N/A N/A 100 0.8586 1.2837 3.7989 13.3872online 1 50 30 0.8671 1.0675 3.9690 12.3278As shown in table 1, when α = 1, M = 50 and 30 GEM iterations areperformed, the separation performance of the online algorithm is close to thatof the offline algorithm. For smaller block size and smaller number of iterations,the performance decreases. For example, for M = 10 and 6 GEM iteration, theSDR is 0.53 dB and the SIR is 3.53 dB. More generally, fig. 1 shows that forα = 1, increasing either the block size or the number of iterations increasesthe SDR, though the block size has less effect on the SDR than the number ofiterations. The results also show that increasing the number of iterations from10 to 30 increases the SDR by 0.2 dB, which can be considered as a significantimprovement.When α < 1, the SDR decreases significantly as can be seen in fig. 1. It canalso be seen that increasing the number of iterations decreases the SDR andchanges of block size have little to no effect on the SDR. This can be explainedby an inaccurate estimation of the model parameters of certain sources in thetime intervals when these sources are inactive. These inaccurate parameters arethen carried over subsequent time frames and may not converge back to accuratevalues. This undesirable effect is particularly salient for those parameters that areless constrained. For instance, with the considered model, the spatial covariancematrices of all sources gradually diverge towards a diffuse spatial covarianceA General Framework for Online Audio Source Separation 7spanning all directions in the mixture, while the effect is more limited for spectralparameters which are fixed or heavily constrained. Potential solutions to thisproblem are presented in the conclusion.α = 0.02number of iterationsM   2  6 10 30 1102050 −1.1 dB 0.8  dBα = 0.05number of iterationsM 2  6 10 30 1102050α = 0.1number of iterationsM 2  6 10 30 1102050α = 0.2number of iterationsM 2  6 10 30 1102050α = 0.5number of iterationsM 2  6 10 30 1102050α = 1number of iterationsM 2  6 10 30 1102050Fig. 1. Mean SDR for all sources and all mixtures, as a function of α, M and step size.5 ConclusionIn this paper, a new framework for online audio source separation was presented.This algorithm offers an increased flexibility both in terms of the range of con-straints that can be specified for each source and of the choice of a trade-offbetween separation accuracy and computational cost. It was shown that theseparation accuracy is higher when the block size is large, but that small blocksizes nevertheless offer an acceptable separation. However, small step sizes causethe spatial covariance matrices to diverge due to the presence of silence intervalsin the sources.This issue is well-known in the beamforming literature where a voice activ-ity detector is used to restrict the time frames in which the model parametersare updated [12]. While this solution does not readily extend to source sepa-ration, we believe that there exist a number of alternative promising solutions,e.g. adding soft constraints over the least constrained parameters by means ofprobabilistic priors, using different step sizes for the most constrained and theleast constrained parameters, and using signal-dependent step sizes related to8 Laurent S. R. Simon and Emmanuel Vincentthe power of Rcj (f, n) such that the parameters are not updated in the timeintervals with low power.Future work should also include an optimisation of the initialisation of themodel parameters for each new block. After these improvements, we expect thatthe proposed framework will reach its full potential and provide a better trade-offbetween separation performance and computational cost.Acknowledgements This work was supported by the EUREKA Eurostarsi3DMusic project funded by Oseo.References1. Makino, S., Lee, T.-W. and Sawada, H.: Blind Speech Separation. Springer (2007)2. Vincent, E., Jafari, M. G., Abdallah, S. A., Plumbley, M. D. and Davies, M. E.:Probabilistic modeling paradigms for audio source separation. In: Machine Audition:Principles, Algorithms and Systems. IGI Global, pp. 162–185 (2010)3. Mukai, R., Sawada, H., Araki, S. and Makino, S.: Real-time Blind Source SeparationFor Moving Speakers Using Blockwise ICA and Residual Crosstalk Subtraction. In:4th Int. Symp. Independent Component Analysis and Blind Signal Separation, pp.975–980 (2003)4. Mori, Y., Saruwatari, H., Takatani, T., Ukai, S., Shikano, K., Hiekata, T., Ikeda, Y.,Hashimoto, H. and Morita, T.: Blind separation of acoustic signals combining SIMO-model-based independent component analysis and binary masking. EURASIP Jour-nal on Advances in Signal Processing, vol. 2006, issue 1, pp. 1–17 (2006)5. Loesch, B. and Yang, B.: Online blind source separation based on time-frequencysparseness. In: Proc. 2009 IEEE Int. Conf. on Acoustics, Speech and Signal Pro-cessing, pp. 117–120 (2009)6. Togami, M.: Online speech source separation based on maximum likelihood of localGaussian modeling. In: Proc. 2011 IEEE Int. Conf. on Acoustics, Speech and SignalProcessing, pp. 213–216 (2011)7. Ono, N., Miyamoto, K. and Sagayama, S.: A real-time equalizer of harmonic andpercussive components in music signals. In: Proc. 2008 Int. Conf. on Music Infor-mation Retrieval, pp. 139 – 144 (2008)8. Wang, D, Vipperla, R. and Evans, N.: Online pattern learning for non-negativeconvolutive sparse coding. In: Proc. Interspeech’11, pp. 65–68 (2011)9. Ozerov, A., Vincent, E. and Bimbot, F.: A general flexible framework for the han-dling of prior information in audio source separation. IEEE Transactions on Audio,Speech, and Language Processing, to appear10. Duong, N.Q.K., Vincent, E. and Gribonval, R.: Under-determined reverberant au-dio source separation using a full-rank spatial covariance model. IEEE Transactionson Audio, Speech, and Language Processing, vol. 18, issue 7, pp. 1830–1840 (2010)11. Vincent, E., Sawada, H., Bofill, P., Makino, S. and Rosca, J. P.: First stereo audiosource separation evaluation campaign: data, algorithms and results. In: Proc. 2007Int. Conf. on Independent Component Analysis and Blind Source Separation, pp.552–559 (2007)12. Brandstein, M.S., Ward, D.B.: Microphone Arrays: Signal Processing Techniquesand Applications. Springer (2001)

HAL-Rennes 1

A general framework for online audio source separationLaurent S. R. Simon, Emmanuel VincentTo cite this version:Laurent S. R. Simon, Emmanuel Vincent. A general framework for online audio source sepa-ration. International conference on Latent Variable Analysis and Signal Separation, Mar 2012,Tel-Aviv, Israel. 2012. <hal-00655398>HAL Id: hal-00655398https://hal.inria.fr/hal-00655398Submitted on 28 Dec 2011HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.A General Framework for Online Audio SourceSeparationLaurent S. R. Simon and Emmanuel VincentINRIA, Centre de Rennes - Bretagne AtlantiqueCampus de Beaulieu, 35042 Rennes Cedex, France{laurent.s.simon@inria.fr, emmanuel.vincent@inria.fr}Abstract. We consider the problem of online audio source separation.Existing algorithms adopt either a sliding block approach or a stochas-tic gradient approach, which is faster but less accurate. Also, they relyeither on spatial cues or on spectral cues and cannot separate certainmixtures. In this paper, we design a general online audio source separa-tion framework that combines both approaches and both types of cues.The model parameters are estimated in the Maximum Likelihood (ML)sense using a Generalised Expectation Maximisation (GEM) algorithmwith multiplicative updates. The separation performance is evaluated asa function of the block size and the step size and compared to that of anoﬄine algorithm.Keywords: Online audio source separation, nonnegative matrix factori-sation, sliding block, stochastic gradient.1 IntroductionAudio source separation is the process of recovering a set of audio signals froma given mixture signal. This can be addressed via established approaches suchas Independent Component Analysis (ICA), binary masking and Sparse Com-ponent Analysis (SCA) [1] or more recent approaches such as local Gaussianmodeling and Nonnegative Matrix Factorisation (NMF) [2]. Most current algo-rithms are oﬄine algorithms which require the whole signal in order to estimatethe sources. In this paper, we focus on online audio source separation, wherebyonly the past samples of the mixture are available. This constraint arises inparticular in real-time scenarios.A few online implementations have been designed for ICA [3] [4], time-frequency masking [5], local Gaussian modeling [6], spectral continuity-basedseparation [7] and NMF [8]. However, these algorithms rely either on spatialcues [3] – [6] or on spectral cues [7, 8] alone. Such algorithms are not capableof separating mixtures where several sources have the same spatial position andseveral sources have similar spectral characteristics. For example, in pop music,the voice, the snare drum, the bass drum and the bass are often mixed to thecentre and several voices or several guitars are present.2 Laurent S. R. Simon and Emmanuel VincentIn order to address this issue, we consider the general flexible source sepa-ration framework in [9]. This framework generalises a wide range of algorithmssuch as certain forms of ICA, local Gaussian modeling and NMF, and enablesthe specification of additional constraints on the sources such as harmonicity.By jointly exploiting spatial and spectral cues, it makes it possible to robustlyseparate difficult mixtures such as above.The two main approaches for online source separation are the sliding block(also known as blockwise) approach, as used in [3] [4] [5] [7], and the stochasticgradient (also known as stepwise) approach, as used in [6] [8]. The sliding blockmethod consists in applying the oﬄine audio source separation algorithm to ablock of M time frames. Once this block of signal has been processed, a frameis extracted for each of the J sources before sliding the processing block by oneframe. This approach is computationally intensive but accurate. The stepwisemethod offers to update the model parameters in every frame using only thelatest available frame and the model parameters estimated in the previous frame.As it uses only the latest available frame at a given time, this approach is fasterthan the sliding block approach but can be inaccurate.In this paper, we propose a general iterative online algorithm for the sourceseparation framework in [9] that combines the sliding block approach and thestepwise approach using two hyper-parameters: the block sizeM and the step sizeα. As a by-product, we provide a way of circumventing the annealing procedurein [9], which would require a large number of iterations per block. Moreover, wedetermine the best trade-off between these two approaches experimentally on aset of real-world music mixtures.The structure of the rest of the paper is as follows: the flexible framework in[9] is introduced in Section 2. Section 3 presents the online algorithm. Experi-mental results are shown in Section 4. The conclusion can be found in Section5.2 General audio source separation frameworkWe operate in the time-frequency (TF) domain by means of the Short-TimeFourier Transform (STFT). In each frequency bin f and each time frame n, themultichannel mixture signal x(f, n) can be expressed asx(f, n) =J∑j=1cj(f, n) (1)where J is the number of sources and cj(f, n) is the STFT of the spatial imageof the j-th source.2.1 ModelWe assume that cj(f, n) is a complex-valued Gaussian random vector with zeromean and covariance matrix Rcj (f, n)cj ∼ Nc(0,Rcj ) (2)A General Framework for Online Audio Source Separation 3and that Rcj (f, n) factors asRcj (f, n) = Rj(f)vj(f, n) (3)where Rj(f) is the spatial covariance matrix of the j-th source and vj(f, n) isits spectral variance.In [9], Rj(f) is expressed as Rj(f) = Aj(f)AHj (f), and Aj(f) is estimatedinstead. This results in an annealing procedure, which would translate into alarge number of iterations within each block in our context. In order to cir-cumvent the annealing, we assume that Rj(f) is full-rank and directly estimateRj(f) instead, similarly to [10].The spectral variance vj(f, n) is modeled via a form of hierarchical NMF [9].The matrix of spectral variances Vj , [vj(f, n)]f,n is first decomposed into theproduct of an excitation spectral power Vxj and a filter spectral power VfjVj = Vxj Vfj (4)where  denotes entrywise multiplication. Vxj is further decomposed into theproduct of a matrix of narrowband spectral patterns Wxj , a matrix of spectralenvelope weights Uxj , a matrix of temporal envelope weights Gxj and a matrixof time-localised temporal patterns Hxj , so thatVxj =WxjUxjGxjHxj . (5)Vfj is decomposed in a similar way.This factorisation enables the specification of various spectral or temporalconstraints over the sources. For example, harmonicity can be enforced by fixingWxj to a set of narrowband harmonic patterns.2.2 Oﬄine EM-MU algorithmIn an oﬄine context, the model parameters are estimated in the Maximum Like-lihood (ML) sense by a Generalised Expectation-Maximisation (GEM) algo-rithm combined with Multiplicative Updates (MU) applied to the complete data{cj(f, n)}.The log-likelihood is defined using the empirical mixture covariance matrixR̂x(f, n) [10] aslogL =∑f,n− tr(R−1x(f, n)R̂x(f, n))− log det(piRx(f, n)) (6)whereRx(f, n) =J∑j=1Rcj (f, n) (7)is the covariance of the mixture x(f, n).4 Laurent S. R. Simon and Emmanuel VincentIn the E-step, the expectation of the natural statistics is computed via [10]Ωj(f, n) = Rcj (f, n)R−1x(f, n) (8)R̂cj (f, n) = Ωj(f, n)R̂x(f, n)ΩHj (f, n) + (I−Wj(f, n))Rcj (f, n) (9)where Ωj is the Wiener filter, I is the I × I identity matrix and I is the numberof channels of the mixture.In the M-step, the model parameters are updated as [9, 10]Rj(f) =1NN∑n=11vj(f, n)R̂cj (f, n) (10)Wxj =Wxj [Ξ̂j Vxj .−2 Vfj .−1](UxjGxjHxj )TVxj .−1(UxjGxjHxj )T(11)Uxj = Uxj WxjT [Ξ̂j Vxj .−2 Vfj .−1](GxjHxj )TWxjTVxj .−1(GxjHxj )T(12)Gxj = Gxj (WxjUxj )T [Ξ̂j Vxj .−2 Vfj .−1]HxjT(WxjUxj )TVxj .−1HxjT(13)Hxj = Hxj (WxjUxjGxj )T [Ξ̂j Vxj .−2 Vfj .−1](WxjUxjGxj )TVxj .−1(14)where .p denotes entrywise raising to the power p, N is the number of timeframes in the STFT of the signal and Ξ̂j = [ξ̂j(f, n)]f,n, withξ̂j(f, n) =1Itr(R−1j (f)R̂cj (f, n)). (15)Wfj , Ufj , Gfj and Hfj are updated in a similar way.After each EM iteration, the model parameters are normalised: the mean ofRj ,Wxj ,Uxj ,Gxj ,Hxj ,Wfj ,Ufj andHfj are normalised to 1 whileGfj is multipliedby the product of the normalisation factors of the other variables.The separated sources are then obtained viaĉj(f, n) = Ωj(f, n)x(f, n). (16)3 Online EM-MU algorithmWe now consider an online context where in each time frame t, the data islimited to a block of M STFT frames indexed by n with t −M + 1 ≤ n ≤ t,whereM = 1 for the stepwise approach andM = N for the full oﬄine approach.We define a step size coefficient α ∈ ]0; 1] to stabilise the parameter updates byaveraging over time. For each block, the spatial covariance matrices R(t)j (f) areinitialised to a diffuse spatial covariance spanning a part of the audio space. Thetemporal weights Gxj(t) are randomly initialised and the normalised to the meanA General Framework for Online Audio Source Separation 5spectral power of the signal. Finally, the temporal patterns Hxj(t) are initialisedto diagonal matrices. The expectation of the natural statistics is computed using(8) and (9) for t−M+1 ≤ n ≤ t, whilst the spatial covariance matrix is updatedas follows:R(t)j (f) = (1− α)R(t−1)j (f) + α(1Mt∑n=t−M+11vj(f, n)R̂cj (f, n))(17)where the superscript (t) denotes is the value of matrix for the block t.Gxj(t) and Hxj(t) are updated using (13) and (14) for t −M + 1 ≤ n ≤ t, asthey are expected to significantly vary between blocks, whereas the updates ofWxj and Uxj becomeWxj(t) =Wxj(t) Mxj(t)Cxj(t)(18)Uxj(t) = Uxj(t) Nxj(t)Dxj(t)(19)whereMxj(t) = (1− α)Mxj(t−1) + α[Ξ̂j Vxj .−2 Vfj .−1](Uxj(t)Gxj(t)Hxj(t))T (20)Cxj(t) = (1− α)Cxj(t−1) + αVxj .−1(Uxj(t)Gxj(t)Hxj(t))T (21)Nxj(t) = (1− α)Nxj(t−1) + αWxj(t)T [Ξ̂j Vxj .−2 Vfj .−1](Gxj(t)Hxj(t))T (22)Dxj(t) = (1− α)Dxj(t−1) + αWxj(t)TVxj .−1(Gxj(t)Hxj(t))T (23)where Ξ̂(t)j is computed as in (16). Mfj(t), Cfj(t), Nfj(t)and Dfj(t)are updatedin a similar way. At each block, several iterations can be performed in order toimprove the estimation of the model parameters.Although equations (17) to (19) look similar to the online update of the localGaussian model in [6] and [8], there are two crucial differences:– The framework introduced in the current paper is more general in the sensethat it uses hierarchical NMF, enabling the user to apply more specific con-straints than when using shallow NMF.– It is not limited to the sole use of the latest audio frame.4 Experimental resultsWe compared the performance of the online audio source separation frameworkto the oﬄine framework introduced in section 2.2, as a function of the number ofEM iterations, α and M . The project aiming at remixing of recordings for soundengineers, DJs and consumers, we processed five 10 s long stereo commercialpop recordings composed of bass, drums, guitars, strings and voice. All the6 Laurent S. R. Simon and Emmanuel Vincentrecordings were recorded at 44100 Hz. The STFT was computed using half-overlapping 2048 sample sine windows. In the oﬄine algorithm as well as in theonline algorithm, each of the modeled sources were constrained in a way similarto section V.C in [9]. In the case of an harmonic source, Wxj(t) was fixed to aset of narrowband harmonic spectral patterns and the spectral envelope weightsin Uxj(t) were updated, whereas for bass and percussive sources, Wxj(t) was afixed diagonal matrix and Uxj(t) was a fixed matrix of basis spectra learned overa corpus of bass and drum sounds.Audio samples of the separated sounds of this experiment can be found onhttp://www.irisa.fr/metiss/lssimon/LVA2012/index.html .Separation performance was evaluated using the Signal-to-Distortion Ratio(SDR), the Signal-to-Interference Ratio (SIR), the source Image to Spatial dis-tortion Ratio (ISR) and the Source-to-Artifacts Ratio (SAR) defined in [11]. Foreach set of conditions over the number of iterations, M and α, each of thesecriteria was averaged over all the mixtures and all the separated sound sources.Over all the results of this experiment, the SDR varied between -1.1 and 0.9 dB,the SIR between -4 and 1 dB, the ISR between 2.3 and 3.9 dB and the SARbetween 10 and 19 dB.Table 1. Separation performance (dB) of the oﬄine and best online algorithms.Algorithm α M number of iterations SDR SIR ISR SARoﬄine N/A N/A 100 0.8586 1.2837 3.7989 13.3872online 1 50 30 0.8671 1.0675 3.9690 12.3278As shown in table 1, when α = 1, M = 50 and 30 GEM iterations areperformed, the separation performance of the online algorithm is close to thatof the oﬄine algorithm. For smaller block size and smaller number of iterations,the performance decreases. For example, for M = 10 and 6 GEM iteration, theSDR is 0.53 dB and the SIR is 3.53 dB. More generally, fig. 1 shows that forα = 1, increasing either the block size or the number of iterations increasesthe SDR, though the block size has less effect on the SDR than the number ofiterations. The results also show that increasing the number of iterations from10 to 30 increases the SDR by 0.2 dB, which can be considered as a significantimprovement.When α < 1, the SDR decreases significantly as can be seen in fig. 1. It canalso be seen that increasing the number of iterations decreases the SDR andchanges of block size have little to no effect on the SDR. This can be explainedby an inaccurate estimation of the model parameters of certain sources in thetime intervals when these sources are inactive. These inaccurate parameters arethen carried over subsequent time frames and may not converge back to accuratevalues. This undesirable effect is particularly salient for those parameters that areless constrained. For instance, with the considered model, the spatial covariancematrices of all sources gradually diverge towards a diffuse spatial covarianceA General Framework for Online Audio Source Separation 7spanning all directions in the mixture, while the effect is more limited for spectralparameters which are fixed or heavily constrained. Potential solutions to thisproblem are presented in the conclusion.α = 0.02number of iterationsM   2  6 10 30 1102050 −1.1 dB 0.8  dBα = 0.05number of iterationsM 2  6 10 30 1102050α = 0.1number of iterationsM 2  6 10 30 1102050α = 0.2number of iterationsM 2  6 10 30 1102050α = 0.5number of iterationsM 2  6 10 30 1102050α = 1number of iterationsM 2  6 10 30 1102050Fig. 1. Mean SDR for all sources and all mixtures, as a function of α, M and step size.5 ConclusionIn this paper, a new framework for online audio source separation was presented.This algorithm offers an increased flexibility both in terms of the range of con-straints that can be specified for each source and of the choice of a trade-offbetween separation accuracy and computational cost. It was shown that theseparation accuracy is higher when the block size is large, but that small blocksizes nevertheless offer an acceptable separation. However, small step sizes causethe spatial covariance matrices to diverge due to the presence of silence intervalsin the sources.This issue is well-known in the beamforming literature where a voice activ-ity detector is used to restrict the time frames in which the model parametersare updated [12]. While this solution does not readily extend to source sepa-ration, we believe that there exist a number of alternative promising solutions,e.g. adding soft constraints over the least constrained parameters by means ofprobabilistic priors, using different step sizes for the most constrained and theleast constrained parameters, and using signal-dependent step sizes related to8 Laurent S. R. Simon and Emmanuel Vincentthe power of Rcj (f, n) such that the parameters are not updated in the timeintervals with low power.Future work should also include an optimisation of the initialisation of themodel parameters for each new block. After these improvements, we expect thatthe proposed framework will reach its full potential and provide a better trade-offbetween separation performance and computational cost.Acknowledgements This work was supported by the EUREKA Eurostarsi3DMusic project funded by Oseo.References1. Makino, S., Lee, T.-W. and Sawada, H.: Blind Speech Separation. Springer (2007)2. Vincent, E., Jafari, M. G., Abdallah, S. A., Plumbley, M. D. and Davies, M. E.:Probabilistic modeling paradigms for audio source separation. In: Machine Audition:Principles, Algorithms and Systems. IGI Global, pp. 162–185 (2010)3. Mukai, R., Sawada, H., Araki, S. and Makino, S.: Real-time Blind Source SeparationFor Moving Speakers Using Blockwise ICA and Residual Crosstalk Subtraction. In:4th Int. Symp. Independent Component Analysis and Blind Signal Separation, pp.975–980 (2003)4. Mori, Y., Saruwatari, H., Takatani, T., Ukai, S., Shikano, K., Hiekata, T., Ikeda, Y.,Hashimoto, H. and Morita, T.: Blind separation of acoustic signals combining SIMO-model-based independent component analysis and binary masking. EURASIP Jour-nal on Advances in Signal Processing, vol. 2006, issue 1, pp. 1–17 (2006)5. Loesch, B. and Yang, B.: Online blind source separation based on time-frequencysparseness. In: Proc. 2009 IEEE Int. Conf. on Acoustics, Speech and Signal Pro-cessing, pp. 117–120 (2009)6. Togami, M.: Online speech source separation based on maximum likelihood of localGaussian modeling. In: Proc. 2011 IEEE Int. Conf. on Acoustics, Speech and SignalProcessing, pp. 213–216 (2011)7. Ono, N., Miyamoto, K. and Sagayama, S.: A real-time equalizer of harmonic andpercussive components in music signals. In: Proc. 2008 Int. Conf. on Music Infor-mation Retrieval, pp. 139 – 144 (2008)8. Wang, D, Vipperla, R. and Evans, N.: Online pattern learning for non-negativeconvolutive sparse coding. In: Proc. Interspeech’11, pp. 65–68 (2011)9. Ozerov, A., Vincent, E. and Bimbot, F.: A general flexible framework for the han-dling of prior information in audio source separation. IEEE Transactions on Audio,Speech, and Language Processing, to appear10. Duong, N.Q.K., Vincent, E. and Gribonval, R.: Under-determined reverberant au-dio source separation using a full-rank spatial covariance model. IEEE Transactionson Audio, Speech, and Language Processing, vol. 18, issue 7, pp. 1830–1840 (2010)11. Vincent, E., Sawada, H., Bofill, P., Makino, S. and Rosca, J. P.: First stereo audiosource separation evaluation campaign: data, algorithms and results. In: Proc. 2007Int. Conf. on Independent Component Analysis and Blind Source Separation, pp.552–559 (2007)12. Brandstein, M.S., Ward, D.B.: Microphone Arrays: Signal Processing Techniquesand Applications. Springer (2001)

https://hal.inria.fr/hal-00655398/document

A general framework for online audio source separation

Abstract

Similar works

Full text

Available Versions

Crossref

HAL-CentraleSupelec

INRIA a CCSD electronic archive server

HAL-Rennes 1