The transform domain normalized LMS (TD-NLMS)-adaptive filtering algorithm is an efficient adaptive filter with fast convergence speed and reasonably low arithmetic complexity. However, it is sensitive to the level of the excitation signal, which may vary significantly over time in speech and audio signals. This paper proposes a new regularized transform domain NLMS (R-TDNLMS) algorithm and studies its mean and mean square convergence performance. The proposed algorithm extends the conventional TDNLMS algorithm by imposing a regularization term on the coefficients to reduce the variance of the estimator. The mean and mean square convergence behaviors of the proposed algorithm are studied to characterize its convergence condition and steady-state excess mean squares error (MSE). It shows that regularization can help to reduce the MSE for coloured inputs by trading slight bias for variance. Moreover, the immunity to varying input signal level is significantly reduced. Computer simulations are conducted to examine the effectiveness of the proposed algorithm and they are in good agreement with the theoretical analysis. © 2010 IEEE.published_or_final_versionThe 2010 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS 2010), Kuala Lumpur, Malaysia, 6-9 December 2010.
In Proceedings of APCCAS, 2010, p. 696-69

Chan, SC

Chu, YJ

Zhang, ZG

HKU Scholars Hub

Title A new regularized transform-domain NLMS adaptive filteringalgorithmAuthor(s) Chan, SC; Chu, YJ; Zhang, ZGCitationThe 2010 IEEE Asia Pacific Conference on Circuits and Systems(APCCAS 2010), Kuala Lumpur, Malaysia, 6-9 December 2010.In Proceedings of APCCAS, 2010, p. 696-699Issued Date 2010URL http://hdl.handle.net/10722/137728Rights IEEE Asia-Pacific Conference on Circuits and SystemsProceedings. Copyright © IEEE.  A New Regularized Transform-Domain NLMS Adaptive Filtering Algorithm S. C. Chan, Y. J. Chu and Z. G. Zhang Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, P. R. China {scchan; yjchu; zgzhang}@eee.hku.hk  Abstract—The transform domain normalized LMS (TD-NLMS)-adaptive filtering algorithm is an efficient adaptive filter with fast convergence speed and reasonably low arithmetic complexity. However, it is sensitive to the level of the excitation signal, which may vary significantly over time in speech and audio signals. This paper proposes a new regularized transform domain NLMS (R-TDNLMS) algorithm and studies its mean and mean square convergence performance. The proposed algorithm extends the conventional TDNLMS algorithm by imposing a regularization term on the coefficients to reduce the variance of the estimator. The mean and mean square convergence behaviors of the proposed algorithm are studied to characterize its convergence condition and steady-state excess mean squares error (MSE). It shows that regularization can help to reduce the MSE for coloured inputs by trading slight bias for variance. Moreover, the immunity to varying input signal level is significantly reduced. Computer simulations are conducted to examine the effectiveness of the proposed algorithm and they are in good agreement with the theoretical analysis.  Keywords—Regularization, performance analysis, NLMS, transform domain. I. INTRODUCTION Adaptive filters are frequently used in applications such as system identification and related problems, in which the statistics of the underlying signals are either unknown a priori, or slowly varying. One of the most commonly used algorithms is the well known least mean square (LMS) algorithm [1] and its variants [2-5], due to their numerical stability and computational simplicity. An important class of the LMS algorithm is called the transform domain normalized LMS (TDNLMS) algorithm [2-5], which exploits the decorrelation property of transformations such as the discrete Fourier transform (DFT), discrete cosine transform (DCT), and wavelet transform (WT), to approximately whiten the input signal. This helps to reduce the eigenvalue spread of the input autocorrelation matrix and hence significantly improve the convergence speed. One problem of the TDNLMS algorithm is its sensitivity to the level of the excitation signal, which may vary significantly over time as in speech and audio signals. In such situations, the estimated power of each transformed coefficient may become very small and the mean square errors may increase significantly. To address this issue, a commonly used technique is to introduce some kind of regularization into these algorithms. The regularization technique has attracted much interest recently as a useful tool for reducing the estimation variance, especially when only a small number of data samples are available [6]. It has also been successfully applied to a wide variety of areas such as audio signal processing [7]. For identification of systems with sparse impulse response coefficients, regularization technique was shown to improve the convergence speed and decrease the misadjustment in the LMS algorithms [8]. In this paper, a new regularized TDNLMS (R-TDNLMS) algorithm is proposed. A weighted regularization term on the adaptive filter coefficients is incorporated in the MSE cost function in order to reduce the estimation variance. To quantify the performance of the proposed algorithm, its mean and mean square convergence analyses are performed. Difference equations describing the mean and mean square convergence behavior of the proposed algorithm are derived. Simulation results show that the R-TDNLMS algorithm has lower excess MSE (EMSE) than the conventional TDNLMS algorithm for colored input and better immunity to the variation in input signal power. The theoretical analysis is also found to agree well with the simulation results. The rest of the paper is organized as follows: in Section II, the R-TDNLMS algorithm is derived. In Section III, the mean and mean square convergence behaviors of the proposed algorithm are derived. Simulation results are presented in Section IV. Finally, a conclusion is drawn in Section V. II. THE R-TDNLMS ALGORITHM Consider the identification of a linear time-invariant (LTI) finite impulse response (FIR) system with an impulse response coefficient vector *w  having L taps by an adaptive filter of the same length. The unknown system and adaptive filter are both excited by an input x(n) and the measured output of the system is d(n), which is assumed to be corrupted by an additive noise )(nη . Hence )()(*)( nnnd T η+= xw , (1) where TLnxnxnxn )]1(,),1(),([)( −−−= ?x  is the input signal vector. d(n) is also applied to the desired input of the adaptive filter and the weight vector is estimated recursively by minimizing the penalized MSE between d(n) and the output of the adaptive filter )()()( nnny T xw=  as follows 222 ||)(||]))()()([(min nnnndE TRMSE Dwxww ξε +−= , (2) where TL nwnwnwn )](,),(),([)( 21 ?=w  is the coefficient vector of the adaptive filter, ][⋅E is the expectation operator 978-1-4244-7456-1/10/$26.00 ©2010 IEEE 696 and the last term represents a regularization which helps to reduce the variance of the estimation especially when the covariance matrix of )(nx  is close to singular due to lacking of excitation or when *w  is spare. ξ  is the regularization parameter and D  can be made adaptive so as to approximate different regularization methods. If the instantaneous MSE is used to approximate the MSE, then a recursive algorithm for determining *w can be obtained by minimizing the objective function in (2) as )()()()( nnndne T xw−= , (3) )]()()([)()1( nnennn wRxww wξμ −+=+ , (4) where DDR Tw = . Following the transformation approach in [9-10], the input can be transformed to speed up the convergence of the LMS algorithms above and the following updating equation can be obtained )()()()( nnndne CT XW−= , (5) ))()()(()()1( nnennn WCx WRXDWW ξμ −+=+ , (6) where TLCCC nWnWnWnn )](,),(),([)()( ,2,1, ?== CwW , )(nCX  TLCCC nXnXnXn )](,),(),([)( ,2,1, ?== Cx  are, respectively, the transformed adaptive weight vector and signal vector. C is an LL ×  transformation matrix such as the DFT or DCT. TwW CCRR = and )](,),(),([diag11211 nnn Lx−−−= εεε ?D  is an element-wise normalization matrix with )(niε  being the estimated power of the i-th signal component after transformation. III. PERFORMANCE ANALYSIS In this section, we analyze the convergence performance of the proposed R-TDNLMS algorithm. The following assumptions are made:  (A1)  { )(nx } is an independent identically distributed (i.i.d) Gaussian random sequence with zero-mean and covariance matrix xxR ;  (A2)  )(nW , )(nx  and )(nη are statistically independent; (A3) the estimated power )(niε averages within a short period of time and hence xD  is a constant matrix. (A2) is the independence assumption, which is a good approximation for large value of L  and for small to medium step-size to simplify the convergence analysis of adaptive filtering algorithms. Moreover, we denote *WCCC dXXXPR 1−= , where )]()([ nndE CdX C XP =  is the ensemble-averaged cross-correlation vector between )(nCX  and )(nd . *W  is related to the optimal Wiener solution OPTw  as *1OPT CWPRw ==−dXXX . A. Mean Convergence Analysis Taking expectation on both sides of (6), we have )]}.([)({)]([)]1([nEnEnEWXXdXxCCCWRRPDWWξμ+−⋅+=+ (7) First we assume that the step-size is appropriately chosen so that the algorithm converges. At the steady state, (7) reads: CCC dXRWXXPWRR =+ )( ξ , (8) where )]([ ∞= WW ER is the desired regularized Wiener solution.   Next we examine the convergence rate by introducing the weight error vector Rnn WWv −= )()(  in (7): )]([))(()]1([ nEnE WXXx CC vRRDIv ξμ +−=+ , (9) where I  is the identity matrix.  Let TUΛU ~~~ be the eigendecomposition of 2/12/1 ~ xXXx CC DRDRxx =  with WXXXX CCCCRRR ξ+=~ . Using (9) and expressing )(nv  in the coordinate, )(~)( 2/1 nn xT vDUV −= , we get the difference equation for the i-th element of )]([ nE V  as follows iii nEnE )]([)~1()]1([ VV λμ−=+ , (10) where iλ~  is the i-th eigenvalue of xxR . Thus, the mean weight vector of the adaptive filter will converge if iλμ~/20 << . (11) Therefore, the maximum possible step-size is maxmax~/2 λμ = , where max~λ  is the maximum eigenvalue of xxR . It can be seen that if the input is so distributed that CC XXR has zero eigenvalues, then these eigenmodes can never converge and the solution may be significantly biased. With sufficient regularization, the eigenvalues of xxR  can be made nonzero and hence the solution will be given by (8) with a controllable bias. B. Mean Square Convergence Analysis To evaluate the mean square behavior, multiplying )(nv  by its transpose and taking expectation, one gets a difference equation of the weight error covariance matrix )(nΞ  )]()([ nnE Tvv=  as follows ,)])()()(())()()([(}~)()]([{)}(~)]([{)()1(2TxWTTCWCxTXXvwTdXwvXXTdXnnennnenEnnEnnEnnCCCCCCDRWXWRXDDRΞPvΞRvPDΞΞxxξξμμμ−⋅−+−+−+=+ (12) where TRvvTTwvvw nEnnnEnn WvΞWvΞΞ )]([)()]()([)()( +===  by noting that )(ˆ)()()](*[)( nnnnne CTCT XvXWW =+−= η  )(nη+ , where )()(*ˆ nnR vWvWWv −Δ=−−= . It can be shown that [11] the R-TDNLMS algorithm will converge in the mean squares sense if the step-size satisfies the following convergence condition  ,))(())(~(21xXXxXX nTrnTr CCCC DRDR +≤μ  (13) 697 which is very close to that of the conventional TDNLMS algorithm ))((31xXX nTr CC DR≤μ  without regularization.      If the algorithm converges, 0v =∞)]([E , and the last term in (12) will reduce to TxXX CCS DRAAAADx ])()()()([)(232102ησμ +∞+∞−∞−∞=∞ , ,))}(({2)(2)( 20CCCCCCCCXXXXvvTTTRRXXvvXXTr RRΞWWRWWRRΞRA WW∞+ΔΔ++∞=∞ ξ WW RΞRWWRAA ))(()()( 21 ∞−=∞=∞ vvXXTRRTCCξξ , WW RΞWWRA ))(()(23 ∞+=∞ vvTRRξ . Therefore, )()()()( 3210 ∞+∞−∞−∞ AAAA  ))(()(2 WW RRΞRR ξξ +∞+= CCCC XXvvXX  ))()((CCCC XXvvTvvXX RΞRRΞR WW ∞+∞− ξCCCCCC XXXXvvXXTTr RRΞRWW ))(( ∞+ΔΔ+WWWW RΞRRWWR )(22∞−+ vvTRR ξξ .      Assuming 2μ and 2ξ  are small, we can drop the last term to get ,])())()(())(()(2[)(22min*2TxTRRXXXXvvTvvXXXXvvXXCCCCCCCCCCJSDRWWRRRΞRRΞRRRΞRRDWWWWWWxξσξξξμ+++∞+∞−+∞+≈∞ (14) where  22min )( ησσ +ΔΔ= CC XXTTr RWW . This also gives an upper bound of the MSE since we dropped only the negative terms.  Substituting (14) into (12) gives )~)()(~()()( TXXvvvvXXvvvv CCCC xx DRΞΞRDΞΞ ∞+∞−∞≈∞ μ  xx DRΞRD CCCC XXvvXX~)(~2 2 ∞+ μCC XXJ RDx )[(2min*2 σμ ++  TxTRR DRWWR ww ]2′′+ ξ . (15) By expressing )(nv  in the transformed coordinate: =)(nV  )(~ 2/1 nxT vDU − , (15) can be further simplified to )(~~~)()( 2/12/1 nVVxXXxTCCΞUDRDUΞΞ VVVV μ−∞≈∞  TXXxVV CCn UDRDUΞ x~~~)( 2/12/1μ− )(2 ∞Γ+ μ  TxXXxTVVxXXxTCCCCUDRDUΞUDRDU ~~~)(~~~2 2/12/12/12/12 ∞+ μ , (16) where 212min* )()( Γ+Γ+=∞Γ σJ , TxXXx CCUDRDU ~~ 2/12/11 =Γ  and TxTRRx UDRWWRDU WW~~ 2/12/122 ξ=Γ . The diagonal values of (16) reads: iiiiiii ,,, )]([~2)]([)]([ ∞−∞≈∞ VVVVVV ΞΞΞ λμ 2μ+  ))]([)]()[()]([~2( ,2,12min*,2iiiiiii J ∞Γ+∞Γ++∞⋅ σλ VVΞ , (17) where iii ,1,1 )]([ Γ=∞Γ , and iii ,2,2 )]([ Γ=∞Γ . On the other hand, the EMSE is given by )~~)(())(( 2/12/1* UDRDUΞRΞ VVvv xXXxTXX CCCCtrtrJ ∞=∞= . (18) If the transform can approximately diagonalize xxR , then IDRDRxx ≈=2/12/1 ~xXXx CC, and U~  are approximately equal to the identity matrix. Hence, 2/12/12/12/12/12/1 ~~~~~~xWxxWxTxXXxTCCDRDΛUDRDUΛUDRDU ξξ −≈−= . Moreover, for diagonal dominance WR , we further have ),~()]([))~)(((,_11,2/12/1*iiWiLiiiixWxtrJRΞDRDΞVVVV−=−∞≈−Λ∞=∑ ξελξ (19) where iiW ,_R  is the i-th diagonal value of WR .  Solving for ii ,)]([ ∞VVΞ  from (17), one gets ))(()~1(~2)]([ ,2,12min*, iiiiii J Γ+Γ+−=∞ σμλλμVVΞ . (20) Consequently, *J is found to be )1()(RNLMS21RNLMS2min21* μφφφσμ ξ−+=J , (21) where ∑=−−−Γ=Li iiiiWiiirR1,_1,1RNLMS )~1(~)~(μλλξελφ , ∑=−−−Γ=Li iiiiWiiirR1,_1,2)~1(~)~(μξ λλξελφ . IV. SIMULATION RESULTS In this section, computer simulations are conducted to evaluate the convergence behavior of the proposed algorithm and verify the analytical results obtained in section III. As a comparison, we also consider the conventional TDNLMS algorithm. The DCT transformation is employed due to its wide usage and efficiency in practice. The power of the input element is estimated recursively by using a forgetting factor )()1()1()( 2, nXnn iCii εε αεαε +−−=  with εα =0.01. All simulations are performed using the system identification model and the results are averaged over 200 independent runs. A. Experiment 1: Colored Gaussian Input In this experiment, the following first order autoregressive (AR) process is employed as the input: )1(95.0)( −= nxnx  )(ng+ , where )(ng  is a zero-mean and white Gaussian noise. The input power has been normalized. The unknown system to be estimated is an L-order (L=15) FIR filter. Different signal-to-noise ratios (SNRs) at the system output (SNR=0, 10 and 20 dB) are used to examine the effect of the regularization on the proposed algorithm. The step-size for the TDNLMS and R-TDNLMS algorithms is chosen as 0.007. The regularization parameter for R-TDNLMS is chosen as =ξ 0.1, 0.02 and 0.004 for different SNRs. The learning curves of EMSE for both algorithms are shown in Figs. 1(a), (b), and (c). It can be seen that the R-TDNLMS algorithm generally converges faster and to a lower steady-state EMSE compared to the conventional TDNLMS algorithm. The advantages are more significant when the SNR is low. The estimated steady-state EMSE agree well with the simulation results. B. Experiment 2: Power Varying Input 698  In this experiment, the input signal is a segment of music. The SNR is set to be 15 dB. The system order is L=8. The step-size for both algorithms are 0.007. The regularization parameter for the R-TDNLMS algorithm is chosen as =ξ 0.02. The learning curves of EMSE are shown in Fig. 2. It can be seen that the TDNLMS algorithm is very sensitive to the input signal because the input power is varying considerably. The R-TDNLMS algorithm, on the other hand, has a high immunity to variation in input power and it achieves a lower steady-state EMSE values compared to the conventional TDNLMS algorithm. Fig. 2  Learning curves of EMSE for the time-invariant channelidentification problem with music input in experiment 2 at SNR=15 dB. V. CONCLUSION A new regularized TD-NLMS adaptive filtering algorithm has been proposed. The algorithm is obtained by adding a penalizing term on the weighted 2-norm of the coefficient vector to the conventional MSE cost function. New difference equations describing the mean and mean square convergence behaviors of the proposed algorithm are developed. New expressions for step-size bound and EMSE are also derived. The R-TDNLMS algorithm is found to have a lower EMSE for coloured input and better immunity to variation of input signal power than the conventional TDNLMS. Theoretical analysis is also found to agree well with computer simulation under different conditions tested. REFERENCES [1] B. Widrow, J. M. McCool, M. G. Larimore, and C. R. Johnson, Jr., “Stationary and nonstationary learning characteristics of the LMS adaptive filter,” in Proc. IEEE, vol. 64, pp. 1151-1162, Aug. 1976. [2] J. I. Nagumo and A. Noda, “A learning method for system identification,” IEEE Trans. Automat. Contr., vol. AC-12, pp. 282-287, June 1967. [3] S. Narayan, A. M. Peterson and M. J. Narasimha, “Transform domain LMS algorithm,” IEEE Trans. Acoust. Speech, Signal Process., vol. ASSP-31, pp. 609-615, June 1983. [4] J. C. Lee and C. K. Un, “Performance of transform-domain LMS adaptive digital filters,” IEEE Trans. Acoustics, Speech, and Signal Process., vol. 34, No. 3, pp. 499-510, 1986. [5] S. C. Chan and Y. Zhou, “On the convergence analysis of the transform domain normalized LMS and related M-estimate algorithms,” in Proc. IEEE APCCAS 2008, Macao, Nov 30 - Dec 27, 2008. [6] J. Fan and R. Li, “Variable selection via nonconcave penalized likelihood and its oracle properties,”. J. Am. Stat. Assoc., vol. 96, no. 456, pp. 1348-1360, Dec. 2001. [7] T. Van Waterschoot, G. Rombouts, and M. Moonen, “Optimally regularized adaptive filtering algorithms for room acoustic signal enhancement,” Signal Process., vol. 88, pp. 594-611, 2008. [8] Y. Chen, Y. Gu, and A. O. Hero, “Sparse LMS for system identification,” in Proc. ICASSP2009, Taipei, Taiwan, Apri 19-24, 2009, pp. 3125-3128. [9] S. C. Chan, Y. J. Chu, Z. G. Zhang, and Y. Zhou, “On the convergence behavior of the noise-constrained NLMS algorithm,” in Proc. IEEE ISCAS 2009, Taipei, May 24-27, 2009. [10] S. C. Chan, Z. G. Zhang Y. Zhou, and Y. Hu, “A new noise-constrained normalized least mean squares adaptive filtering algorithm,” in Proc. IEEE APCCAS 2008, Macao, China, Dec. 2008. [11] S. C. Chan, “A new regularized transform domain NLMS adaptive filtering algorithm – performance analysis and applications,” Internal report, Department of EEE, The University of Hong Kong, 2010.   (a)  (b)  (c) Fig. 1  Learning curves of EMSE for the time-invariant channel identification problem with first-order AR input in experiment 1 at SNR=  (a) 0dB  (b) 10dB  and (c) 20dB. 699

A new regularized transform-domain NLMS adaptive filtering algorithm

http://hub.hku.hk/bitstream/10722/137728/1/Content.pdf

A new regularized transform-domain NLMS adaptive filtering algorithm

Abstract

Similar works

Full text

Available Versions

HKU Scholars Hub