Cheong Took, Clive

Mandic, Danilo

Mohomad Ibunu, Mohomad

Moore, Karl

English

Constraining a group of taps of an adaptive filter to a single value may seem like a futile task, as weight sharing reduces the degree of freedom of the algorithm, and there are no obvious advantages for implementing such an update scheme. On the other hand, weight sharing is popular in deep learning and underpins the success of convolutional neural networks (CNNs) in numerous applications. To this end, we investigate the advantages of weight sharing in single-channel least mean square (LMS), and propose weight sharing LMS (WSLMS) and partial weight sharing LMS (PWS). In particular, we illustrate how weight sharing can lead to numerous benefits such as an enhanced robustness to noise and a computational cost that is independent of the filter length. Simulations support the analysi

Ibunu, Shamahil

Royal Holloway - Pure

Weight sharing for single-channel LMSShamahil Ibunu, Karl Moore and Clive Cheong TookDepartment of Electronic EngineeringRoyal Holloway, University of LondonTW20 0EX United Kingdom{phee004, pjee001}@live.rhul.ac.uk, clive.cheongtook@rhul.ac.ukDanilo MandicDepartment of Electrical and Electronic EngineeringImperial College LondonSW7 2BX United Kingdomd.mandic@ic.ac.ukAbstract—Constraining a group of taps of an adaptive filterto a single value may seem like a futile task, as weight sharingreduces the degree of freedom of the algorithm, and there are noobvious advantages for implementing such an update scheme. Onthe other hand, weight sharing is popular in deep learning andunderpins the success of convolutional neural networks (CNNs) innumerous applications. To this end, we investigate the advantagesof weight sharing in single-channel least mean square (LMS),and propose weight sharing LMS (WSLMS) and partial weightsharing LMS (PWS). In particular, we illustrate how weightsharing can lead to numerous benefits such as an enhancedrobustness to noise and a computational cost that is independentof the filter length. Simulations support the analysis.I. INTRODUCTIONWeight sharing is a common practice in machine learningdue to its numerous benefits such as minimising the risk ofoverfitting to the data and easier hyperparameter tuning dueto a reduction in the number of parameters. However, this lowcomputational complexity advantage is not the main objectivefor weight sharing. The main motivation for weight sharingis that it enforces the algorithm to detect common featuresscattered across the data [1]. As such, this particular propertyis not exhibited by LMS-type algorithms designed to havelow computational complexities such as [2], the partial updateLMS [3], and the online censoring algorithms [4], [5].In fact, properties such as robustness to noise, convergenceand stability are usually of high interest to the signal pro-cessing community including adaptive filtering algorithms.As such, the main aim for reducing the degrees of freedomof algorithms is to alleviate the computational cost at theexpense of these benefits. Weight sharing (WS) goes beyondthe issue of computational complexity. It was introduced formultichannel least mean square (MLMS) algorithms and thedesirable properties of WS were also investigated in our workin [6]. In particular, weight sharing has been shown to lead toi) an improved stability of the adaptive algorithms especiallyfor high condition numbers; ii) an enhanced robustness tonoise; and iii) its ability to cope with large number of channelssuch as in massive multiple inputs multiple outputs (MIMO)applications.Our previous work implemented weight sharing acrossdifferent channels as shown in Section II-A. This presentwork, instead, considers weight sharing across time for sin-gle channel processing. Moreover, partial weight sharing isalso considered to enhance the learning ‘capacity’ for morecomplex signals/systems by relaxing the strict requirementsof full weight sharing [6].II. WEIGHT SHARING FOR ADAPTIVE FILTERINGA. Weight Sharing for Multichannel LMSThe weight sharing multichannel least mean square (WS-MLMS) algorithm was introduced to illustrate how weightsharing can be incorporated into the class of LMS algorithms.In the context of multichannel adaptive processing, weightsw(n) can be shared across data channels as [6]yq(n) = wT(n)xq(n) q = 1, . . . , Q (1)where xq(n) and yq(n) represent the qth input and outputchannel respectively and (·)T is the transpose operator. Weightsharing mechanism can be expressed asy1(n)y2(n)...yQ(n) =L∑ℓ=1wℓ(n)x1(n− ℓ+ 1)x2(n− ℓ+ 1)...xQ(n− ℓ+ 1) (2)where wℓ(n) denotes the weight for the ℓth lag of the tapinput. This work addresses weight sharing across time andintroduces the weight sharing LMS (WSLMS) and the partialweight sharing (PWS) LMS.B. Proposed Weight Sharing LMS (WSLMS)In weight sharing LMS (WSLMS), a single coefficientfiltering w(n) can be shared amongst all taps asy(n) = w(n)L∑ℓ=1x(n− ℓ+ 1) (3)Based on the instantaneous quadratic error e2(n), it can beshown that the weight update for WSLMS is given byw(n+ 1) = w(n) + µe(n)L∑ℓ=1x(n− ℓ+ 1) (4)Eq. (3), however, imposes a strict form of weight sharingin WSLMS, and may not be appropriate in models wherethe coefficients are significantly different from each other. Toaddress this shortcoming, a more flexible model of weightsharing is proposed in the subsequent algorithm referred aspartial weight sharing (PWS). PWS allows some taps toshare the same coefficient, whilst also enabling the othertaps to have different values. In this way, PWS can benefitfrom the additional degrees of freedom, whilst exhibiting theadvantageous properties of weight sharing as mentioned in theintroduction.C. Proposed Partial Weight Sharing LMS (PWS)In PWS, the first K taps are allowed to have different weightcoefficients, whilst the last L−K taps share the same weight.As such, the output y(n) can be computed asy(n) =K∑ℓ=1wℓ(n)x(n−ℓ+1)+w(n)L∑ℓ=K+1x(n−ℓ+1) (5)For the first K taps, the ℓth coefficient wℓ(n) can updatedaswℓ(n+1) = wℓ(n)+µPartial1[e(n)x(n−ℓ+1)] ℓ = 1, ...,K(6)The last L − K taps share the same weight w(n), whichcan be updated asw(n+ 1) = w(n) + µPartial2[e(n)L∑ℓ=K+1x(n− ℓ+ 1)] (7)Observe that (6) and (7) requires two different stepsizes,µPartial1 and µPartial2 which is discussed in the next section.III. COMPARATIVE ANALYSESThis section analyses and compares the proposed algorithmsWSLMS and PWS with LMS from a practical perspective. Inparticular, their robustness against noise, together with theircomputational and convergence properties are next analysed.A. Robustness against noiseWhile LMS is known to be robust against noise comparedto the recursive least squares at low signal-noise-ratio (SNR)values [7], its weight update relies on the raw values ofthe input x(n). Its weight update does not have any inbuiltmechanism to combat the effect of the noise. Unlike the LMS,the update for WSLMS (4) averages/smoothes its input as:w(n+ 1) = w(n) + µe(n)L∑ℓ=1x(n− ℓ+ 1)= w(n) + ηe(n)1LL∑ℓ=1x(n− ℓ+ 1)︸ ︷︷ ︸Moving average of inputs(8)Similarly, PWS benefits from the moving average mecha-nism in (7), which mitigates the effects of noise (albeit to alesser extent). As such, it is expected that PWS performs betterthan LMS in noisy settings, yet worse than WSLMS.Step Algorithm Addition MultiplicationEstimate LMS L− 1 Ly(n) WSLMS (3) L− 1 1PWS (5) L− 1 K + 1Error LMS 1 0e(n) WSLMS 1 0PWS 1 0Update LMS L L+ 1w(n+ 1) WSLMS (4) L+ 1 2PWS (6) and (7) L K + 2Total LMS O(L) O(L)WSLMS O(L) O(1)PWS O(L) O(K)TABLE ICOMPUTATIONAL COMPLEXITIES OF LMS, WSLMS, AND PWSALGORITHMS, WHERE L REPRESENTS THE FILTER LENGTH, ANDK THE NUMBER OF NON-SHARED COEFFICIENTS IN PWS.B. Computational ComplexitiesTable II compares the computational cost of each algorith-mic step of LMS, WSLMS, and PWS. They all have the samecomputational cost in terms of additions.However, the computational cost for multiplication is sig-nificantly different for each algorithm. For WSLMS, thecomputational cost is constant, and does not depend on thefilter length. For PWS, the computational cost is dependenton the number of non-shared weights. For example, if only50% of the weights are updated, the computational cost isreduced by half - a non-trivial benefit for long filters.C. Convergence issuesAn optimal stepsize plays a fundamental role in ensuringthe convergence of an adaptive algorithm. To this end, opti-mal stepsizes are derived based on the one-step ahead errorprediction e(n + 1). A first order approximation of a Taylorseries expansion of a function f(n) can be expressed asf(n+∆n) ≈ f(n) + ∆nf ′(n) (9)where f ′(n) denotes the first order derivative. Thus, the one-step ahead error prediction e(n+ 1) can be approximated ase(n+ 1) ≈ e(n) +L∑ℓ=1∂e(n)∂wℓ(n)∆wℓ(n) (10)which provides the basis for deriving optimal stepsizes.1) Optimal Stepsize for LMS: For the LMS algorithm, ob-serve that both ∂e(n)∂wℓ(n) and ∆wℓ(n) depends on the individualinput x(n− ℓ+ 1), i.e.∂e(n)∂wℓ(n)= −x(n− ℓ+ 1) (11)∆wℓ(n) = µx(n− ℓ+ 1)e(n) (12)Replacing (11)-(12) into (10) yields the one-step ahead errorprediction, given bye(n+ 1) ≈ e(n)− µe(n)L∑ℓ=1x2(n− ℓ+ 1) (13)-20 -15 -10 -5 0 5 10 15 20SNR (dB)-25-20-15-10-50510MSE (dB)WSLMSPWSLMSFig. 1. Evaluation the robustness of LMS, WSLMS, and PWS over a rangeof signal-to-noise ratio values.Factorising the common terms in (13) leads toe(n+ 1) ≈ e(n)[1− µL∑ℓ=1x2(n− ℓ+ 1)](14)For the one-step ahead quadratic error to go to zero, we have[1− µL∑ℓ=1x2(n− ℓ+ 1)]= 0 (15)which leads to the well-known normalised stepsizeµLMS =1∑Lℓ=1 x2(n− ℓ+ 1)(16)2) Optimal Stepsize for WSLMS: Observe that both ∂e(n)∂wκ(n)and ∆wκ(n) depend on the sum of inputs, i.e.∂e(n)∂wκ(n)= −L∑κ=1x(n− κ+ 1) (17)∆wκ(n) = µL∑κ=1x(n− κ+ 1)e(n) (18)Replacing (17)-(18) into (10) yields the one-step ahead error:e(n+ 1) ≈ e(n)−L∑ℓ=1µe(n)[ L∑κ=1x(n− κ+ 1)]2≈ e(n)− µe(n)L[ L∑κ=1x(n− κ+ 1)]2(19)Factorising the common error term in (19) leads toe(n+ 1) ≈ e(n)[1− µL( L∑κ=1x(n− κ+ 1))2](20)∴ µWSLMS =1L(∑Lκ=1 x(n− κ+ 1))2 (21)0 1000 2000 3000 4000 5000 6000Samples10-210-1100101102e2(t)WSLMSPWSLMSFig. 2. Transient performance of algorithms considered for the identificationa time-varying system.Algorithms LMS µLMSWeight Sharing (4) µWS 1/LPartial Weight Sharing (6) µPartial1 L/KPartial Weight Sharing (7) µPartial2 L/(L−K)2TABLE IICOMPARATIVE ANALYSIS IN TERMS OF OPTIMAL STEPSIZES.Remark: The normalising factor in the stepsize depends onthe sum of squares in LMS in (16), whereas it is the square ofthe sum in WSLMS in (21). These two differing normalisingfactors converges to the same value for independent andidentically distributed (i.i.d.) samples.3) Optimal Stepsize for PWS: Unlike WSLMS and LMS,PWS requires two optimal stepsizes for its updates in (6) and(7). Based on Section III-C1, the optimal stepsize for (6) canbe obtained asµPartial1 =1∑Kℓ=1 x2(n− ℓ+ 1)(22)The optimal stepsize for (7) can be similarly derived asµPartial2 =1(L−K)(∑Lκ=K+1 x(n− κ+ 1))2 (23)Table II summarises all stepsizes in terms of that of the LMSalgorithm for optimal performances of the adaptive algorithmsfor i.i.d samples.IV. SIMULATIONSThe aim of the simulations is twofold. First, the robustnessof the adaptive algorithms against noise was investigated.Second, the adaptive capability of the proposed algorithmsto track time-varying systems/signals in noise-free settings.The filter length of L = 50 and the number of non-sharedcoefficients of K = 25 in PWS were set in all experiments.The validity of the optimal sizes in Table II was verified inTable III.0 500 1000 1500 2000 2500 3000-2024Actual0 500 1000 1500 2000 2500 3000-2024WSLMS0 500 1000 1500 2000 2500 3000-2024PWS0 500 1000 1500 2000 2500 3000Samples-2024LMS0 500 1000 1500 2000 2500 3000Samples10-1410-1210-1010-810-610-410-2100102e2(t)  [dB]WSLMSPWSLMSFig. 3. Transient performances of algorithms considered in a one-step ahead prediction of three chaotic signals. The left panels show the estimated signals,with artifacts highlighted by arrows. The right panel shows the learning curves of WSLMS, PWS, and LMS.A. Simulation 1: Robustness against noise in signal predictionAdditive white Gaussian noise was added to the Mackey-Glass signal [8] to vary the signal-to-noise ratio (SNR) from-20 dB to 20 dB. For each SNR value, 100 trials were run andaveraged to yield the performance curves in mean-squared-error (MSE) sense, as shown in Fig. 1. Observe that WSLMSexhibited the lowest MSE performance consistently. On theother hand, PWS was superior to LMS only at very lowSNRs, but performed similarly to LMS. K (the number ofnon-shared coefficients) was reduced to zero, its performancewould approach that of WSLMS.B. Simulation 2: Modelling more complex systems or signalsSim 2a on time varying system identification: Thefinite impulse response h(n) of the system was changedevery 2000 samples as follows: h1 = [0.9, 0.3, 0.5,−0.1],h2 = [0.4, 0.1, 0.2,−0.6], and h3 = [0.7, 0.4, 0.3, 0.1]. Thesynthetic input was generated as x(n) = 0.8x(n− 1) + ω(n),where ω(n) a unit variance white Gaussian noise. Fig. 2 showsthat PWS which outperformed both LMS and WSLMS. Thiswas confirmed by its lowest MSE of -3.52 dB compared to-2.12 dB (LMS) and 3.61 dB (WSLMS).Sim 2b on time-varying signal prediction: To assess theability of the considered algorithms to track a time-varyingsignal, we used a synthetic signal made up of 3 differentsignals: Mackey-Glass [8], Saito [9] , Lorenz [10] - each oflength 1000 samples as shown in top left panel of Fig. 3. Thelearning curves are shown in the right panel of Fig. 3. Observethat PWS and LMS performed similarly and outperformedWSLMS. The artifacts of the estimate by WSLMS were morepronounced than those of PWS and LMS in the left panels.Stepsizes µLMS µWSLMS µPartial1 µPartial2Sim 1 (×10−3) 5 0.2 (0.1) 6 (10) 0.3 (0.4)Sim 2a (×10−3) 1 0.04 (0.02) 2 (2) 0.08 (0.08)Sim 2b (×10−2) 1 0.02 (0.02) 1 (2) 0.04 (0.08)TABLE IIICOMPARISON BETWEEN EXPERIMENTAL AND THEORETICAL OPTIMALSTEPSIZES. THE THEORETICAL VALUES IN (·) WERE DERIVED FROMTABLE II.C. On the validation of the optimal stepsizesTable III compares the experimental values with the theo-retical values. Observe that the theoretical values for µWSLMSare more accurate than those of µPartial1 and µPartial2. This isbecause the two updates in (6) and (7) of PWS affect eachother, although their optimal stepsizes were derived as if theywere independent of each other. Yet, these theoretical stepsizesprovide useful indicative values to initialise the stepsizes, giventhey were derived based on an i.i.d assumption.V. CONCLUSIONWe have proposed two novel adaptive algorithms to intro-duce weight sharing across time rather than across channelsinto the LMS family of algorithms [6]. In particular, it wasshown that weight sharing provides an additional degree ofrobustness against noise in the form of a moving average in(8). On the other hand, weight sharing constrains the modellingcapability. To leverage the advantages of both weight sharingand the standard LMS, the partial weight sharing algorithmhas been introduced and shown to offer a good trade-off toimplement weight sharing in adaptive filtering algorithms.REFERENCES[1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learningapplied to document recognition,” Proceedings of the IEEE, vol. 86,no. 11, pp. 2278–2324, 1998.[2] F. G. A. Neto and V. H. Nascimento, “A novel reduced-complexitywidely linear QLMS algorithm,” in IEEE Statistical Signal ProcessingWorkshop, pp. 81–84, 2011.[3] M. Godavarti and A. Hero, “Partial update LMS algorithms,” IEEETransactions on Signal Processing, vol. 53, no. 7, pp. 2382–2399, 2005.[4] D. Berberidis, V. Kekatos, and G. B. Giannakis, “Online censoringfor large-scale regressions with application to streaming big data,”arXiv:1507.07536, 2015.[5] E. C. Mengüç, M. Xiang, and D. P. Mandic, “Online censoring basedcomplex-valued adaptive filters,” Signal Processing, vol. 200, p. 108638,2022.[6] C. Cheong Took and D. Mandic, “Weight sharing for LMS algorithms:Convolutional neural networks inspired multichannel adaptive filtering,”Digital Signal Processing, vol. 127, p. 103580, 2022.[7] S. Haykin, Adaptive Filter Theory. Prentice Hall (5th Edition), 2014.[8] M. Mackey and L. Glass, “Oscillation and chaos in physiological controlsystems,” Science, vol. 197, pp. 287–289, 1977.[9] K. Mitsubori and T. Saito, “Torus doubling and hyperchaos in a fivedimensional hysteresis circuit,” IEEE Int. Symposium on circuit andsystems, vol. 6, pp. 113–116, 1994.[10] L. E. Norton, “Deterministic nonperiodic flow,” Journal of the Atmo-spheric Sciences, vol. 20, pp. 130–141, 1963.

Weight sharing for single-channel LMS

https://pure.royalholloway.ac.uk/ws/files/56170288/SSP2023-6.pdf

Weight sharing for single-channel LMS

Abstract

Similar works

Full text

Available Versions

Royal Holloway - Pure