Van Hecke, Tanja

Ghent University Academic Bibliography

Applied Probability Trust (27 October 2016)DIRECT AND INDIRECT LEAST SQUARES APPROXIMATINGPOLYNOMIALS FOR THE FIRST DERIVATIVE FUNCTIONT. VAN HECKE,∗ Ghent UniversityAbstractFinite difference methods are useful to give a discrete approximation of thederivative function f ′ based on a set of data points (xi, fi) (i = 0, 1, 2, . . . , n).If a continuous function is required to represent the derivative functionand only a scatter of data points is available, finite difference formulas areinsufficient. This paper describes two different approaches to derive ananalytical description of the derivative function based on data points. Theirperformances are compared on several test functions where Monte Carlosimulations give statistics on the errors.Keywords: derivative; polynomial approximation; finite difference; least squaresdifference2010 Mathematics Subject Classification: Primary 65D25Secondary 65D101. IntroductionDerivatives are used for multiple purposes to represent change or in algorithms suchas the one of Newton-Raphson or Levenberg-Marquadt. In order to obtain an analyticaldescription of the derivative function one can start by constructing the approximatingpolynomial obtained by a regression method (e.g. least squares regression), which isdifferentiated afterward. Polynomials are a preferred class of functions due to the easeof differentiation. But it is an interesting research question to investigate whether amethod, where fitting is followed by differentiating, performs better than a methodwhere these actions are performed in reversed order. This differentiation processwill require a numerical method such as finite difference methods. In literature [5]one finds derivatives of interpolation functions as continuous approximations of the∗ Postal address: Ghent University, Faculty of Engineering and Architecture, Voskenslaan 270, Ghent,Belgium, Tanja.VanHecke@ugent.be12 T. VAN HECKEfirst derivative function. However for a limited number of interpolation points thisapproach leads to the classical finite difference formulas, while for a larger numberof interpolation points the strong oscillations of the approximation function are anundesired side effect. Another approach is based on nonparametric regression analysis[1]. Differentiating the nonparametric estimate with respect to the independent vari-able to obtain the first order derivative of the regression function only works well ifthe original regression function is extremely well estimated. When the data is noisy,it can lead to wrong derivative estimates. Two main approaches to overcome thisproblem are smoothing splines [4] and local polynomial regression [1]. They are basedon limiting the approximation analysis to local regions for functions having varyingestimation difficulty over the interval. In this paper we want to compare differentglobal approaches when the measurements are contaminated by overall noise.2. Mathematical background2.1. Combining least squares and finite differenceLeast Squares Difference (LSD) [2] is a well known method to fit data given by thepoints (xi, yi) (i = 0, 1, . . . , n), where a distance functionD =n∑i=0(yi − P (xi))2 (1)is minimized by optimized estimations of the model coefficients aj , j = 0, 1, . . . , k incase of a polynomial approximation P (x) =∑kj=0 aj xj of degree k (k < n).Finite differences [3] are widely used to approximate derivatives. Among the lowestorder finite difference approximations for the first derivative y′(xi) are• forward, first order: y′(xi) ≈yi+1 − yih• backward, first order: y′(xi) ≈yi − yi−1h• central, second order: y′(xi) ≈yi+1 − yi−12h• central, fourth order: y′(xi) ≈−yi+2 + 8 yi+1 − 8 yi−1 + yi−212h• central, sixth order: y′(xi) ≈yi+3 − 9 yi+2 + 45 yi+1 − 45 yi−1 + 9 yi−2 − yi−360hwith h = xi+1−xi. We refer to a method as a pth-order method if the truncation errorDirect and Indirect derivative approximation 3is of the order of O(hp).A direct method M1 can be applied to generate a polynomial approximation on thepoints (xi, yi) (i = 0, 1, . . . , n). The differentiation of the fitted polynomial is theapproximation result for the required derivative function y′. Indirect methods startby using finite difference methods to generate approximations for the derivative valuesyˆ′i for each of the data points (xi, yi). The least squares regression analysis is appliedafterward on (xi, yˆ′i) to generate a continuous approximation of the derivative functiony′. The indirect methodM2 is based on the first order forward finite difference formula,where the indirect methods M3, M4 and M5 are based on the second, fourth andsixth order central finite difference formulas respectively. To evaluate the methods Mj(j = 1, 2, 3, 4, 5) we consider an error function Ej withEj =n−1∑i=1(yˆ′i[j]− y′(xi))2, j = 1, 2, 3, 4, 5 (2)with estimation yˆ′i[j], being an approximation generated by method Mj for the valueof the derivative function in x = xi.2.2. Error analysisWe assume the output y(x) to be troubled by zero mean Gaussian noise ǫ(x) suchthaty(x) = T (x) + ǫ(x), (3)with T (x) the pure output. We call aˆ = (a0, a1, a2, . . . , ak)T the resulting vector ofthe least squares optimization with the direct method M1, which makesTˆ1′(x) = c(x)T aˆ with c(x)T = (0, 1, 2x, 3x2, . . . , k xk−1). (4)This implies that the variance of the estimated Tˆ1′(x) is related to the covariancematrix Σaˆ of the least squares estimates of the polynomial coefficients asVar(Tˆ1′(x)) = c(x)T Σaˆ c(x). (5)The accuracy of the estimation of the derivative function obtained with M1 will bedirectly influenced by the quality of the least square estimation of vector a as a resultof the noise ǫ.4 T. VAN HECKEWith bˆ = (b0, b1, b2, . . . , bk−1)T the resulting vector of the least squares optimiza-tion with the indirect method Mj (j = 2, 3, 4, 5), the estimated derivative functioncan be written asTˆj′(x) = d(x)T bˆ with d(x)T = (1, x, x2, . . . , xk−1). (6)This implies that the variance of the estimated Tˆj′(x) is related to the covariancematrix Σbˆof the least squares estimates of the polynomial coefficients asVar(Tˆ1′(x)) = d(x)T Σbˆd(x). (7)The accuracy of the estimation of the derivative function obtained with Mj (j =2, 3, 4, 5) will be directly influenced by the quality of the least square estimation ofvector b that reflects the noise as well as the order of the applied finite differencescheme.3. Validation resultsWe compared the methods discussed in the previous paragraph on the testfunctionsT1(x) = ex (8)T2(x) = cos(3x− 2).Within the interval [−1, 2] we generated data (xi, yi) with xi = −1 + i h, yi =T (xi) + ǫi (i = 0, 1, . . . , 30) and step size h = 0.1. The added noise ǫi is Gaussiandistributed with zero mean and variance σ. In Figure 1 a polynomial degree of k = 7and σ = 0.01 is used for different test functions Tj (j = 1, 2). The residuals (yˆ′i[j]−T ′(xi)) are plotted for x ∈ [−1, 2] when the methods M1, M2 and M3 are applied onT1 and T2. We see that the loss in accuracy due to lower order finite difference formulasis most prominent near the borders of the interval.A Monte-Carlo simulation with 100 trials leads to box plots of the errors Ej (j =1, 2, 3, 4, 5) for σ = 0.2 and σ = 0.4 (h = 0.3) as in Figure 2 with k = 6 and k = 8when the test function T1(x) = ex is used. The loss of accuracy due to highly oscillatorybehavior when the error scale for k = 8 is compared with k = 6. When k = 6 theDirect and Indirect derivative approximation 5Figure 1: Residuals for the derivative function with method M1 (*), method M2 (.) andmethod M3 (-) in case of T1(x) = ex (left) and T2(x) = cos(3x− 2) (right).method M2 performs worst as the finite difference order is too low and cannot becorrected by an accurate regression polynomial.1 2 3 4 50102030405060method  σ=0.2σ=0.41 2 3 4 5050100150200250300method  σ=0.2σ=0.4Figure 2: Boxplots of a 100-trials Monte-Carlo simulation of the residuals for the derivativefunction with methods Mi (i = 1, 2, 3, 4, 5) in case of k = 6 (left) and k = 8 (right) andT (x) = ex.4. ConclusionOur comparative analysis revealed that when higher order central finite differenceformulas are used, the indirect method where least squares approximation is precededby finite difference formulas, performs better than the direct method M1, where thecontinuous least squares approximation function is derived to create a continuousapproximation of the derivative function. Results from polynomials of lower degree6 T. VAN HECKEused for least squares regression with limited accuracy, can be improved when combinedwith higher order finite difference formulas.References[1] Fan, J. and Gijbels, I. (1996). Local Polynomial Modelling and its Applications,Chapman and Hall.[2] Fornberg, B. (1988). Generation of Finite Difference Formulas on ArbitrarilySpaced Grids. Mathematics of Computation 51 (184), 699-706.[3] Hazewinkel, M. (2001). Finite difference calculus, Encyclopedia of Mathemat-ics, Springer.[4] Stone, C. (1985). Additive regression and other nonparametric models, Ann.Statist. 13(2), 689-705.[5] Suli, E. and Mayers, D. (2003). An Introduction to Numerical Analysis,Cambridge University Press.

Direct and indirect least-squares approximating polynomials for the first derivative function

https://biblio.ugent.be/publication/8532419/file/8532423

Direct and indirect least-squares approximating polynomials for the first derivative function

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography