For the given data $(p_i,t_i,f_i),$ $i=1,ldots,m$, we consider the
existence problem of the best parameter approximation of the exponential model function in the sense of ordinary least squares and total least squares.
Results related to that problem which have been obtained and published by the authors so far are given in the paper, as well as some new results on nonuniqueness of the best parameter approximation

D. Jukić

R. Scitovski

For the given data $(p_i,t_i,f_i),$ $i=1,\ldots,m$, we consider the

existence problem of the best parameter approximation of the exponential model function in the sense of ordinary least squares and total least squares.

Results related to that problem which have been obtained and published by the authors so far are given in the paper, as well as some new results on nonuniqueness of the best parameter approximation

Scitovski, R.

Jukić, D.

Hrčak - Portal of scientific journals of Croatia

Mathematical Communications 4(1999), 53-61 53Analysis of solution of the least squares problem∗Rudolf Scitovski† and Dragan Jukic´‡Abstract. For the given data (pi, ti, fi), i = 1, . . . ,m, we considerthe existence problem of the best parameter approximation of the expo-nential model function in the sense of ordinary least squares and totalleast squares. Results related to that problem which have been obtainedand published by the authors so far are given in the paper, as well assome new results on nonuniqueness of the best parameter approximation.Key words: exponential growth model, ordinary least squares, totalleast squares, existence problemAMS subject classifications: 65D10, 62J021. IntroductionWe consider the parameter estimation problem for the exponential model functionf(t; b, c) = b ect, (1)on the basis of experimental or empirical data (pi, ti, fi), i = 1, . . . ,m, where tidenote the values of the independent variable, fi the respective function values andpi > 0 are the data weights. Mathematical models described by an exponentialfunction or a linear combination of such functions are very often used in differentareas of applied research, e.g. biology, chemistry, electrical engineering, economy,nuclear physics, medicine, etc. (see [4], [12], [13], [17]).If the errors in the measurements of the independent variables are negligible,and the errors in the measurements of the dependent variable are independentrandom variables following the normal distribution with expectancy zero, then inpractical applications the unknown parameters b and c of the function (1) are usuallyestimated in the sense of the ordinary least squares (OLS) (see [3], [2]):min(b,c)∈R2S(b, c), S(b, c) = 12m∑i=1pi(becti − fi)2. (2)∗This paper was presented at the Special Section: Appl. Math. Comput. at the 7th InternationalConference on OR, Rovinj, 1998.†Faculty of Electrical Engineering, University of Osijek, Kneza Trpimira 2B, HR-31 000 Osijek,Croatia, e-mail:scitowsk@drava.etfos.hr‡Faculty of Food Technology, University of Osijek, F.Kuhacˇa 18, HR-31 000 Osijek, Croatia,e-mail:jukicd@oliver.efos.hr54 R. Scitovski and D. Jukic´If considerable errors occur in the measurements of independent variables as wellas dependent variables, it is reasonable to estimate the parameters b and c byminimizing all errors. Geometrically, it means that we will minimize the weightedsum of squares of the distance di from the data points (ti, fi) to the curve t 7→f(t; b, c) (see Fig. 1):min(b,c,δ)∈R2×RmF (b, c, δ), F (b, c, δ) = 12m∑i=1pi[(bec(ti+δi) − fi)2 + δ2i], (3)where δ = (δ1, . . . , δm)T ∈ Rm. This approach is known in literature as the totalleast squares (TLS) problem (see [1], [7], [16]).-6qq qεi(ti, fi)yta) OLS approach-6qq qytb) TLS approach@@HAAεiδidi(ti, fi)Figure 1. Ordinary and total least squares approachRemark 1. A 3-parametric exponential regression model f(t; a, b, c) = a + b ect isvery popular in applied research. For example, in [12] it is used in analysis oflong–term selection experiments in biology. Also, this model is frequently used as atest function for testing numerical algorithms for function minimization. In [8] weconsider the existence problem for the best least squares approximation of parametersfor this model function.If among the data strongly deviating data can appear, so-called “outliers”, theninstead of the least squares approximation of parameters the so-called robust approx-imation can be used. In general, Lp, p > 0 aproximation can be considered. In [5]the total Lp-norm approximation problem for the exponential function is considered.The following example shows that neither the OLS problem (2) nor the TLSproblem (3) has always a solution.Example 1. Let the given data (1, i, fi), i = 1, . . . ,m, satisfyf1 = f2 = . . . = fm−2 = 0, fm−1 = −1 and fm = 1.If (bn, cn) is the sequence in R2, such thatbn = e−mcn and cn →∞,Analysis of solution of the least squares problem 55then S(bn, cn) → 12 . It is easy to show that S(b, c) > 12 for all (b, c) ∈ R2, whichmeans that in this example the OLS problem (2) does not have a solution.In this example also the TLS problem (3) does not have a solution. Namely,F (bn, cn,0)→ 12 . On the other hand, F (b, c, δ) > 12 for all (b, c, δ) ∈ R2 × Rm.It can be shown that both the OLS problem (2) and the TLS problem (3) havethe solution, provided the data satisfy only the natural conditions:Theorem 1. Let the data (pi, ti, fi), i = 1, . . . ,m, be given, such thatt1 < t2 < . . . < tm & fi > 0 (∀i = 1, . . . ,m),and let B = {(b, c) ∈ R2 : b > 0}.(i) Then there exists a point (b?, c?) ∈ B, at which the functional S defined by(2) attains the global minimum on the set R2.(ii) Then there exists a point (b?, c?, δ?) ∈ B × Rm, at which the functional Fdefined by (3) attains the global minimum on the set R2 × Rm.The proof of the statement (i) can be found in [7], and the proof of the statement(ii) can be found in [4].The results of Theorem1. will be specified further depending on the data char-acteristic.2. Data classificationUnder natural conditions on the data Theorem1. ensures the existence of optimalparameters b?, c? both in the sense of OLS and in the sense of TLS. However, itdoes not say anything whether the model function (1) should be sought in the classof increasing or decreasing exponential functions. It is natural to describe the in-creasing data by an increasing, and the decreasing data by a decreasing exponentialfunction. For that purpose the data will be classified into three main groups andfor each of them the existence problem will be analysed separately. In that way thechoice of the initial approximation of the parameters will be used. Therefore, weintroduce the following definition (see also [6], [10], [15]).Definition 1. The data (pi, ti, fi), i = 1, . . . ,m, are said to have the preponderantincrease (resp. preponderant decrease) property if the slope of the associated linearregression is positive (resp. negative). If this coefficient is equal to zero, then thedata are said to be preponderantly stationary.Remark 2. The coefficients k and l of the corresponding linear regression ϕ(t) =k t+ l can be found by minimizing the functionalG(k, l) = 12m∑i=1pi (k ti + l − fi)2 .56 R. Scitovski and D. Jukic´By equating the gradient of the functional G with zero, it is easy to show thatk = D1D , whereD =∣∣∣∣∣∑pit2i∑piti∑piti∑pi∣∣∣∣∣ , D1 =m∑i=1pitifim∑i=1pi −m∑i=1pitim∑i=1pifi.Because of the Cauchy-Schwarz inequality, D > 0. This means that the data(pi, ti, fi), i = 1, . . . ,m, have the preponderant increase property if and only ifD1 > 0, and they have the preponderant decrease property if and only if D1 < 0.The data are preponderantly stationary if and only if D1 = 0.The condition of preponderant increase [resp. preponderant decrease] of datais weaker than the condition of increase [resp. decrease] of data, as stated in thefollowing proposition (for the proof see [15]).Proposition 1. Let the data (pi, ti, fi), i = 1, . . . ,m, be given. If(f1 ≤ . . . ≤ fm)& (f1 < fm),then the data have the property of preponderant increase. If(f1 ≥ . . . ≥ fm)& (f1 > fm),then the data have the property of preponderant decrease.Remark 3. Note that conditions D1 > 0, resp. D1 < 0, correspond to well-knownChebyschev’s inequalities:m∑i=1pitifim∑i=1pi −m∑i=1pitim∑i=1pifi > 0, (4)resp.m∑i=1pitifim∑i=1pi −m∑i=1pitim∑i=1pifi < 0. (5)(see [11], [14], [16]).Note also if the data are centered around the origin, then conditions (4), resp.(5), become simpler:m∑i=1pitifi > 0, resp.m∑i=1pitifi < 0, (6)The following theorem shows that preponderantly increasing [resp. preponder-antly decreasing] data can be described by an increasing [resp. decreasing] expo-nential model (for the proof see [4], [15], [16]). The existence problem for prepon-derantly stationary data will be considered in the next section.Theorem 2. Let the data (pi, ti, fi), i = 1, . . . ,m, be given and suppose that fi > 0,i = 1, . . . ,m. ThenAnalysis of solution of the least squares problem 57(i) If the data have the property of preponderant increase (4),a) then there exists a pair (b?, c?) ∈ intU ,U = { (b, c) ∈ R2 : b ≥ 0, c ≥ 0} ,which minimizes on U the functional S defined by (2).b) then there exists an (m+ 2)- tuple (b?, c?, δ?) ∈ intU ×Rm, which min-imizes on U × Rm the functional F defined by (3).(ii) If the data have the property of preponderant decrease (5),a) then there exists a pair (b?, c?) ∈ intV,V = { (b, c) ∈ R2 : b ≥ 0, c ≤ 0} ,which minimizes on V the functional S defined by (2).b) then there exists an (m+ 2)- tuple (b?, c?, δ?) ∈ intV ×Rm, which min-imizes on V × Rm the functional F defined by (3).2.1. Preponderantly stationary data. Nonuniqueness of thebest LS approximationTheorem1. and Theorem2. assure the existence of the best least squares approxima-tions, but do not tell us anything about either the uniqueness or about a method forfinding such a best approximation. By [2] the probability that the sum of squaresfor any nonlinear regression has at least two local minima is positive. The followingexample confirms this statement.Example 2. Consider the data (pi, ti, fi), i = 1, . . . , 9, given by the tableti -4 -3 -2 -1 0 1 2 3 4fi 10 0.1 0.1 0.1 0.1 0.1 0.1 9 9where p1 = . . . = p9 = 1 (see Fig. 2.a)One can easily check that the corresponding functional S has two local minima(See Fig. 2.b):b? = 0.794763, c? = 0.632571, S(b?, c?) = 123.307bˆ = 0.00045, cˆ = −2.50019, S(bˆ, cˆ) = 162.556The graphs of the corresponding regressions are shown in Fig. 2.a.58 R. Scitovski and D. Jukic´Figure 2.a. Exponential regressions Figure 2.b. Two local minimaof the functional SOn the basis of Definition 1. and Remark 2. and Remark 3. it is easy to show thatthe data are preponderantly stationary if and only if they satisfy the condition:m∑i=1pi(ti − tp)(fi − fp) = 0, (7)wheretp := 1κm∑i=1piti, fp := 1κm∑i=1pifi, κ :=m∑i=1pi.In that case the graph of the associated linear regression is parallel with the t-axis.It directs us to the conclusion that in this case the exponential model function coulddegenerate into a linear function t 7→ b, that is, that the local minimum of the OLSproblem (2) could be attained at the point PS(fp, 0), and the local minimum of theTLS problem (3) could be attained at the point PF (fp, 0,0).Using the condition (7) it can easily be shown thatgradS(fp, 0) = 0, and gradF (fp, 0,0) = 0,Analysis of solution of the least squares problem 59which means that PS(fp, 0) is the critical point of the functional S, and PF (fp, 0,0)is the critical point of the functional F . It remains to examine the positive definite-ness of the Hessian HS(fp, 0) of the functional S at the point PS(fp, 0),[ ∑pi fp∑pitifp∑piti fp(2fp∑pit2i −∑pit2i fi) ] ,resp. Hessian HF (fp, 0,0) of the functional F at the point PF (fp, 0,0),∑pi fp∑piti 0 · · · 0fp∑piti fp(2fp∑pit2i −∑pit2i fi)p1fp(fp − f1) · · · pmfp(fp − fm)0 p1fp(fp − f1) p1 · · · 0...............0 pmfp(fp − fm) 0 · · · pm .Let us note that HS(fp, 0) and HF (fp, 0,0) will be indefinite matrices if tp = 0and2fp <∑mi=1 pit2i fi∑mi=1 pit2i. (8)In this case the functional S does not have the local minimum at the point PS(fp, 0),and the functional F does not have the local minimum at the point PF (fp, 0,0).Now we are going to construct the preponderantly stationary data, such that thefunctional S does not have the local minimum at the point PS(fp, 0), and functionalF does not have the local minimum at the point PF (fp, 0,0).Moreover, we will showthat for such defined data functionals S and F attain their global minimum at atleast two points.Example 3. Let T,A > 0 and m = 2k, where 3 ≤ k ∈ N. Choose ε > 0 such that2(k − 1)ε+Ak<ε3[12k(k − 1)]2 + T 2Aε2[12k(k − 1)]2 + T 2 (9)and define the data:(1,−T,A), (1,−(k − 1)ε, ε), (1,−(k − 2)ε, ε), . . . , (1,−2ε, ε), (1,−ε, ε),(1, ε, ε), (1, 2ε, ε), . . . , (1, (k − 2)ε, ε), (1, (k − 1)ε, ε), (1, T, A).Note that these data are preponderantly stationary, tp = 0 and the inequality (9)represents the inequality (8). According to the above mentioned, the functional Sdoes not have the local minimum at the point PS(fp, 0), and the functional F doesnot have the local minimum at the point PF (fp, 0,0).Let (b?, c?) ∈ B be a point such that inf(b,c)∈R2S(b, c) = S(b?, c?) (see Theorem1).Since S(b, 0) ≥ S(fp, 0) for all b ∈ R and the functional S does not have the localminimum at the point PS(fp, 0), then c? 6= 0. Furthermore, since S(b, c) = S(b,−c),we have inf(b,c)∈R2S(b, c) = S(b?, c?) = S(b?,−c?), i.e. the functional S attains itsglobal minimum at at least two points.60 R. Scitovski and D. Jukic´Similarly, because F (b, c, δ) = F (b,−c,−δ) for all (b, c, δ) ∈ R2+m, if (b?, c?, δ?) ∈B×Rm such that inf(b,c,δ)∈R2×RmF (b, c, δ) = F (b?, c?, δ?), then inf(b,c,δ)∈R2×RmF (b, c, δ) =F (b?, c?, δ?) = F (b?,−c?,−δ?), i.e. and the functional F attains its global mini-mum at at least two points.3. Parameter estimationOptimal parameters of the functional S defined by (2) can be estimated by usingclassical methods: Gauss–Newton’s method or Levenberg–Marquardt’s method (see[2], [3], [20]), but that requires a good initial approximation. The problem of choiceof a good initial approximation is considered in [4], [6], [15]. Localization of theregion of a good initial approximation is obtained from the proof of the existencetheorem.Very few papers deal with the problem of the optimal parameter estimation bynonlinear TLS problems (see e.g. [1]). In [9] a special class of nonlinear TLS problemis considered, where the model function is of the form f(x; a, b) = φ−1(ax + b),where a 6= 0 and b are some real parameters, and the function φ : I → R, I ⊆ R iscontinuous and one-to-one. Specially, the exponential model function (1) belongsto this class.Since the symmetry preserves the distances, instead of the TLS problem for themodel function f we can consider the TLS problem for the inverse model functionf−1. In this way for this class of model functions we obtain a linear TLS problem,which can be solved much simpler (see [18], [19]).For this class of model functions in the paper [9] we proved the existence theoremand proposed an efficient algorithm for searching optimal parameters.References[1] P. T. Boggs, R. H. Byrd, R. B. Schnabel, A stable and efficient algorithmfor nonlinear orthogonal distance regression, SIAM J. Sci. Statist. Comput.8(1987), 1052–1078.[2] E Z. Demidenko, Optimization and Regression (in Russian), Nauka, Moscow,1989.[3] J. E. Dennis, Jr, R. B. Schnabel, Numerical Methods for Unconstrained Op-timization and Nonlinear Equations, SIAM, Philadelphia, 1996.[4] D. Jukic´, Parameter Identification Problem in Some Special MathematicalModels, Ph.D. thesis, Department of Mathematics, University of Zagreb, Za-greb, 1996 (in Croatian).[5] D. Jukic´, T. Marosˇevic´, R. Scitovski, Discrete total Lp-norm approxima-tion problem for exponential function, Appl. Math. Comput. 94(1998), 137–143.[6] D. Jukic´, R. Scitovski, Existence of optimal solution for exponential modelby least squares, J. Comput. Appl. Math. 78(1997), 317–328.Analysis of solution of the least squares problem 61[7] D. Jukic´, R. Scitovski, The exponential growth model in: Proceedings of the6th International Conference on Operational Research ( T.Hunjak, Lj.Martic´and L.Neralic´, Eds.), Zagreb, 1996, 107–112.[8] D. Jukic´, R. Scitovski, The best least squares approximation problem for a3-parametric exponential regression model, Manuscript[9] D. Jukic´, R. Scitovski, H. Spa¨th, Partial linearization of one class of thenonlinear total least squares problem by using the inverse model function, Com-puting, in press.[10] D. Jukic´, R. Scitovski, Existence results for special nonlinear total leastsquares problem, J. Math. Anal. Appl. 226(1998) 348–363.[11] D. S. Mitrinovic´, J. Pecˇaric´, A. M. Fink, Classical and New Inequalitiesin Analysis, Cluwer, Dordrecht, 1993.[12] P. E. Rudolph, G. Herrendo¨rfer, Optimal experimental design and accu-racy of parameter estimation for nonlinear regression models used in long-termselection, Biom. J. 37(1995), 183[13] R. Scitovski, S. Kosanovic´, Rate of change in economics research, Eco-nomics analysis and workers management 19(1985), 65–75.[14] R. Scitovski, Condition of preponderant increase and Tchebycheffs inequality,in: Proceedings of VI. Conference on Applied Mathematics, (B. S. Jovanovic´,Ed.), Belgrade, 1988. 189–194.[15] R. Scitovski, Some special nonlinear least squares problems, Radovi matema-ticˇki 4(1988), 279–298.[16] R. Scitovski, D. Jukic´, Total least squares problem for exponential function,Inverse Problems 12(1996), 341–349.[17] R. Scitovski, Analysis of parameter identification problem, Appl. Math. Com-put. 82(1997), 39–55.[18] H. Spa¨th, On discrete linear orthogonal Lp-approximation, ZAMM 62(1982),T354–T355.[19] G. A. Watson, Numerical methods for Linear Orthogonal Lp approximation,IMA J.Numer. Anal. 2(1982), 275–287.[20] S. Wolfram, The Mathematica Book, WolframMedia, Inc., Champaign, 1998.

Analysis of solution of the least squares problem

Abstract

Similar works

Full text

Available Versions

Hrčak - Portal of scientific journals of Croatia