In this paper a new line search for a Newton Rhapson learning control algorithm is presented. Theorems and rigorous proofs of its increased robustness over existing line searches are provided, and numerical examples are used to further validate the theorems. Also, the previously posed open question of whether robust optimal trajectory learning is possible is also addressed. It is shown that the answer is generally no, at least for gradient-based learning control algorithms

Driessen, B. J.

Kwok, K. S.

Sadegh, N.

UNT Digital Library

A Robust Line Search for Learning Control. CWF- 7 531x8- - RECEIVE MAR 2 0 1998 Brian J. Driessen Sandia National Laboratories, Albuquerque, NM 87 185-0439, bjdries @ sandia.gov Q S T I  Nader Sadegh Georgia Institute of Technology, Atlanta, GA 30332, nader.sadegh@me.gatech.edu Kwan S. Kwok Sandia National Laboratories, Albuquerque, NM 87 185- 1003, kskwok@ sandia.gov Abstract. In this paper a new line search for a Newton Rhapson learning control algorithm is presented. Theorems and rigorous proofs of its increased robustness over existing line searches are provided, and numerical examples are used to further validate the theorems. Also, the previously posed open question of whether robust optimal trajectory learning is possible is also addressed. It is shown that the answer is generally no, at least for gradient-based learning control algorithms. 1. Introduction Learning control is a method of control that feeds the system inputs for a specific task repetitively and uses the actual on-line measured response of the system to evaluate the quality or goodness of the input. The actual responses are used in a feedback loop in which the inputs are adjusted to reduce measured errors in the output. Example applications include robotics and manufacturing where a certain output tracking task is to be performed repeatedly. Usually the output is the position or velocity history of the robot's joints although sometimes it also includes measured forces at the end effector (see Cheah and Wang [3]). Learning control has a history dating back to 1984 (see Arimoto et a1 [I]) when it was first applied to robot motion control. Horowitz [7] gives a nice history of the development and usage of learning controllers for (rigid) robot manipulators. He compares and contrasts different learning algorithms and also provides an experimental demonstration of a robot that learns to make its end effector track a circular trajectory. He insightfully points out that an open area of research is in finding methods for robust optimal (e.g., minimum energy, minimum vibration, or minimum time) trajectory learning, as opposed to only finding a control history that meets output requirements. In Section 5 of the present paper, the question of whether this is possible will be addressed. Examples of work that have empirically investigated approaches to this problem include Gorinevsky ([4], [5], and [6]), who considered the use of the Levenberg- Marquardt optimization method for least squares, and Sadegh and Driessen [8] who considered the use of gradient-based algorithms for constrained optimization. Herein we will present a new line search for a Newton Rhaphson learning control algorithm in Section 2, discuss applications in Section 3, demonstrate its advantages over existing line searches on a numerical example in Section 4, discuss whether robust optimal trajectory learning is possible in Section 5, and finally end with conclusions in Section 6. 2. On the Robustness Against Erroneous Constraint Jacobean of a New Line Search for a Newton Rhaphson Learning Control Algorithm This section will prove that the modified- line-search Newton Rhaphson algorithm, to be presented shortly, is robust to erroneous constraint gradients. We will show that, under a very weak requirement on the accuracy of the constraint Jacobean, the Newton Rhaphson algorithm, with the newly proposed line search, is guaranteed to create a sequence of iterates whose constraint error goes to zero. Then we will show that the standard existing line search fails under the same requirements. * Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under Contract DE-AC04-94AL85000. DISCLAIMER This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or use- fulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any spe- cific commercial product, process, or service by trade name, trademark, manufac- turer, or otherwise docs not necessarily constitute or imply its endorsement, recom- mendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors exptessed herein do not necessarily state or refiect those of the United Statu Government or any agency thereof. ._ DISCLAIMER Portions of this document may be illegible in electronic image products. Images are produced from the best available original document. Theorem 2.1: Consider the problem to find a solution X, X E R", that satisfies equation (2.1) below wherefmaps from R" to R". Let J(x) denote the n by n Jacobean offat x: f (x) = 0 (2.1) (2.2) f 3 f  (x) J (  x) = -f3X  Suppose, that for any x, the value of Ax) is available, but we have only an inaccurate estimate j(x) of the true Jacobean J(x) . Suppose that the merit function for the algorithm is z(x) given by equation (2.3) below 1 z (x)  = yf'(x)f(x> (2.3) which is half the sum of the squares of the constraint violations. Let z, (x) = - = J'(x)f(x). Suppose that for all iterates x, and x,+~ of the algorithm, the Hessian of z is norm- bounded so that there exists a G>O such that f3z(x) f3X z(x,+, 1 - z(x, 1 I f'& )J(x, )(X,+I -x, )+ GIIXk+I - Xk [ I 2  Xt 7 xk+l  (2.4) Suppose that for all iterates X, , there exist a y > 0 for which min(eigenvulue$jr3>) 2 y >  o and that for all iterates xt there exists a 6> 0 such that (2.5) min(eigenvulues(J.Y' / 2 + Suppose that for every iterate x,, the j - ' JT /2 ) )26>O (2.6) search direction p ,  is calculated by: Pk = -j-' (x, and that x,+' is chosen by: xk+l = (+r Pk + x k  (2.8) where m is the smallest nonnegative integer such that Then, the sequence z ( x , )  + 0 as k + 00, implying f( x, ) + 0. Proof of Theorem 2.1: We will prove that if z(x,) is lower bounded away from zero, 2 i.e., z(x,)2 q > O  Vk, then ~z(~,+~)-z(x~)~ islower bounded away from zero. So, suppose z ( x , )  2 q > 0 Vk . From (2.7) and (2.8), we have (2.10) Substituting (2.10) into (2.4) gives Vxk 7 x k + 1  (2.1 1) If we choose m so that the right hand side of (2.11) is I the right hand side of (2.9), then clearly this is sufficient for such an m to satisfy the line search (2.9). So, let us write out this sufficient condition. (Note: for conciseness we will omit the arguments x, (e.g., f ( x , )  will be written f and J(x,) will be written J ) ) .  After a tiny amount of algebra, we have where Z is given in (2.13) below and fix(*) is the smallest nonnegative integer2 * . m 2 f i x ( E ) + l  (2.12) (2.13) By hypothesis (2.5), the largest possible eigenvalue of Gj-'? + I  is G /  y+l  so that f'(G.?-'? + Z ) f I ( G /  y+l)f'f; and, by hypothesis (2.6), f'J.?-'f I W'f. Therefore, m I mmax where mmax is given below mmax =ln( (G/y+l ) /F) / ln(2)  (2.14) If we substitute the associated value of m, ( m  = m' = fix(m,,)+l), into (2.9) and the current assumption that z ( x , ) 2  q > O  V k ,  we - obtain (2.15) 2m' 2m' -(2)(i) z ( x , ) I - m ( i )  q Thus, z(x, ) 2 q > 0 implies 2 m* z(x,+, >-z(x, 15 -(2)( 3) q (2.16) Now, since the sequence z(x,) is monotonically decreasing and z(x,  ) 2 0, the sequence z(x,) must converge. We now prove that there cannot exist a p > O  such that z(x, ) 2 pVk . If there did exist such a p , then z(x,+,)-z(x,)<-(2) - p for all k, implying that z(x,) decreases indefinitely, which is impossible. Therefore, no such p exists and z(x, ) -+ 0 as k + 00. This completes the proof of Theorem 2.1. QED. Theorem 2.2: Consider the hypotheses of Theorem 2.1, but with the following standard Armijo line search (see Bazarra [2], page 307) replacing the line search (2.9): (;Im* (2.17) with a E ( 0 , l ) .  Then, the Newton Rhaphson algorithm can fail to cause z(x,) to converge to zero. Proof of Theorem 2.2: Again, choosing rn so that the right hand side of (2.11) is 2 the right hand side of (2.17) would be sufficient to guarantee that line search (2.17) is satisfied. Writing this sufficient condition gives, after a tiny amount of algebra 2"( f ' J j - l f  - @f) 2 Gf '.?-'j-' f (2.18) We see that if a is chosen too large (i.e., a2 6) ,  then the left hand side of (2.18) could be negative. Then, even if G=O, there would not exist a nonnegative integer m satisfying (2.18). Thus, the line search could fail. This completes the proof of Theorem 2.2. QED. The advantages of line search (2.9) over the standard Armijo line search (2.17) are twofold. First, since 6 is not known, one could easily choose a 2 6 by accident, thus causing the standard Armijo line search to fail while line search (2.9) would not fail. Second, even if a conservative value of 6 were known, this value is surely going to be over conservative, leading to the use of a much smaller value of a than would actually be required for convergence to occur. Such an over conservative Armijo parameter a could lead to unnecessarily slow convergence of the algorithm. Remark 2.2: The reader may be wondering why we do not just calculate J(x)  by numerically differentiating Ax). The reason is as follows. Let N be the number of time steps in a Remark 2.1: digitally controlled system; then, the number of variables (length of the vector x) is proportional to N .  Such numerical differentiation would take Order(N*) time. The number of variables is on the order of 100's to 1OOOO's. Thus, the numerical differentiation would not be practical. Banded matrix methods that use the model of the system, on the other hand, allow the user to calculate an approximate Newton Rhaphson search direction p ,  in Order(N) time. Thus, while the model- based calculation of the search direction is practical, the numerical differentiation approach is not. 3. Overview of Learning Control Learning control is a method of output tracking that does not make any assumptions about whether the system is minimum-phase or non-minimum-phase nor any assumptions about knowledge of the system's order (number of state variables). The method uses the actual measured system's response to an input in a feedback loop, in order to obtain tracking robustness in the presence of large model mismatch. Example applications include robotics and manufacturing where a certain output tracking task is to be performed repeatedly. An initial guess of the inputs to the system can be obtained off-line by using an approximate model of the system. Then, in the learning feedback loop, the actual output response errors are measured. The gradients of these output errors with respect to the inputs can be obtained from strictly the approximate model or can be improved in accuracy by calculating gradients based on the model but evaluated along the actual measured trajectory of the system (see Sadegh and Driessen [8]). The learning feedback loop is continued at least until the actual system's output error is zero. However, the learning feedback loop is often continued after zero output error is achieved, in order to obtain output tracking robustness against slowly varying dynamics of the actual system. For example, the dynamic properties of the joints of a robot may be changing over time, and the learning feedback loop will maintain zero output error in spite of these time-varying system properties. Mathematically, the learning control problem considered herein can be viewed as the Newton Rhaphson problem illustrated in (3.1) below 3 U +(System) 4 P (3.1) where 0 denotes a vector of input values that define the input over the time interval of interest and P denotes an output vector which we desire to be zero. For example, 0 may be parameters that define a spline of the input history over the interval. The inaccuracy of the Jacobean J = d%u is a result of the fact that we never have a perfect model of the dynamic system. 4. Numerical Example In this section we demonstrate the robustness advantage of the proposed line search over an existing Armijo line search. The proposed line search yields convergence of the output error to zero while the existing Armijo line search fails to do so. This numerically supports Theorems 2.1 and 2.2. The dynamic system considered is a two- massltwo-spring system. The springs are nonlinear. The spring forces are given by where 6 is the deflection of the spring, F, the spring force, and k and a constants. If we let q1 and q2 denote the positions of the masses, measured from equilibrium positions, and relative to an inertial reference frame, and let each mass value be m and let both springs be equivalent and let there be control forces u1 and u2 on the first two masses, then the equations of motion are F, = -k6-  a63 (4.1) m i ,  =~(q , -q , )+a(q2-q l )3 -kq , -aq13+ul  mi2 =-k(q2 -41) -4q2  -4d3+u2  (4.3) (4.2) The inputs u l ( t )  and u2(t) will be approximated by 2-node linear splines. Thus, 0 in (3.1) is a 4-vector, Le., U'=(u,(O), ul(tr), u,(O),  u 2 ( t f ) ) .  The output is the difference between the state ~ ~ ( q l , q 2 , q 3 , q , , q 2 , q 3 ) '  at the final time t = tr and the desired value of Z(tr ) , denoted by (q1d,q2d,q3d,0,0,0)', where qld=O.l and qZd=0.1. The true values of m, k, and a are 1.0, 0.1, and 0.001. The model values are 1.5, 1.42, and 0.001, (a percentage error of 50% in m and 1300% in k). A final time of tf=2.0 seconds and 100 Euler integration steps were used. The initial guess was 0'=(1.3179, -1.6680, -0.5748, 0.79 17). Using the proposed line search, the solutions for u1 (t) and u2 (t) were successfully found. On the other hand, the existing Armijo line search locked up andfailed to converge to the solution. The reason was precisely the one stated in the proof of Theorem 2.2. Namely, f'J.?-'f - af'f (see equation (2.18)) was less than zero so that no integer m existed that would satisfy the standard Armijo line search criterion (2.17). Yet, all the requirements for convergence with the newly proposed line search were satisfied so that it yielded successful convergence. 5. Can Robust Optimal Trajectory Learning Be Accomplished? We note that the title of this section is essentially the research question posed by Horowitz [7]. The discussion of this section will point out that in general the answer to the posed question is no, at least for gradient-based learning schemes. We will see that for such schemes, convergence to a truly optimal trajectory cannot be guaranteed. The reason is very simple. If we have an approximate (erroneous) gradient, denoted in this section by Vz, which is "close" to the true gradient, denoted in this section by Vz, then the search direction is null if Vz = 0 (even if Vz f 0). Thus, while convergence of Vz to zero may be provable under mild assumptions, the convergence of the true gradient Vz to zero clearly cannot be guaranteed. In summary, unlike robust learning control for output tracking, gradient-based optimal trajectory learning schemes cannot be guaranteed to converge to a stationary point. It is worthwhile to point out that similar problems hold for the case when the cost function z is a sum of the squares of measurable outputs for the case of more outputs than inputs. Namely, the same problem of obtaining a null search direction at point(s) where the true gradient is not zero occurs in such over-determined least squares problems where the erroneous gradient is given by .?'f wherefis the (measured) error vector and j its approximate (erroneous) Jacobean with respect to the input vector. Of course the true gradient is J'f where J is the exact Jacobean. With such least squares problems J T  has fewer rows than columns and thus has a null space of non-zero dimension, and a tiny difference between J^ and A A h 4 J can renderfin the null space of j even though Sadegh, N. and Driessen B., "Minimum Time f is not in the null space of J ,  thus, again, Trajectory Learning," American Control rendering Vz zero even though Vz f 0. 8. A Conference, Seattle, Washington, June, 1995. 6. Conclusion This work presented a new line search for a Newton Rhaphson based robust learning control method, where output measurements are available but the system model is known only approximately. The mild conditions that guarantee the convergence of the proposed method were given and the associated convergence proof provided. Existing Armijo line searches were proven to be susceptible to locking up under these mild conditions, which reveals that the newly proposed line search is more robust. A numerical example was presented that demonstrated these facts. References 1. 2. 3. 4. 5.  6. 7. Arimoto, S. et al, "Bettering Operation of Robots by Learning," Journal of Robotic Systems, Vol. 1, No. 2, 1984, pp. 123-140. Bazarra, M. et al, Nonlinear Programming, Theory and Applications, Second Edition, New York: John Wiley and Sons, Inc., 1979. Cheah, C. and Wang, D., "Learning Control for a Class of Nonlinear Differential- Algebraic Systems With Application to Constrained Robots," Journal of Robotic Systems, Vol. 13, No. 3, 1996, pp. 141-151. Gorinevsky, D. et al, "Learning Approximation of Feedforward Dependence on the Task Parameters: Experiments in Direct-Drive Manipulator Tracking," American Control Conference, Seattle Washington, 1995, pp. 883-887. Gorinevsky, D., "An Algorithm for On-Line Parametric Nonlinear Least Square Optimization," 33rd ZEEE CDC, Lake Buena Vista, Florida, December, 1994. Gorinevsky, D., "An Application of On-Line Parametric Optimization to Task-Level Learning Control," American Control Conference, Seattle, Washington, June 1995, Horowitz, Roberto, "Learning Control of Robot Manipulators," ASME Journal of Dynamic Systems, Measurement, and Control, Vol. 115, June 1993, pp. 402-41 1. pp. 862-866. 5 

A robust line search for learning control

https://digital.library.unt.edu/ark:/67531/metadc708643/m2/1/high_res_d/677192.pdf

A robust line search for learning control

Abstract

Similar works

Full text

Available Versions

UNT Digital Library