Discussed here are recent theorems proving that artificial neural networks are capable of approximating an arbitrary mapping and its derivatives as accurately as desired. This fact forms the basis for further results establishing the learnability of the desired approximations, using results from non-parametric statistics. These results have potential applications in robotics, chaotic dynamics, control, and sensitivity analysis. An example involving learning the transfer function and its derivatives for a chaotic map is discussed

Gallant, A. Ronald

Hornik, Kurt

Stinchcombe, Maxwell

White, Halbert

English

NASA Technical Reports Server

 91-21+80
NEURAL NETWORK REPRESENTATION
AND LEARNING OF MAPPINGS
AND THEIR DERIVATIVES
April 1990
Halbert White, Ph.D.
University of California, San Diego
Collaborative research with:
K. Hornik, M. Stinchcombe and A.R. Gallant
59
https://ntrs.nasa.gov/search.jsp?R=19910012467 2020-03-19T19:00:47+00:00Z
ABSTRACT
We discuss recent theorems proving that artificial neural networks are
capable of approximating an arbitrary mapping and its derivatives as
accurately as desired. This fact forms the basis for further results
establishing the leamability of the desired approximations, using results
from non-parametric statistics. These results have potential applications in
robotics, chaotic dynamics, control, and sensitivity analysis (physics,
chemistry, and engineering). We discuss an example involving learning the
transfer function and its derivatives for a chaotic map.
60
Jordan (1989), "Generic Constraints on Underspecified Target Trajectories,"
Proceedings IJCNN, Washington D.C.:
The Jacobian matrix OzlOx... is the matrix that relates small changes in the
controller output to small changes in the task space results and cannot be
assumed to be available a priori, or provided by the environment. However,
all of the derivatives in the matrix are forward derivates. They are easily
obtained by differentiation if a forward model is available. The forward
model itself must be learned, but this can be achieved directly by system
identification. Once the model is accurate over a particular domain, its
derivatives provide a leaming operator that allows the system to convert
errors in task space into errors in articulartory space and thereby change the
controller.
61
UNIVERSAL APPROXIMATION OF AN UNKNOWN
MAPPING AND ITS DERIVATIVES USING
MULTILAYER FEEDFORWARD NETWORKS *
by
Kurt Homik, Maxwell Stinchcombe
and
Halbert White
January 1990
We are indebted to Angelo Melino for pressing us on the issue addressed here and to
the referees for numerous helpful suggestions. White's participation was supported by
NSF Grant SES-8806990.
62
ABSTRACT
We give conditions ensuring that multilayer feedforward networks with as few as a
single hidden layer and an appropriately smooth hidden layer activation function are
capable of arbitrarily accurate approximation to an arbitrary function and its derivatives.
In fact, these networks can approximate functions that are not differentiable in the
classical sense, but possess only a generalized derivative, as is the case for certain
piecewise di_rendable functions. The conditions imposed on the hidden layer
activation function are relatively mild; the conditions imposed on the domain of the
function to be approximated have practical implications. Our approximation results
provide a previously missing theoretical justification for the use of multilayer
feedforward networks in applications requiring simultaneous approximation of a function
and its derivatives.
63
Relevant Application Areas:
1. Robotics
2. Chaotic Dynamics
3. Control
4. Sensitivity Analysis (Physics, Chemistry, Engineering)
64
Intuition suggests that networks having smooth hidden layer activation functions
ought to have output function derivatives that will approximate the derivatives of an
unknown mapping. However, the justification for this intuition is not obvious. Consider
the class of single hidden layer feedforward networks having network output functions
belonging to the set
,__,(G)- {_ " _r ....) _ I g(x)_. (1__jG(._T)j);
j=l
x _ IRr, fl j _ IR, )j _ IRr + l , j = l,..., q, q _ IN},
where x represents an r vector of network inputs (r _ /N--- {1,2, ...}), _" -- (1,xT) 7
(the superscript T denotes transposition), flj represents hidden to output layer weights
and yj represents input to hidden layer weights, j = 1,..., q, where q is the number of
hidden units, and G is a given hidden unit activation function. The first partial
derivatives of the network output function are given by
Og(x)/ =
j=l
i = 1,..., r,
where xi is the ith component ofx,)ji is the ith component of)j, i = 1,..., r (?'j0 is the
input layer bias to hidden unit j ), and DG denotes the first derivative of G.
65
output
hidden
x1 x2 x3 x4
Figure 2
Single Hidden Layer Feedforward Network
input
66
Outline:
1. Mathematical Background
2. Approximation Results
3. Learning Results
4. Example: Learning Chaotic Map
67
I. MATHEMATICAL BACKGROUND
Let U be an open subsetof JStr,and letC (U) be the setof allfunctionscontinuous
on U. Let a be an r-tuple a = (a 1,... ,O_r)T of non-negative integers (a "multi-index").
Ifx belongs to _r, let x a -- x_ _ a, D a• ... • x,.. Denote by the partial derivative
olal/Oxa_-olal/(OxC_ t 0_2_a2...,,,._r: a, )
of order lal--t_l +t_2+...+o_ r. For non-negative integers m. we define
c'n(u) = {f _ C (U): D a f _ C(U) for all a, lal m} and C**(U) = tnmml Cm(U).
We let D O be the identity, so that C°(U) = C(U). Thus, the functions in Cm(U) have
continuous derivatives up to order m on U, while the functions in C**(U) have
continuous derivatives on U of every order. We shall be interested in approximating
elements of cm(u) using feedforward networks. When U g _r, the fact that network
output functions (elements of Y.(G)) will belong to cm( JR r ) necessitates considering
• their restriction to U, written g Iu for g in Z(G). Recall that g IU(x) = g (x)for x in U
and is not defined forx not in U, thus g IU _ cm(u), as desired.)
68
DEFINITION 2.1: Let U be a subset of _r, letS be a coUextion of functions f.
U ---) /R and let p be a metric on S. For any g in Z(G) (recall g : _r ....) _ ) define
the restriction of g to U, g Iu as g Iu(x) = g (x) for x in U, g l u (x) unspecified for x
not in U.
Suppose that for any f in S and e > 0 there exists g in E(G) such that
P(f, g Itl) <e. Then we say that E(G) contains a subset p-dense in S. If in addition
g ItJ belongs to S for every g in Z(G), we say that E(G) is p -dense in S. []
DEFINITION 2.2: Let m, I _ {0] u/N, 0 < m </, and U c /R r be given, and let
S c CI(u). Suppose that for any f in S, compact K c U and e > 0 there exists g tn
g(G) such that maxlal <m supz_ g I Daf(x)-Dag(x) I <e. Then we say that
g(G) is m-uniformly dense on compacta in S. []
When Y,(G) is m-uniformly dense on compacta in S, then no matter how we choose
an f in S, a compact subset K of U, or the accuracy of approximation e > 0, we can
always find a single hidden layer feedforward network having output function g (in
E(G)) with all derivatives of g Itl on K up to order m lying within e of those off on K.
This is a strong and very desirable approximation property.
69
The space Lp(U, tz) is the collection of all measurable functions f such that
[Ifllp,u,g = [fu If IPd#] lip <0% 1 <p <oo, where the integral is defined in the
sense of Lebesgue. When/.t =A, we may write either _ofd_, or fvf(x)dxto denote
the same integral. We measure the distance between two functions f and g belonging to
Lp(U,#) in terms of the metric Pp, v,a(f, g) mlJf- g Ib, Two functions that differ
only on sets of/z-measure zero have pp, U._ (f, g) = O. We shall not distinguish between
such functions.
The first Sobolev space we consider is denoted S_(U,I.t), defined as the collection
of all functions fin cm(u) such thatUDaf[]p,U.lz < oo for all I o_ I <m. We define
the Sobolev norm [[fl[m,p, u._ - (_l a I< m IID" fll_. u._) up- TheSobolevmetric is
mPp,_(f,g)-llf -gllm.p.v._ f,g _ S_(U,_t),
Note that mp p,_ depends implicitly on U, but we suppress this dependence for notational
convenience. The Sobolev metric explicitly takes into account distances between
derivatives. Two functions in S_(U, tz ) are close in the Sobolev metric ppmlz when all
derivatives of order 0 < I c_ i < m are close in Lp metric.
70
We also consider the Sobolev spaces
W_(U)_ {f _ Ll, loc(U) l Oaf _ Lp(U,_),O< la l <m }.
This is the collection of all functions having generalized derivatives belonging to
Lp(U,/q,) of order up to m. Consequently, W'_(U) includes S"fl(U,2), as well as
functions that do not have derivatives in the classical sense, such as piecewise
differentiable functions.
The norm on W_(U) generalizes that on S_(U,_,); we write it as
Ilfllm,p,u-( E [I/_fllppu.x) 1/p f_W'_(U).
lal<m
For the metric on Wn_(U) we suppress the dependence on U and write
P'_(f,g)-llf-gllm,p,U f,g_ W'_(U).
Two functions are close in the Sobolev space Wn_(U) if all generalized derivatives are
close in Lp(U,X) distance.
71
Our results make fundamental use of one last function space, the space C_ ( IR r)
of rapidly decreasing functions in CO°(]Rr). C_( ]R r) is defined as the set of all
functions in COo(_r) such that for all multi-indices a andS, x#Daf(x)-->O as
IX I"->*% where x_X_llX_22..2 " and Ix I=maxl<i<r Ixi I. Note that
cb*( IRr) c C_ (IRr).
Desired results:
1.) Z(G) is m-uniformly dense on compacta in C_,( _r ), Sty(U, _,)
2.) Z(G)isp_,u-denseinS_(#_r, #)
3.) Z(G) isp_-dense in W_(U)
72
2. APPROXIMATION RESULTS
THEOREM 3.1: Let G ;e 0 belong to S_(_,_) for some integer m >_0. Then Z(G)
i
is m-uniformly dense on compacta in CT (/R r ). []
DEFINITION 3.2: Let l e {0} u/N be given. G is l-finite if G ecl(IR) and.
0<_1 DiG I d_, < oo. []
LEMMA 3.3: If G is l-finite then for all 0 < m < l there exists H e s_n(_,_,), H ;e 0,
suchthatZ(H) c Z(G). []
l-finite activation functions G with I DIG dg ;e 0 have f IDmGI dZ =.o for an m < l,
and for m > I all l-finite activation functions G have IDmG dA = 0 (provided Drag
exists).
It is informative to examine cases not satisfying the conditions of the theorems. For
example, if G = sin then G e C**(//?), but for all l, I I DIG I d_l. = **. If {7 is a
polynomial of degree m then again G e C**(/R), but for l<m we have
I IDIG I dA. =-0, although f I DIG I d_, = 0 for l > m. Consequendy, neither
trigonometric functions nor polynomials are 1-finite.
73
COROLLARY 3.4: If G is l-finite, then for all 0 < m < I, 7_,(G) is m-uniformly dense
on compacta in C_ ( _r ). []
COROLLARY 3.5: If G is l-finite, 0 <_m < l, and U is an open subset of
,Y_,(G) is m-unifonnly dense on compacta in S_(U,P_ ) for l <-p <.o. []
_r then
COROLLARY 3.6: If G is l-firfite and/_ is compactly suplxnXed, then for all 0 -< m < 1
E(G) c s_n(/R r, _) and Z(G) isp_,_-dense in S_n ( /R r, /z ).
COROLLARY 3.8: If G is/-finite, 0 < m </, U is an open bounded subset of _r and
C_ (/R') is p_-_nse in W_(U) then E(G) is also p_-dense in W_(U).
These results rigorously establish that suffaciently complex multilayer feedforward
networks with as few as a single hidden layer are capable of arbitrarily aceurate
approximation to an unknown mapping and its (generalized) derivatives in a variety of
precise senses. The conditions imposed on G are relatively mild; the conditions required
of U have practical implications.
74
g131 132
XI 711
Figure 1.
0 input unit
GO activation unit
"_21 _1_2"_22 x2
Feedforward Network
(_ multiplication unit
(_ additionunit
Note: biases not shown
7S
gl
g
g2
DG GO
I
DG
Figure 2. Derivative Network
(_ input unit (_ multiplication unit
GO activation unit 0 addition unit
DGC) activation derivative unit
Note: biases not shown
76
On Learning the Derivatives of an Unknown Mapping
with Multilayer Feedforward Networks
by
A. Ronald Gallant
Department of Statistics
North Carolina State University
Raleigh, NC 27696-8203 USA
Halbert White
Department of Economics, D-008
University of California, San Diego
La Jolla, CA 92093
October 1989
77
ABSTRACT
Recently, multiple input, single output, single hidden layer, feedforward
neural networks have been shown to be capable of approximating a nonlinear map
and its partial derivatives. Specifically, neural nets have been shown to be dense
in various Sobolev spaces (Hornik, Stinchcombe and White, 1989). Building
upon this result, we show that a net can be trained so that the map and its
derivatives are learned. Specifically, we use a result of Gallant (1987b) to show
that least squares and similar estimates are strongly consistent in Sobolev norm
provided the number of hidden units and the size of the training set increase
together. We illustrate these results by an application to the inverse problem of
chaotic dynamics: recovery of a nonlinear map from a time series of iterates.
These results extend automatically to nets that embed the single hidden layer,
feedforward network as a special case.
78
3. LEARNING RESULTS
SETUP. We consider a single hidden layer feedforward network having network
output function
K
gr(x,o) - y__jC(xTTj)
j=l
where x represents an r x 1 vector of network inputs (including a "bias unit"),/_j
represents hidden to output layer weights, yj represents input to hidden layer
weights, K is the number of hidden units,
o'= (ill, r_,/h, r2,... ,fix,rk),
and G is the hidden unit activation function.
We assume that the network is trained using data {Yt, Xt} generated
according to
Yt = g *(xt) + et t = 1, 2, ..., n.
xt denotes the observed input and et denotes random noise. The number Kn of
hidden units employed depends on the size n of the training set. The network is
trained by finding gx, (x, 0 ) that minimizes
K.
s,,(O) = 1 _, [yt _ E PjG(xTyj)] 2 ,
n t=l j=l
subject to the restriction that gtc,(x, 0 ) is a member of the estimation space q.
79
REGULARITY CONDITIONS:
Input space. The input space X is the closure of a bounded, open subset of _r.
Parameter space. For some integer m, 0 < m < _, some integer p, 1 < p < 0_, and
some bound B, 0 < B < _, g is a point in the Sobolev space Wm+[r/pl+l,p, x and
Ilg*llm+[r/pJ+l,p,x < B.
Activation function. The activation
_** (dra/dum)G(u) du < 0-. See Section
.--oo
(1989).
function G belongs to cm(_) and
3 of Homik, Stinchcombe and White
Estimation space, gx,(x, O) is restricted
the optimization of sn(g).
to _--- {g: I[g llm+[r/p]+l,p, x < B} ill
Training set. The empirical distribution of {xt}_=1converges to a distribution
_t(x) and/g(O) >Ofor every open subset Oof._
Error process. The errors {et} are independently and identically distributed with
common probability law P having _eP(de):O and 0<_ e2p(de)<**.
(_e2p(de) = 0 implies = 0 for all t.)et
8o
Independence. The probability law P of the errors does not depend on {xt}7*=l;
that is, P(A) can be evaluated without knowledge of {xt)_=l,
limn_._(1/n)Y_,t= 1 xt, etc.
81
THEOREM 1. Under the Regularity Conditions
tim IIg*- g_c.(-,o)llm.**,x= o
n_
almost surely
provided lirnn _ Kn = .o almost surely. In particular,
lirn atgK.(x, 0 )1=a(g* )
n_
almost surely
provided tr is continuous with respect to I1"lira,**,x" []
82
4. EXAMPLE: LEARNING CHAOTIC MAP
Our investigation studies the ability of the single hidden layer network
K
gK(Xt-5 ,...,Xt-1) = _.__jG(,75jxt-5 +"" + _'ljxt-I + 70j)
j=1
with logistic squasher
G(u) = 1/[1 + exp(-u)]
to approximate the derivatives of a,discretized variant of the Mackey-Glass
equation (Schuster, 1988, p. 120)
[ (0"2)Xt---.-.-._5 _ (0.1)Xt_l] "g(Xt-5, Xt-1) = Xt-1 + (10.5) 1 + ( t_5) 10
The values of the weights _j and _'ij that minimize
n1 _., [xt -gK(Xt-5,..., Xt-1)] 2
s,,(gx) = n t--1
were determined using the Gauss-Newton nonlinear least squares algorithm. Our
rule relating K to n was of the form K *_.log(n) because asymptotic theory in a
related context (Gallant, 1989) suggests that this is likely to be the relationship
that will give stable estimates.
83
(I)
-4---
em
4--
-I---
(I)
E
m
0
L
C
-(3
E
0
CL
L II
0 c
"-- II
C_
0
E
-0
0
O.
E
91
(I)
O_
q)
O)
o_
LL.
!
I
I
I
I
I
!
I
I
I'"' ' • • I .... ! .... I .... I , • ' • ! " • L, IS,, I .... i • ' • ; !
,_,.o ill o It1 0 _ 0 _ 0
X • • •
i i i i
o
o
,el
11")
0
o
o
o
I
o
I
I
o
o.i
I
?
X
0
o"
I
X
II
X
¢-
"0
01
0
"I:;
0
E
I,!
0
Z
CC_
._, c:
-C_v
-I
IS)
C'q
::3
C71
I,
l.l.l.l-l.l.l.l.l.l.l.l.l.l.j_.._-,_, r
_o _ o _0 _ 0 _ 0 _o _ o _ o _ 0
x
I I I I I I l I I I
I
x
o
o
@
d
x
II
o ?
o _
,_ Z
I
o
I
8S
ZJ
oi
t.u.
¢o
,el
° ,_,
c_
,:5
c_
II
o
o Ld
o
IEI
1
0
¢'U
!
86
(3
(D
--I.-.
im
oo
(].)
(D
E
m
O
L.
E
-(3
E
U
0
>o
U II
> c
g_
II
-(3
-0
0
O_
E
L.
O_
L
-s
_u
L_
I " I ' i ' i " | ' ' i " ! ' I " I '" 'i ' i ' I ' I ' I ' i " i'_'_T
.'_O II'] o li'l <3 113 o li3 o li'] O ii3 o li3 o _ O
X
_.j00 Cq OJ -_ -'_ 0 0 0 "_ " OJ ed Pl P] _" _"
I I I I I I I I I I
in
I
x
0
El
o
o X
El
o
I
o
_e4
I
tO
,IM
I
o
OJ
I
0
(3
I
x
v
II
x
C
_m
J
"13
Q}
r"
0
-0
(/I
0
E
g_
=4-
l.d
.=I-=
0
Z
87
-I--
Om
M.-.
c-
CJ
C
7_
C
0
(7
Oo
m
c
• .
C_-
c- II
O_
C
-0
0
(/)
0
CL
E
t_
q_
(7
L.
o_
I,
l .... I .... I .... 0 .... I .... I .... 0 .... I .... i
x
i i i n
o
o
o
o
o
o
I
o
I
I
o
i
I
X
0
I
X
II
X
C
r"
0
"0
_m
G}
-4-=
0
E
Wm
W
0
Z
88
oL"LI
'-4'---
--!
"O
<.-
_o
<D °
,> o
"O !1
--O v
o
C)_
or)
C7)
I,
I • i ' | • i • i • i • i • i • i • | ' I • i • i • i ' i • t • i
X
f I I I I t I i I I
I
x
g'l
0
tn
o
o ?
o X
tl'l
0
I
0
I
In
I
o
rd
I
0
!
x
li
x
6
c-
gm
m
ID
I-
lel
0
o_
tll
-I,--
E
o_
ffl
W
0
Z
89

Impact of Application of Fuzzy Theory to Industry
(Paper not provided by publication date.)
pRBCEDtNG PAGE BLgflcIKNOT FILMED

Time-sweeping Mode Fuzzy Computer-o Forward and Backward
Fuzzy Inference Engine
(Paper not provided by publication date.)
93
PRE:_CF_..D!NGPAGE BLAr.IX NOT FILMED



Neural network representation and learning of mappings and their derivatives

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server