To take advantage of prior knowledge (hints) about the function one wants to learn, we introduce a method that generalizes learning from examples to learning from hints. A canonical representation of hints is defined and illustrated. All hints are represented to the learning process by examples, and examples of the function are treated on equal footing with the rest of the hints. During learning, examples from different hints are selected for processing according to a given schedule. We present two types of schedules; fixed schedules that specify the relative emphasis of each hint, and adaptive schedules that are based on how well each hint has been learned so far. Our learning method is compatible with any descent technique

Abu-Mostafa, Y. S.

English

Caltech Authors - Main

Proceedings o f  1993 International Joint Conference on Neural Networks 
An Algorithm for Learning from Hints 
Y. S. Abu-Mostafa 
California Institute of  Technologx Pasadena, CA 91 125, USA 
Abstract 
To take advantage of pr ior  knowledge (hints) about 
the  f u n c t i o n  w e  w a n t  t o  learn, we  introduce a 
method that  generalizes learning f r o m  examples to  
learning f r o m  hints. A canonical representation of 
hints is defined and illustrated. All hints are repre- 
sented t o  the  learning process by examples, and ex- 
amples  of the  f u n c t i o n  are treated o n  equal foot ing 
wi th  the rest of the  hints. Dur ing  learning, exam-  
ples f r o m  diflerent hints are selected f o r  processing 
according t o  a given schedule. We present  two types  
of schedules; fixed schedules that  specify the  relative 
emphasis  of each hint, and adaptive schedules that  
are based on h o w  well each hint has  been learned 
so far .  OUT learning method is compatible wi th  a n y  
descent technique. 
1 Introduction 
Consider the situation where we want to learn an 
unknown function f .  Typically, f is represented to 
us by a set of input-output examples. Hints de- 
scribe the situation where, in addition to the set 
of examples of f ,  we have prior knowledge of cer- 
tain facts about the function. We wish to use this 
side information to  our advantage. However, hints 
come in different shapes, and the main difficulty of 
using them is the lack of a systematic way of incor- 
porating heterogeneous pieces of information into 
a manageable learning process. If what we know 
about f is that  it is scale-invariant, monotonic over 
part of its domain, and represented by a given set 
of examples, we still have to integrate this informa- 
tion before we can learn the function. This paper 
concerns itself with the development of a system- 
atic method that  integrates different types of hints 
in the same learning process. 
How to  take advantage of a given hint can be 
an art just like how to choose a learning model. In 
the case of invariance hints for instance, preprocess- 
ing of the input can achieve the invariance through 
normalization, or the model itself can be explic- 
itly structured to satisfy the invariance. Whenever 
such a method of direct implementation is feasible, 
the full benefit of the hint is automatically real- 
ized. This paper does not offer a superior alterna- 
tive to direct implementation. However, w h e n  di- 
rect implementat ion is not  a n  option, we prescribe 0, 
systematic  method f o r  incorporating practically a n y  
hint in a n y  descent method f o r  learning. The goal 
is to automate the use of hints in learning to a de- 
gree where we can effectively use a large number 
of simple hints that may be available in a practical 
situation. 
We start by introducing the basic nomenclature 
and notation. The environ,ment X is the set 011 
which the unknown function f is defined. The 
points in the environment are distributed accord- 
ing to some probability distribution P. f takes on 
values from some set Y 
f : X - i Y  
Often, Y is just (0 , l )  or the interval [O, l ] .  The 
learning process takes pieces of information about 
(the otherwise unknown) f as input and produces 
a hypothesis g 
that  attempts to approximate f .  The degree to 
which a hypothesis g is considered an approxima- 
tion of f is measured by a distance or ‘error’ 
g : X - + Y  
The error E is based on the disagreement between 
g and f as seen through the eyes of the probability 
distribution P. 
Two popular forms off the error measure are 
E = Pr[g(z) # f (41 
E = E[(s (x )  - f(4)21 
and 
1653 
where Pr[.] denotes the probability of an event, and 
E[. ]  denotes the expected value of a random vari- 
able. The underlying probability distribution is P. 
E will always be a non-negative quantity, and we 
will take E(g,  f) = 0 to  mean that g and f are 
identical for all intents and purposes. We will also 
assume that  when the set of hypotheses is parame- 
terized by real-valued parameters (e.g., the weights 
in the case of a neural network), E will be well- 
behaved as a function of the parameters (in order 
to  allow for derivative-based descent methods). We 
make the same assumptions about the error mea- 
sures that  will be introduced in section 2 for the 
hints. 
In the learning f r o m  examples paradigm, a num- 
ber of points z1,. , X N  are picked from X (usually 
independently according to  the probability distri- 
bution P )  and the values of f on these points are 
provided. Thus, the input to  the learning process 
is the set of examples 
(21, f ( z 1 ) ) ,  . ’ ‘ , ( X N ,  f ( z N ) )  
and these examples are used to  guide the search for 
a good hypothesis. In this paper, we will consider 
the set of examples of f as only one of the avail- 
able hints and denote it by Ho, The other hints 
HI , .  . . , KM will be additional known facts about 
f, such as invariance properties for instance. 
The paper is organized as follows. Section 2 de- 
velops a canonical method for representing differ- 
ent hints. This is the first step in dealing with any 
hint that  we encounter in a practical situation. Sec- 
tion 3 lays the foundations for learning from hints 
in general, and section 4 presents specific imple- 
mentations. We discuss the overall picture in the 
conclusion. 
2 Representation of Hints 
We have so far described what a hint is in very 
general terms such as ‘a known property of f’ or ‘a 
fact about f’. Indeed, all that  is needed to  qualify 
as a hint for our purposes is to  have a litmus test 
that  f passes and that  can be applied to  the set of 
hypotheses. In other words, a hint H, is formally 
a subset of the hypotheses, namely those satisfying 
the hint. 
This definition of H,  can be extended to  a def- 
inition of ’approximation of H,’ in several ways. 
For instance, g can be considered to  approximate 
H,  within c if there is a function h that strictly 
satisfies H,  for which E(g,  h) 5 E. In the context 
of learning, it is essential to  have a notion of ap- 
proximation since exact learning is seldom achiev- 
able. Our definitions for approximating different 
hints will be part of the scheme for representing 
those hints. 
The first step in representing H, is to  choose 
a way of generating ‘examples’ of the hint. For 
illustration, suppose that H ,  asserts that 
f : [-1,+1] + [-1,+1] 
is an odd function. An example of H,  would have 
the form 
f(-) = -f 
for a particular 2 E [-1,+1]. To generate N ex- 
amples of this hint, we generate z1,. . . , X N  and as- 
sert for each z, that f(-z,) = -f(z,). Suppose 
that we are in the middle of a learning process, and 
that the current hypothesis is g when the example 
f(-z) = -f(z) is presented. We wish to  measure 
how much g disagrees with this example. This leads 
to  the second component of the representation, the 
error measure e,. For the oddness hint, e, can be 
defined as 
em = ( g ( z )  + g(-z)12 
so that e ,  = 0 reflects total agreement with the 
example (i.e., g(-z) = -g(z)). The form of the 
examples of H,  as well as the choice of the error 
measure e ,  are not unique. 
Once the disagreement between g and an exam- 
ple of H, has been quantified through e,, the dis- 
agreement between g and H ,  as a whole is auto- 
matically quantified through E,, where 
Em = &(em) 
The expected value is taken w.r.t. the probability 
rule for picking the examples. This rule is also not 
unique. Therefore, E, will depend on our choices 
in all three components of the representation; the 
form of examples, the probability distribution for 
picking the examples, and the error measure e,. 
In what follows, we will construct representations 
for different types of hints. 
Perhaps the most common type of hint is the 
invariance hint. This hint asserts that  f(z) = 
f(z’) for certain pairs z,z’. For instance, “f is 
shift-inva.riant” is formalized by the pairs z, z‘ that 
are shifted versions of each other. To represent the 
invariance hint, an invariant pair (z, z’) is picked as 
an example. The error associated with this example 
is 
em = (g(z) - g(z’)I2 
1654 
A plausible probability rule for generating (z, z’) is 
to pick x and x’ according to the original proba- 
bility distribution P conditioned on x,x‘ being an 
invariant pair. 
Another related type of hint is the monotonic- 
ity hint (or inequality hint). The hint asserts for 
certain pairs x,x’ that  f(x) 5 f(x’). For instance, 
“f is monotonically nondecreasing in x” is formal- 
ized by all pairs x, x’ such that x < d. To represent 
a monotonicity hint, an example (x,x’) is picked, 
and the error associated with this example is 
The third type of hint we discuss here is the 
approximation hint. The hint asserts for certain 
points 2 E X that f(x) E [az ,bz ] .  In other words, 
the value of f at z is known only approximately. 
The error associated with an example x of the ap- 
proximation hint is 
(s(x) - ad2  if g(x) < a2 
0 if g(x) E [a,, bz] 
e m =  { (dz) - b d 2  if g(x) > bz 
Another type of hints arises when the learning 
model allows non-binary values for g where f itself 
is known to be binary. This gives rise to the bi- 
nary hint (or Boolean hint). Let X C X be the set 
where f is known to be binary (for Boolean func- 
tions, X is the set of binary input vectors). The 
binary hint is represented by examples of the form 
x, where x E X .  The error function associated with 
an example x (assuming 0/1 binary convention, and 
assuming g(x) E [0, I]) is 
em = g(x)(l- dz)) 
This choice of e ,  forces it to be zero when g(x) 
is either 0 or 1, while it would be positive if g(z) 
is between 0 and 1. A natural probability rule for 
generating the examples is to  pick x according to 
the original probability distribution P conditioned 
on x E X .  
It is worth noting that  the set of examples of 
f can be formally treated as a hint, too. Given 
( z l , f ( z ~ ) > , - . - ,  ( ~ N , ~ ( z N ) ) ,  the examples hint 
asserts that these are the correct values off a t  these 
particular points xn. Now, to generate an ‘example’ 
of this hint, we independently pick a number n from 
1 to N and use the corresponding (xn, f(xn)). The 
error associated with this example is eo (we fix the 
convention that m = 0 for the examples hint) 
Assuming that the probability rule for picking ri is 
uniform over { 1, . . , N } ,  
. N  
Eo = €(eo) = ; C ( S ( X n )  - f ( zn ) )2  
n=l 
In this case, EO is also the best estimator of E = 
€[(g(z)-f(x))’] given x1,-*- ,XN that areindepen- 
dently picked according to the original probability 
distribution P. This way of looking at the exam- 
ples of f justifies their treatment exactly as one of 
the hints, and underlines the distinction between E 
and EO. 
In a practical situation, we try to infer as many 
hints about f as the situation will allow. Next, 
we represent each hint according to the guide- 
lines discussed in this section. This leads to a list 
HO H I ,  1 .  , HM of hints that are ready to produce 
examples upon the request of the learning algo- 
rithm. We now address how the algorithm should 
pick and choose between these examples as it moves 
along. 
3 Learning Schedules 
If the learning algorithm had complete information 
about f, it would search for a hypothesis g for which 
E ( g , f )  = 0. However, f being unknown means 
that the point E = 0 cannot be directly identified. 
The most any learning algorithm can do given the 
hints Ho,  H I , .  , HM is to reach a hypothesis g for 
which all the error measures EO, El ,  , EM are ze- 
ros. Indeed, we have required that E = 0 implies 
that E, = 0 for all m. 
If that point is reached, regardless of how it is 
reached, the job is done. However, it is seldom the 
case that we can reach the zero-error point because 
either (1) it does not exist (i.e., no hypothesis can 
satisfy all the hints simultaneously, which implies 
that no hypothesis can replicate f exactly), or (2)  
it is difficult to reach (Le,, the computing resources 
do not allow us to exha.ustively search the space of 
hypotheses looking for that point). In either case, 
we will have to settle for a point where the Em’s 
are ’as small as possible’. 
How small should each E,  be? A balance has to 
be struck, otherwise some Em’s may become very 
small at the expense of the others. This situation 
would mean that some hints are over-learned while 
the others are under-learned. We will discuss learn- 
ing schedules that use different criteria for balanc- 
ing between the hints. The schedules are used by 
the learning algorithm to simultaneously minimize 
the Em’s. 
1655 
The implementation of a given schedule goes 
as follows: (1) The algorithm decides which hint 
(which m for m = 0,1,. . . , M )  to  work on next, ac- 
cording to  some criterion; (2) The algorithm then 
requests a batch of examples of this hint; (3) It per- 
forms its descent on this batch; and (4) When it is 
done, it goes back to  step (1). We make a distinc- 
tion between fixed schedules, where the criterion for 
selecting the hint can be ‘evaluated’ ahead of time 
(albeit time-invariant or time-varying, determinis- 
tic or stochastic), and adaptive schedules, where the 
criterion depends on what happens as the algorithm 
runs. Here are some fixed and adaptive schedules: 
Simple Rotation: This is the simplest pos- 
sible schedule that tries to  balance between the 
hints. It is a fixed schedule that  rotates between 
Ho,  Hl, ... , H M .  Thus, at step k, a batch of N ex- 
amples of H ,  is processed, where m = k mod ( M +  
1). This simple-minded algorithm tends to  do well 
in situations where the Em’s are somewhat similar. 
Weighted Rotation: This is the next step in 
fixed schedules that  tries to  give different emphasis 
to  different Em’s. The schedule rotates between the 
hints, visiting H ,  with frequency v,. The choice 
of the v,’s can achieve balance by emphasizing the 
hints that  are more important or harder to  learn. 
Maximum Error: This is the simplest adap- 
tive schedule that  tries to  achieve the same type 
of balance as simple rotation. At each step k ,  the 
algorithm processes the hint with the largest error 
E,. The algorithm uses estimates of the Em’s to  
make its selection. 
Maximum Weighted Error: This is the adap- 
tive counterpart to  weighted rotation. It selects the 
hint with the largest value of vmEm. The choice of 
the u,’s can achieve balance by making up for dis- 
parities between the numerical ranges of the Em’s. 
Again, the algorithm uses estimates of the Em’s. 
Adaptive schedules attempt to  answer the ques- 
tion: Given a set of values for the Em’s, which hint 
is the most under-learned? The above schedules 
answer the question by comparing the individual 
Em’s. Although this works well in simple cases, it 
does not take into consideration the correlation be- 
tween different hints. As we deal with more and 
more hints, the correlation between the Em’s be- 
comes more significant. This leads us to  the fi- 
nal schedule that  achieves the balance between the 
Em’s through their relation to  the actual error E. 
Adaptive Minimization: Given the estimates 
of Eo, El ,  - a  . , E M ,  make M +  1 estimates of E ,  each 
based on all but one of the hints: 
-@(a, El,  E2 , . . . , E M )  
E(.E;o, 0 ,  E2, * * , E M )  
-qEO,El,.,.*. , E M )  
&(Eo, El, E%, * * * , 0) 
... 
and choose the hint for which the corresponding 
estimate is the smallest. 
In other words, E becomes the common thread 
between the Em’s. Knowing that we are really try- 
ing to  minimize E ,  and that the Em’s are merely a, 
vehicle to  this end, the criterion for balancing the 
E,’s should be based on what is happening to E as 
far  as we can tell. 
References 
Abu-Mostafa, Y. S. (1990), Learning from hints 
in neural networks, Journal of Complexity 6, 192- 
198. 
Abu-Mostafa. Y. S. (1993), A method for learn- 
ing from hints, Advances in Neural Information 
Processing Systems - 5, S. Hanson et a1 (eds), Mor- 
gan Kaufmann, 1993. 
Al-Mashouq, K. and Reed, I. (1991), Including 
hints in training neural networks, Neural Compu- 
tation 3, 418-427. 
Minsky, M. L. and Papert, S. A. (1969), “Per- 
ceptrons,” MIT Press. 
Omlin, C. and Giles, C. L. (1992), Training 
second-order recurrent neural networks using hints, 
Machine Learning: Proceedings of the Ninth Inter- 
national Conference (ML-921, D. Sleeman and P. 
Edwards (ed.), Morgan Kaufmann. 
Suddarth, S. and Holden, A. (1991), Symbolic 
neural systems and the use of hints for developing 
complex systems, International Journal of Machine 
Stiidies 35, p. 291. 
Towell, G. and Shavlik, J. W. (19921, Inter- 
pretation of artificial neural networks: mapping 
knowledge-based neural networks into rules$ Ad- 
vances in Neural Information Processing Systems 
- 4, J. Moody et a1 (eds), Morgan Kaufmann, 1992. 
Wismer, D. A. and Chattergy, R. (1978), “In- 
troduction to  Nonlinear Optimization,” North Hol- 
land. 
This work was supported by the United 
States AFOSR under Grant No. F49620-92- 
J-0398. 
1656 


An algorithm for learning from hints

A method for learning from hints,

Including hints in training neural networks,

Interpretation of artificial neural networks: mapping knowledge-based neural networks into rules$

Introduction to Nonlinear Optimization,” North Holland. This work was supported by the United States AFOSR under Grant No.

Learning from hints in neural networks,

Symbolic neural systems and the use of hints for developing complex systems,

Training second-order recurrent neural networks using hints,

http://authors.library.caltech.edu/7013/1/ABUijcnn93.pdf

An algorithm for learning from hints

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main