A formulation of the problem of making decisions concerning the state of nonstationary stochastic processes is given. An optimal decision rule, for the case in which the stochastic process is independent of the decisions made, is derived. It is shown that this rule is a generalization of the Bayesian likelihood ratio test; and an analog to Wald's sequential likelihood ratio test is given, in which the optimal thresholds may vary with time

Schaefer, B.

English

NASA Technical Reports Server

Enclosures:
1. Final Technical Report
2. "Sequential Decision Analysis for Non-Stationary Stochastic Processes"
by Brian Schaefer.
Copies sent to:
5 copies to NASA Scientific and Technical Information Facility
P. O. Box 33
College Park, Maryland 20740
2 copies to National Aeronautics and Space Administration
Proposal Control Officer
Office of University Affairs
Washington, D. C. 20546
2 copies to C. Ray Davis, Jr., Grants Officer
N -tional Aeron .L±c~s ina apace Administration
Langley Research Center
Hampton, Virginia 23365
1 copy to John H. Painter
National Aeronautics and Space Administration
Langley Research Center
Hampton, Virginia 23365
(NASA-CR-140625) SEQUENTIAL DECISION N74-34984
ANALYSIS FOR NONSTATIONARY STOCHASTIC
PROCESSES Final Technical Report.
(Northwestern.Univ.) 16 p HC $4.00- Unclas
CSCL 12A .G3/19 51111.
S- -t;
-C C
s~s~~IF~-4
https://ntrs.nasa.gov/search.jsp?R=19740026871 2020-03-23T02:43:42+00:00Z
Final Technical Report
on NASA Grant
NGR 14-007-129
"Decision Directed Adaptive Estimation"
submitted by the Principal Investigator
Professor Brian Schaefer
Department of Industrial Engineering
and Management Sciences
Northwestern University
Evanston, Illinois 60201
October 1974
1. Progress of the Ph.D. Student Supported by this Grant.
Mr. Bombran Shetty, a Ph.D. student in the Industrial Engineering and
Management Sciences Department at Northwestern University was supported
(tuition and stipend) by this grant. He was unable to make significant
progress toward the solution of the problem addressed in the research pro-
posal, although he spent a great deal of time on it and worked very dili-
gently. Finally, in March 1974, he was forced to drop out of school for
personal reasons. This unfortunate series of circumstances caused con-
siderable delay in the progress of the research.
2. Sequential Decision Analysis.
The principal investigator began to work full time on the grant after
the leparture of Mr. Shetty. It was felt that in order to deelops an adap-
tive estimator for processes in which the mean and variance of the obser-
vation noise are unknown and may be changing in time, a procedure must be de-
veloped for making sequential decisions on non-stationary stochastic processes.
Current statistical decision theory deals only with time independent ran-
dom variables, and the results of optimal stochastic control theory, which
do deal with the above problem, are usually not amenable to actual algo-
rithmic implementation. Research toward development of such a procedure
produced some independently interesting results, and are contained in the
accompanying paper. These results also solve a major portion of the prob-
lem addressed in the research proposal. This paper is being submitted for
publication to the Institute of Mathematical Statistics, and will be pre-
sented at the 1975 ORSA meeting in Chicago.
-2-
3. Continuing Work.
The principal investigator is continuing research on the problems
addressed in the research proposal for this grant. One paper on model
evaluation is currently being revised and the use of the above sequential
decision algorithm in adaptive Kalman filtering is being considered. In
the Kalman filter, the losses incurred for using an incorrect model are
well-known, and these will be used as the loss function in the decision
algorithm. Results of this research will be forwarded to NASA, as an
addendum to this final report, when they are completed.
SEQUENTIAL DECISION ANALYSIS
FOR
NON-STATIONARY STOCHASTIC PROCESSES
by
Brian Schaefer
Northwestern University
Evanston, Illinois
September 1974
Abstract
A formulation of the problem of making decisions concerning the state
of non-stationary stochastic processes is given. An optimal decision rule,
for the case in which the stochastic process is independent of the decisions
made, is derived. It is shown that this rule is a generalization of the
Bayesian likelihood ratio test; and an analog to Wald's sequential likelihood
ratio test is given, in which the optimal thresholds may vary with time.
* Research supported in part by NASA Grant NGR 14-007-129.
I. Introduction
The general framework of the sequential decision problem has
remained in essentially the same form as originally formulated by Wald
[1947]. This formulation involved a sequential decision concerning the
choice of obtaining another sample or making a final decision.
In this paper we generalize this problem to include making de-
cisions on the state of a non-stationary stochastic process and are able
to obtain a convenient solution for the case in which the state of the
process is independent of the decisions made.
Such a formulation is of interest in problems involving estimation
or signal detection as used in the tracking of missiles or commercial aircraft.
In these problems, the decisions usually do not influence the original
prcess. Another area in which LLiz fEcrmulation is appropriate would be
problems involving economic decisions where the processes under ob-
servation, such as stock prices or government indices, are relatively
independent of decisions made on a personal or corporate level.
II. Sequential Decisions for Stochastic Processes
We will let T be a linearly ordered parameter set, and will assume
that z(t), t E T is a stochastic process defined on a probability
space (®8,,?7). We will also let y(t), t E T be a stochastic process defined
on a family of probability spaces (Q,B,Pe): e E 6 .
The set of admissible actions will be given as a measurable space
(7,A). An action process, a(t), t E T, will be defined, on 67; such a process
is similar to a stochastic process without the probability measure, that is,
a: 7- RT
A measurable loss function L: 9 x 67 - R will be defined as will a set,
D, of measurable decision functions, where d E D and d: Q -67.
-2-
The function d will be required to operate on 
0 only as a causal function
of y(t), t E T; that is, for any - E T
a(r) = d(y(t), t 5 T)
The Bayesian decision problem consists then in 
finding a d* E D such that
the risk function r(d) = E[L(G,d(w))] is minimized
(1) r(d*) = min L(9,d(w))dP (w)d7?( )
d E D (9 Q
The complexity of the above minimization problem is determined 
by the
nature of the probability spaces 0 and C2, the 
loss function L, and the
decision set D. The loss function L will be defined 
on 8 through the
process z(t), t E T, so that L(e,d(w)) 
= L(z(t), t E T; d(y(t),t E T)).
If z(t), t E T, is a function of d(y(t), t E T) 
then the minimization in
(1) is a problem usually studied in stochastic optimal 
control theory, see
f~r e~xapo -rushner [19671 . In the particular case wiiei. L(T + 
A) is a function
only of z(T) and a(T) the problem is usually referred 
to as a Markov decision
problem.
Although in the study of stochastic control theory 
and Markov decision
processes it is possible to obtain necessary 
conditions for the optimal de-
cision rule in some cases, these conditions often 
do not lead to a practical
explicit solution. In the next section we 
will make several assumptions that
will lead to a simple explicit solution to (1). These 
assumptions will
usually be true in the case where the problem involves decisions 
concerning
the state of the z(t), t E T process, rather than the control 
of this process.
In Wald's original formulation the process z(t), t E T is a constant 
z(t) =
al, or a2 for all t E T, and the 
loss function is a constant until 
a decision
is made, representing the cost of observations, and 
zero after the decision
has been made. Although Wald did not adopt the Bayesian 
context, his results
-3-
would be unaltered if equal a priori probabilities were assumed. In the
following section we will derive an interesting analog to Wald's results.
III. Non-Controlled Processes and Independent Action Processes
In this section we will assume that z(t), t E T is not a function
of a(t), t E T and will consider the following form of the loss function
(2) L(e,a) = J L(z(t), 
a(t)) dt
where L(z(t),a(t)) > 0 V t E T
= L(z(t), d(y(T), T t))dt
The notation f(z(t)) will imply that f(.) may be a function of t as well as
the value of z at t; that is, f(z(t)) = f(z(t),t). Assuming that the following
integrals exist, the risk function, (1), becomes, with some abuse of
notation,
r(d) = i L(z(t), d(y(T), T : t)) dt dP (w)d7?(G)
= e L(z(t),d(y(r),T 5 t)) dM(ew)dP(w)dt
Sr R L(z(t),d(y(T),T r t))d(z(t)ly(T),T < t)dP(y(T),T 5 t)dt
T Rt R
We will let C7(t) represent the set of admissible actions,
at time t, and we will make the assumption that c7(t) is independent of a(T),
t # T for any t, T E T. That is, the set of admissible actions at any given
time does not depend on an action taken at any other time. Such processes
a(t), t E T, we will call independent action processes
If we let
(3) 1(,y,t) = L(z(t),d(y(T), T t) = a)d?(z(t)ly( ),. T t), acE a(t)
R
-4-
then the above risk function becomes
(4) r(d) ;= (,y:,t)dP(y(T),T : t)dt
T R
Theorem
If:a(t), t E T is an independent action process; z(t), t E T does
not depend on a(t), t E T; and the loss function is of the form given in (2),
then the risk (1) is minimized by the following decision rule:
d(y(T), T ! t) = ct* a* E 7(t)
iff
(5) (a*,y,t) ; ?(a,y,t) Va E c7(t)
Proof
Since L(z(t), d(y(T), T ! t) > 0 Vt E T, from (3) we have
Z(cx,y,t) 2 0 for all a E c(t), t E T, and all y(T), T r t. Thus (4) will
be minimized by choosing the a E 7(t) that minimizes £(a,y,t) for each t E T
and cauh y(i,), 5 t. Given (4),-chis iast statement,may be proved simply
by contradiction. 0
We will also define
(6) Q(a,y,t) = L(zt),d(y(T),T < t)P(y(T),T 2 tlz(t))d(z(t))
R
then
Corollary
Given the assumptions of the above theorem, the risk (1) is
minimized by the decision rule
d(y(T), T 5 t) = a * 6 E(t)
iff
(7) Q(Vayt) f Q(c,y,t) VaE at
-5-
Proof
From (3) and (6) Q(a,y,t) =£(aO,y,t)P(y(T), r 5 t)
and therefore
Q(ad*,y,t) 5 Q(a,y,t)
iff ?(o*,y,t) L (c,y,t)
a) Estimation Given a Quadratic Loss Function
If
L(z(t),d(y(T), T t))
= (z(t) - d(y(T),T t)) 2
then the optimal decision rule as determined from (3) and (5) is
(8) d(y(T), T t) = E[z(t)j y(T),T 5 t].
This result is given in Doob [19531.
The conditions required of y(t) and z(t), t E T by the Kalman
filter, Kman 1960], ar precisely thozz such that (8) can b-computed rccursively
in time.
b) Finite State and Action Spaces
Suppose
z(t) E ~ji: i = 1, ... , n) V t E T
and a(t) E (Bi: i = 1, ... , m) V t E T
and that the observation process y(t), t E T is some process related to
z(t), t E T, as depicted in Figure 1.
-6-
z (t)
T
y(t)
I
a(t)
Figure 1: Sample realizations of z(t), y(t), a(t), t E T
-7-
a(t) is a function of y(T), T 5 t, and we will define
L(z(t) = ai,d(y(T), T 5 t) = ) = L.(t) > 0.
In a practical situation we might be trying to determine the state of z(t)
from noisy data, and the loss would be minimized if a(t) = z(t), t E T.
The above Theorem gives the following optimal decision rule.
a(t) =  j
n n
iff Li..(t) ?(z(t) = a.i y(T),T t) L..(t) (z(t) = .iY(T), T t)
i=1i=l
j = 1,..., m.
(i) Likelihood Ratio Test
It is well known that for simple binary random variables and fixed
sample size, both the optimal Bayes test and the Neyman-Pearson test result
in comparing the likelihood ratio with a simple threshold. If z(t) E (ala21,
and a(t) E 1 ,a2 V t E T and assuming that Lii (t) L. ij(t), i = 1,2,
th^- frc- (7) the optimal decisinr rulk is shown-to be
aP(y(8), < tiz(t) = l) [L 2 1 (t) - L2 2 (t)T]?(z(t) = a2 )
a(t) =  y(), tz(t) = a2 ) [L 1 2 (t) - L1 1 (t)](z(t) = 1)
a(t) = a2  otherwise.
Thus if z(t) is constant in time, and the loss function is independent
of time, the above decision rule reduces to the familiar likelihood ratio
test. On the other hand, the result shows that for a stochastic process
the optimal decision rule consists of a time varying likelihood ratio
test.
(ii) Analog to Wald's Original Formulation
In Wald's original formulation, he considered processes which were
in one of two states for all t E T; that is, the process was a simple
-8-
binary random variable. This will be a special case of the following, in
which we will assume that the process may be in either of two states at any
time t E T.
z(t) E fa1,' 2 V t T
We will let our action space be such that
a(t) E al',a2 ,a 33 V t E T
where:
a(t) = a 1 corresponds to the decision that z(t) = al;
a(t) = o2 corresponds to the decision that z(t) = a2;
and, a(t) = a3 corresponds to the action of making no decision at this time,
other than to wait until we have obtained another observation. We will
specify a loss function that requires us to pay for this additional informa-
tion and waiting time. We will let
Ll(t) < Ll3(t) < Ll(t) Vt Fr T
and L2 2 (t) < L2 3 (t) < L2 1 (t) V t E T.
That is, the loss incurred for making a correct decision is less than the
loss for making no decision, which is less than the loss for making the
wrong decision. For simplicity we will normalize the losses such that
Ll 1 (t) = L2 2 (t) = 0.
If we assume that P(z(t) = al) = P(z(t) = a2 ) V t E T then the decision
rule determined by (7) consists of computing the likelihood ratio
(y(T)' , T < t z(t) = a)
(9) A(y,t) = ( (y(T), T 5 tlz(t) = ad2)
and setting
a(t) = al if A(y,t) > L2 1 (t) - L2 3 (t)
L (t)
'13
-9-
a(t) = c2 if A(y,t) < L2 3 (t)
L1 2 (t) - L1 3 (t)
a(t) = a3 otherwise.
If the loss function is time independent, then the similarity
of this to Wald's result of constant thresholds is striking. In particular
if we let L12 = L21 and L23 = Probability of a Type I error.
L21
L 1 3
-- = Probability of a Type II error.
21
then these are exactly the Wald thresholds.
If we wish, however, we can make L13 (t) and L2 3 (t), increasing
functions of time, which correspond to making the loss incurred by in-
decision greater the longer a decision is delayed. This
would LiirL Oie above t1ireshol_ clusem Logether as showu in Figure 2,
although we note that the inequality
L 12 (t) > L1 3 (t) + L2 3 (t) V t E T
must be satisfied.
This inequality is a statement of the fact that if the total cost of
indecision is greater than the cost of a wrong decision, then a decision should
always be made.
Q(y,t)
Region 1
a(t) = al
L21 L23
constant threshold =
13
'\ time dependent
\ threshold = L2 1 (t) - L2 3 (t)
Region 3 L(t)
a(t) = 3  L13(t)
Region 2 L23
a(t) = L2 L12-L13
-10-
A major difference between this test and Wald's test is, that
the test continues for a fixed time interval, T. Once a threshold is
crossed, the test does not stop, rather the action is constant until the
same threshold is recrossed in the opposite direction.
IV. Conclusion
This paper has considered the problem of making decisions on
the state of a stochastic process. A solution to the problem has been
derived from the case in which the state of the process is independent
of the decisions made, the set of admissible actions at any time is in-
dependent of the action taken at any other time, and the loss function
is of the form given in (2). These assumptions are usually made im-
plicitly in the derivation of the conditional expectation as the solution
to the minimum variance estimator,.and this solution is shown to
foilow from the decision rule derived in this paper. The above assumptions
are also true in the formulation of the standard Bayesian likelihood ratio
test, and the Neyman-Pearson test, and this paper therefore, is a generation
of these tests.
In the Wald test, the assumption of the set of admissible actions
being independent of actions taken at any other time, is not true, since
once a threshold is crossed no more observations may be taken. Often,
however, one would like to formulate a statistical test, in a sequential
manner, and continue to accept observation after one hypothesis has tentatively
been accepted. This would be particularly true if the hypothesis that was
true, could change over time. The solution to such a formulation is given
in this paper.
References
Doob, J. L. (1953), Stochastic Processes, Wiley, New York.
Kalman, R. (1960), A new approach to linear filtering and prediction,
ASME, J. Basic Engr., D82, 35-44.
Kushner, H. J. (1967), Stochastic Stability and Control, Academic Press,
New York.
Wald, A. (1947), Sequential Analysis, Wiley, New York.


Sequential decision analysis for nonstationary stochastic processes

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server