Howitt, Richard E.

English

Research Papers in Economics

Multiperiod Optimization:
Dynamic Programming  vs.
Optimal  Control:  Discussion
Richard E.  Howitt
The  title  of this  session  pitting  Dynamic
Programming against Control Theory  is  mis-
leading  since  dynamic  programming  (DP) is
an  integral  part  of  the  discipline  of control
theory.  However,  it is timely  to  discuss  the
relative  merits  of DP  and  other  empirical
solution  approaches  to  control  problems  in
agricultural  economics,  and  equally  impor-
tantly,  how  control  theory  can  be  used  to
pose  more  useful  and  realistic  problems.
From  this  point  in  the  paper  I  shall,  like
Burt,  use  the  term  Dynamic  Programming
(DP) to  mean  the  classic solution  procedure
developed by Bellman.  All the other solution
approaches  used  to  solve  multiperiod  prob-
lems,  including  Differential  Dynamic  Pro-
gramming [Jacobson  and Mayne],  are termed
Control  Theory.
Burt  takes  the  session  title  literally  and
rises to defend and extend the long history of
DP  research  and  application.  I  agree  with
him  concerning  the difficulty  of teaching  ap-
plied  problem  formulation,  but  like  Zilber-
man,  feel that lack of popularity  of DP cannot
be attributed  to lack of exposure  of graduate
students.
In his  section  on "Obstacles  to Implemen-
tation"  Burt is unjustifiably  pessimistic in his
judgement  of the practicality  and theoretical
basis  of  solutions  based  on  the  Pontryagin
Maximum  principle.  Given  the long history
of  Pontryagin  based  control  applications  in
engineering  and  operations  research,  the
problems cannot be categorized in general  as
"trivial"  or  without  "theorems  on  the  struc-
ture  of the  solution."  A comprehensive  sur-
Richard  E.  Howitt is  an  associate professor  in  the  De-
partment of Agricultural  Economics at the University of
California,  Davis.
vey  of solution  approaches  to  control  prob-
lems may be found in Polak [1973],  and some
specific  texts  are  Bryson  and  Ho,  Canon  et
al.,  Dyer  and  McReynolds,  Jacobson  and
Mayne,  Polak  [1971].  In  the  context  of ap-
plied economics,  Zilberman cites many stud-
ies  that  most  members  of the  profession
would  not  classify  as  trivial.  His  citations
include,  of course,  studies using DP solution
approaches.
The applied  economics  literature  has  pre-
dominantly  used  the  maximum  principle  in
the  analysis  of  the  theoretical  problems  of
dynamic  economic systems.  While there  are
special  cases  in  which the first  order  condi-
tions can be derived  using Calculus  of Varia-
tions,  Dynamic  Programming  or  the  Max-
imum principle (and shown to be the same),
the  Maximum  principle  is  both  less restric-
tive  in  the  form  of  the  controls  and  con-
straints than calculus of variations,  and easier
to interpret  than DP.  The maximum  princi-
ple has an overriding advantage  for economic
problems  in  that  it  explicitly  specifies  the
intertemporal  qualitative  properties  of  the
imputed  value  of state  variables  as  costate
variables.  In addition,  Lagrangian interpreta-
tions for binding constraints are incorporated
in the first order conditions.
In the  same  section,  Burt generalizes  the
major  problem  with  DP  - the  "curse  of
dimensionality"  to  "an  inherent  characteris-
tic of dynamic  optimization  problems in gen-
eral  when  the  number  of state  variables  is
large  (more  than  3  or  4)."  However,  much
larger  problems  are  routinely  solved  by
analytic  DP  or  programming  approaches.
The  difference  between  the  dimensionality
increases for DP and nonlinear programming
413Western Journal  of Agricultural  Economics
solutions  is  substantial.  The DP  memory  re-
quirement  is the grid size raised to the power
of the number of states, whereas the memory
requirements  for  a  nonlinear  programming
problem  are proportional  to the square of the
product of state and time period  dimensions.
Clearly,  the  effect of adding additional  state
dimensions  is  very  different.  Intrilligator
cites  a  very  difficult  DP  problem  of  grid
size/state  of 100 and four  states.  Assuming 20
time periods  and four controls  with inequali-
ty constraints  on all states  and controls  in all
time periods,  the  same problem would yield
a nonlinear  programming  problem with  160
columns  and  240  rows,  a routine  operation
using  modern  algorithms.  A  problem  with
nine states  and thirty yearly time periods was
solved  as  a  quadratic  program  in  Noel  and
Howitt.
Burt characterizes  the  dilemma of the ap-
plied  analyst  in  the statement  that  "the pri-
mary objective  in  all  modeling  is  to  capture
the  essential  aspects  of  the  phenomenon
under study  and yet keep  the model  as  sim-
ple  as  possible."  The  choice  of solution  ap-
proach  thus  depends  on  the  appropriate
model characteristics.  I agree  with Burt that
for low dimension problems  DP is a superior
method,  particularly  if the  functions  are  ir-
regular  and  a  range  of  stochastic  values  is
important  to  the  problem.  However,  for
problems  in which the  simultaneity of many
states is more important than smooth approx-
imations to  irregular functions there are sev-
eral additional  solution approaches.  Further-
more  given  the  structures  of  micro  theory
and  limitations  of  many  least  squares  and
maximum  likelihood  approaches,  most
applied  economic  problems  satisfy  the
requirements of non-DP approaches.  Specifi-
cally: continuous and convex or concave func-
tions,  differentiability,  and  stochastic  prop-
erties  that can  be characterized  by  sufficient
statistics.
Considerable  work  has  been  done  on the
stochastic  control  problem  since  Kushner
and  Schweppe's  cited  article,  a  more recent
article  with illustrative  examples  is  found  in
Haussman.  The direct applicability of a range
414
of econometric  models to optimum control is
shown  in Rausser and Hochman,  and Chow.
Burt provides a comprehensive  review and
extension  of methods  to reduce  state  vector
decisions.  The  wheat  storage  study  [Burt,
Koo and Dudley] would make  a good vehicle
for  study  of  alternative  solution  methods,
being  apparently  solvable  by  the  linear
Quadratic  Gaussian  formulation  for  Differ-
ential  Dynamic  Programming  or  nonlinear
programming,  without  any  simplification  of
state varibles,  a wider choice of controls,  and
the simplification  of a stochastic  error term.
Among  the  methods  for  reducing  dimen-
sionality in DP I would add that the regener-
ation point approach  [Dryfus  and Law] has a
natural  application  to  problem  of  capacity
expansion  and capital  replacement.  In these
latter problems,  the  advantage  that  DP en-
joys  in  integer  problems  makes  it  a logical
choice.  I am less convinced  as to the value  of
DP in analyzing  farm firms where crop alter-
natives,  cash flows,  asset stocks  and updated
rational expectations  would probably expand
the  state  dimension  to  an  unmanageable
level  for DP.
Zilberman's  paper  is written  from  a more
general  view point  which  sees  dynamic  pro-
gramming  as  an  important part of the set  of
solution  approaches  to the general  stochastic
control problem.  Zilberman skirts the details
of  solution  procedures  and  concentrates  on
the values  of optimal control theory in posing
theoretical empirical  and policy models  in an
extension  of comparative  statics.  To  his  re-
view  of  economic  applications  of  control
theory,  I would add the areas of management
science  [Bensoussan,  Kleindorfer  and
Tapiero],  consumer  demand  [Houthakker
and  Taylor]  and  rational  expectations  [Tay-
lor].  At  the  end  of  Zilberman's  review  of
control  applications  to  agricultural  econom-
ics,  he  remarks  correctly  that  "it  is  still  a
limited  tool  in  its  application  and impacts."
He  attributes  this  outcome  to the  emphasis
on  solution  techniques  rather  than  policy
results,  a crime  of which the first part of this
paper  is guilty. However,  as  Burt points out,
empirical  problems  have  to  be  formulated
December 1982Discussion
appropriately  to  be  solvable,  and  I  think  it
equally  important  that  practitioners  are
familiar  with  enough  solution  methodolgies
to  minimize  the "damage"  to the theoretical
problem.
The  majority  of  Zilberman's  paper  de-
velops  optimal control solutions  to problems
in  domains  other than  time.  The results  are
stimulating  and  demonstrate  the  power  of
the  maximum  principle  to  extend  micro-
economics.  In  his  two-stage  optimal  control
development  over  time  and  nontime  do-
mains,  Zilberman  develops  the  mi-
croeconomic equivalent of Isard et al.'s speci-
fication.  Given  the  fixed  costs  of irrigation
adoption,  it  is  not  clear  that  the  combined
problem  can  be  always  decoupled  to  allow
the  two-stage  optimization  procedure  sug-
gested.  For instance,  a major problem facing
irrigated agriculture  is the decline in produc-
tivity from rising water tables,  salinity build-
up or both.  Evidently the distribution of land
class  is  in  this  case  a  dynamic  phenomena
which is affected by both the temporal alloca-
tion of water and the adoption rates of irriga-
tion technology.  Zilberman's  development is
attractive as it lays out a microeconomic foun-
dation for rational  technological  change.
Throughout  Zilberman's  exposition of time
and nontime  domain optimization  I had mis-
givings  on  the  data  availability  to  use  the
approaches.  Zilberman  is aware  of these con-
straints,  but  is  optimistic  that  they  can  be
ameliorated  by  the  rapid  diffusion  of mini-
computers and networks  in the industry.  He
is probably correct, although compatability of
the  variable  data  sources  is  likely  to  be
troublesome.
To  return  to the  central  direction  of this
session  the two widely  differing  papers  each
point to problems in the evolution of any new
applied  methodology.  The first  priority is  to
emphasize  Zilberman's  point  that  optimal
control  must  first  offer  additional  mi-
croeconomic  insights  over comparative  stat-
ics.  Once the value to policy questions  of the
additional  time  dimensions  is  clear,  there
will be an  incentive  to extend  the  compara-
tive  static  models  of  traditional  agricultural
economics.  The  qualitative  properties  of
comparative  dynamic equilibria are often not
easy to obtain  in "nice"  forms.  An  additional
source  not mentioned by Zilberman  or  Burt
which  works  towards  comparative  dynamics
is  Kamien  and  Schwartz.  Like  Zilberman,  I
feel that there are valuable and relevant rela-
tionships to be derived using control theory.
Burt emphasizes  the  empirical  intricacies
of solution by Bellman's original DP method,
however,  the  disparagment of other solution
methods  is  unnecessary.  The  fundamental
concept of dynamic programming,  the princi-
ple of optimality,  has been used to develop a
wide  variety  of solution  procedures  that are
not  subject  to  the  curse  of dimensionality.
The stage wise solution  is maintained but the
discretization  of state  and control  space  and
its attendant curse is avoided by storing func-
tional  forms  that  characterize  the  optimal
solution in the backward solution phase. This
class  of solution  approaches  is  called  differ-
ential  dynamic  programming  DDP.  A
through treatment of DDP may be found in
Jacobson  and  Mayne  (1970),  and  converg-
ence of the approximated  problem to the true
nonlinear  problem  has  been  shown  to  be
quadratic.
The simplest example of DDP is the wide-
ly  used  Linear  Quadratic  Gaussian  (LQG)
procedure  briefly  referred  to  by  Burt.  This
procedure  while not without drawbacks,  has
immediate  appeal  for  many  economic  prob-
lems  where  the  system  dynamics  are  es-
timated by several  (usually greater than four)
linear  relationships  with  additive  normal
stochastic  terms.  The  objective  function  is
often characterized  as a weighed combination
of  producer  and  consumer  surplus,  which
given linear estimates  of demand and supply
over the relevant range, results in a quadratic
objective  function.  An  example  of a  similar
problem  specification  is  found  in  Burt,  Koo
and Dudley.
The  problem  of inequality  constraints  in
DDP is illustrated by Murray and Yakowitz's
application  to  the  multireservoir  control
problem. In many other cases inequality con-
straints  can  be  approximated  by  suboptimal
415
HowittWestern Journal  of Agricultural Economics
truncation  or penalty  functions incorporated
in the  LQG approach.
An  alternative  solution  approach  that  has
considerable  appeal  and  future  potential  is
the  solution  of  deterministic  control  prob-
lems  by  nonlinear  programming.  Efficient
general  algorithms  for  sparse  problems  that
are  nonlinear  in  both the  objective  function
and the constraints  are  now generally availa-
ble  in  MINOS,  Murtagh  and  Saunders  or
Box  and  can  be  used  to  solve  substantial
empirical  problems  such  as  Richardson  and
Ray and Noel  and  Howitt.  Further develop-
ments to  combine  this  approach  with  stoch-
astic simulations are likely to be forthcoming.
Where  the  constraint  set  can  be  success-
fully  embedded  in  the  objective  function,
very  large  control  models  can  be solved  by
unconstrained  optimization  methods.  Fair
discusses  methods for solving up to  100 state
variable  models  and  uses  a  nineteen  state
twenty  time period  model to  compare  alter-
native  approaches.  Methods  to  extend  this
type  of  solution  to  stochastic  problems  are
developed  in Tinsley,  Craine  and  Havenner
and are currently implemented  (but not pub-
lished) on the four hundred equation Federal
Reserve  model.
Inevitably  there  are  trade-offs  but  no  in-
surmountable  barriers  in  empirically  imple-
menting control problems, particularly  in the
form that agricultural  economists  have tradi-
tionally  specified them.  What is  now needed
from  advocates  of control  theory  (that  of
course  includes  dynamic  programming)  is  a
clearer  analytical  exposition  of  the  advan-
tages  of comparative  dynamics  and  dynamic
policy  models over their static  counterparts.
Box, M.  J.  "A New Method of Constrained Optimization
and  A Comparison  with Other  Methods,"  Computer
Journal 8(1965):42-52.
Burt,  O.  R.,  W.  W.  Koo  and  N.  J.  Dudley.  "Optimal
Stochastic  Control  of  U.S.  Wheat  Stocks  and  Ex-
ports,"  American Journal of Agricultural Economics
62(1980):173-87.
Bryson,  A.  E.  and Y. C.  Ho.  Applied Optimal Control,
John  Wiley,  New  York,  1975.
Canon,  M.  D.,  C.  D.  Cullum and E.  Polak.  Theory  of
Optimal Control and  Mathematical Programming,
McGraw-Hill,  New York,  1970.
Chow,  G.  C.  Analysis  and Control of  Dynamics
Economics Systems,  John  Wiley,  New  York,  1975.
Dryfus,  S. E.  and A.  M.  Law.  The Art and Theory of
Dynamic Programming, Academic  Press,  New York,
1977.
Dyer,  P.  and S.  R.  McReynolds.  The Computation and
Theory  of Optimal Control, Academic  Press,  New
York,  1970.
Fair,  R.  C.  "On  the Solution  of Optimal  Control  Prob-
lems  as Maximization  Problems,"  Annals of Econom-
ics and Social Measurement, 311(1974):135-54.
Haussman,  U.  G.  "Some Examples  of Optimal Stochast-
ic  Controls Or:  The Stochastic  Maximum  Principle  at
Work,"  SIAM  Review 23(1981):292-307.
Houthakker,  H.  S.  and  L.  D.  Taylor.  Consumer De-
mand in the United States: Analyses and Projections,
Harvard  University Press,  1970.
Intrilligator,  M.  D.  Mathematical Optimization and
Economic Theory, Prentice  Hall,  1971.
Isard, W.,  P. Liossatos,  Y. Kanemoto,  and P.  C.  Kaniss.
Spatial Dynamics and Optimal Space-Time  Develop-
ment,  North  Holland,  New York,  1979.
Jacobson, D.  H. and D. Q.  Mayne.  Differential  Dynam-
ic Programming, American  Elsevier New  York,  1970.
Kushner, H.  J.  and F.  C. Schweppe.  "A Maximum Prin-
ciple for Stochastic Control Systems," J. Mathematical
Analysis and Applications 8(1964):287-302. References
Bellman,  R.  Dynamic Programming, Princeton  Univer-
sity  Press,  1957.
Bensoussan,  A.,  P.  R.  Kleindorfer  and  C.  S. Tapiero.
Applied Optimal Control, Studies in the Management
Sciences,  Vol.  9 North  Holland,  Amsterdam,  1978.
416
Kamien,  M.  I.  and  N.  L  Schwartz.  Dynamic Optimza-
tion, North Holland,  New  York,  1981.
Murray,  D.  M.  and S. J.  Yakowitz.  "Constrained Differ-
ential  Dynamic  Programming  and  its  Application  to
December 1982Discussion
Multireservoir  Control,"  Water Resources Research,
15(1979):1017-27.
Murtagh,  B.  A.  and  M.  A.  Saunders.  "Large  Scale
Linearly  Constrained  Optimization,"  Mathematical
Programming  14(1978):41-72.
Noel,  J.  E.  and R.  E.  Howitt.  "Conjunctive  Multibasin
Management:  An Optimal  Control Approach,"  Water
Resources Research (forthcoming  1982).
Polak,  E. Computational Methods in Optimization Aca-
demic Press,  New York  1971.
. "An  Historical  Survey of Computational  Meth-
ods  in Optimal Control,"  SIAM  Review 15(1973):553-
84.
Rausser,  G.  C. and E.  Hochman. Dynamic Agricultural
Systems: Economic Prediction  and Control North Hol-
land,  New York,  1979.
Richardson,  J.  W.  and  D. E.  Ray.  "Commodity  Pro-
grams  and  Control  Theory,"  American Journal of
Agricultural  Economics 64(1982):28-38.
Taylor,  J.  B.,  Editor.  "Issue  on  Rational  Expectations
Analysis." Journal  of Economic Dynamics and  Control
2:1  February  1980.
Tinsley,  P.,  R.  Craine,  and  A.  Havenner.  "Optimal
Control  of  Large  Nonlinear  Stochastic  Econometric
Models,"  Proceedings  Summer  Computer Simulation
Conference  Chicago,  IL,  IEEE,  1977.
417
Howitt

MULTIPERIOD OPTIMIZATION: DYNAMIC PROGRAMMING VS. OPTIMAL CONTROL: DISCUSSION

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics