Kleijnen, J.P.C.

English

Research Papers in Economics

O~EG4  The Int.  Jl of Mgrnt $ci..  Vo{. 7.  No. 6. pp.  53g to 543  0305-0a83 791101-0539$02.00  0 
0  Pergamon  Press  Lid  ]g7g  Prmtcd  in  Great Britain 
Evaluation of Management 
Information Systems 
JPC  KLEIJNEN 
Katholieke Hogeschool, Tilburg 
(Rec'eit'ed January.  1979~ 
The  economic  evaluation  of  Management  Information  Systems  may  be  based  on  the  following 
theories and techniques: Control Theory, System Dynamics, (discrete-event)  simulation, and gaming. 
Applications  of these  approaches  are  summarized. Advantages  and  disadvantages  of  the  various 
approaches  are  presented. 
1.  INTRODUCTION 
TRADITIONALLY  the  emphasis  in  computer 
science and  information  science (called  'infor- 
matica' in  Europe) has been on  the evaluation 
of the  technical  performance of computer  sys- 
tems, measured by such criteria as throughput, 
response time,  CPU  (Central  Processing Unit) 
utilization, (see,  for  instance,  [7]).  Only  a  few 
attempts have been made  to model the econo- 
mic  performance of computerized  information 
systems--or  MIS  (Management  Information 
Systems)--measured  by  criteria  like  profits, 
sales,  inventories  etc.  In  this  paper  we  shall 
evaluate economic evaluation studies based on 
System Dynamics models and 'discrete' simula- 
tion. 
2. CONTROL THEORY 
Control Theory highJights  the  role  of feed- 
back  and  feedforward  information.  Negative 
feedback means that information on a  variable 
is compared to some desired goal value, and in 
the  case  of deviation  between  the  actual  and 
the goal value, corrective action is taken.  Feed 
forward means that predictions or forecasts are 
made, such as sales forecasts. The advantage of 
Control Theory over techniques such  as simu- 
lation  is  that  it provides more general  conclu- 
sions.  The  disadvantage  of Control  Theory  is 
that it must introduce drastic simplifications in 
order  to  keep  the  mathematical  problems 
within manageable limits. Hence its models are 
usually continuous, i.e.  a  very aggregated  view 
of the world is maintained. The models are also 
linear in order to simplify analytical  solutions 
(using  Laplace  transformations).  Steady-state 
solutions are  relatively easy to  obtain  so  that 
transient behavior tends to be neglected. Deter- 
ministic models are  simpler to solve  than  sto- 
chastic  models.  Hence  bias  but no  stochastic 
noise was examined in the study by Politzer & 
Wilm6s [9]  which we shall discuss below. 
Delays are  modeled through,  e.g. first  order 
exponential  delays.  The  effects  of such  delays 
are illustrated  in  Fig.  1:  In  the  upper  part  the 
input  shows  a  temporary  increase  (pulse)  and 
in the lower part the input shows a  permanent 
increase  (ramp  function).  The  reaction  of the 
output depends on the order of the delay func- 
tion, as the figure illustrates.  These delay con- 
cepts correspond with a continuous view of the 
world as is maintained in electrical engineering. 
For a critical discussion of such delay functions 
(also  used  in  System  Dynamics)  we  refer  to 
Riethmiiller &  Schreiber [10].  In  Kleijnen  [6] 
several sources of delay in  MIS  are  discussed, 
such  as  update  interval  (say,  weekly  versus 
539 540  Kleijnen---Eraluarion of Management Information Systems 
input 
i 
I  N 
i 
I 
I 
i 
I 
I 
I 
I 
I 
I,  a/ 
I! 
.~'---''-~output  for second order delay 
,\  ,/  xx  x  \\111  \\ 
/\  \ 
\  //  N\ 
I 
~'N  \\ 
N  N 
""" """-.output for first order delay 
input 
output for first order deloy  ~-~'~ 
/  /  f-~ 
ti  /  j/ 
/  / 
1  /  /  / 
/  / 
¢  ,/output for second order delay 
/ 
/  I 
/ 
/  / 
/  ¢ 
/  // 
/ 
FtG.  1.  Delays of  first and second order 
monthly  processing),  retrieval  delay  in  real- 
time-systems, decision  delay (time  to  reach  a 
decision and to affect the physical system) etc. 
These sources of delay would require non-con- 
tinuous,  non-linear,  stochastic  models  if  we 
want a  realistic  model.  So  Control  Theory  is 
far removed from  the  mental models  used  by 
System Analysts and  most  users  in  data  pro- 
cessing. It remains possible that at higher levels 
of decision-making a  more aggregated view as 
proposed  by  Control  Theory,  becomes  more 
appropriate. 
Politzer  &  Wilm6s  [9]  applied  Control 
Theory to the evaluation ofa MIS. They restric- 
ted  their  study to  a  classical  model  for  plan- 
ning inventories and production originated by 
Holt  et  al.  [-5] and  known  as  the  HMMS 
model.  This  model  assumes  simple  quadratic 
cost  relationships.  The  authors  derived  a 
steady-state  solution  based  on  a  continuous 
version of the HMMS model, and they studied 
the  effects  of  (exponential)  'delay'  and  bias. 
Gross value was found to react asymmetrically 
to multiplicative bias, i.e. overestimation had a 
different  effect  compared  to  underestimation. 
Delay  was  found  to  have  an  optimal  value 
larger  than  zero,  i.e.  on-line  systems  do  not 
necessarily  yield  higher  gross  benefits.  How- 
ever, remember that these conclusions were de- 
rived  under  a  number  of  restrictive  assump- 
tions? 
If the  Control  Theoretic  model  cannot  be 
solved analytically then we may resort to simu- 
lation, especially System Dynamics simulation. 
3. SYSTEM  DYNAMICS 
System Dynamics (SD)  or  Industrial  Dyna- 
mics has as its philosophy that each socio-tech- 
nical system can be looked upon as a  dynamic 
system with (negative)feedback. Since  it takes 
time to collect and process data on the output, 
to  compare  the  resulting  information  to  the 
norm, and to adapt the decision variables, time 
lags are created which  make the  model dyna- 
mic.  Delays  in  information  transmission  may 
cause  fluctuations  within  the  company,  as 
opposed  to  externally caused  business  cycles. 
The  computer  language  that  has  been  devel- 
oped  especially  for  this  SD  approach  is 
DYNAMO,  but  other  languages  can  also  be Omega.  Vol.  7.  No.  O  541 
used.  e.g.  FORTRAN  or  the  simulation  lan- 
guage CSMP. 
There  has  been  considerable  controversy 
about  the  usefulness  of the  SD  methodology. 
SD  may  tackle  any  problem,  ranging  from 
some  subsystem  within  a  company  to  such 
world problems as world energy resources. The 
approach  does  not  pretend  to  give  exact  nu- 
merical predictions but it does show  the  dyna- 
mic  properties  of a  system  [1].  Thissen  [13] 
examined a variety of techniques for improving 
the  understanding  of  complex  SD  models. 
Nordhaus [8] emphasized that in  SD  relation- 
ships and variables are not based on empirical 
data--in  contrast  with  traditional  econome- 
trics-while changing an assumption may dras- 
tically  affect  the  model's  outcome.  Compared 
to  other  simulation  approaches,  SD  has  the 
advantage  of  being  more  than  a  technique. 
However, the very advantage of being a frame- 
work  means  that  the  modeler  has  to  adopt  a 
world  view  (with  levels  and  rates)  that  is  not 
spontaneous,  but  requires  learning  (or  indoc- 
trination?). For instance, a discrete-event simu- 
lation of a  queuing system is a direct represen- 
tation of the spontaneous, intuitive way such a 
system  would  be  described  by  'anybody'.  Un- 
fortunately, such a discrete-event description is 
too detailed for 'large' systems. Note that if one 
is  willing  to  adopt  the  SD  view  of the  world, 
mastering  the  SD  technique  needs  practice  in 
the art of model building. 
Of special  interest  to  us  is  the  concept  of 
delay  in  SD.  As  in  Control  Theory  SD  uses 
delays of first order, second order, etc: see Fig.  1 
above. We find  a  discrete  view  more  compat- 
ible with the MIS world view (see section 2). 
Though a great many studies have been per- 
formed using the  SD  methodology, only a  few 
of these  studies  concern  the  explicit  study  of 
information systems. 
Swanson  [12]  investigated  the  effects  of 
error,  distortion  (bias),  delay,  and  sampling 
(update  interval).  He  modeled  the  control  of 
manpower,  production,  and  inventory,  based 
on  information on  sales  and  inventory. Swan- 
son  found  that  delay  had  the  largest  effect, 
while error had the smallest effect. His conclu- 
sions, however, are  not  based  on  solid  statisti- 
cal methods: only two simulation runs per case 
with no interactions measured. 
At  Philips  Industries a  series of studies  was 
performed under the acronym IPSO:  Initiating 
Production by Sales Orders  [3, 14].  This stud- 
ied  the  effects  of  delays  in  information  and 
examined  the  scope  of the  IS.  In  a  chain  of 
production  processes--or  more  generally,  a 
network  of  production  processes--a  process 
may  be  provided  with  information  on  the 
demand  from  the  next  process.  Alternatively, 
this information may be augmented  with  mar- 
keting information, i.e. the demand by the ulti- 
mate customers of the company. Extending the 
scope  of the  data  base  so  that  besides  "local' 
data 'global' data are also available, was found 
to  dampen  the  oscillations  in  the  company's 
inventories and  activities.  Note that  this  series 
of studies  combined  a  number  of  techniques 
and ideas ('models'). The models  were strongly 
influenced  by  System  Dynamics.  The  simula- 
tion experiments were  combined  with  analyti- 
cal  results  derived  for  simplified  models,  e.g. 
Control  Theory  models.  Analysis  suggested 
which  SD  models  to  simulate,  and  simulation 
results were used to test analytical approxima- 
tions. 
In  conclusion,  SD  is  an  extremely  popular 
method. It is much more than a  technique; it is 
a world view. SD is best known for its study of 
large  scale  systems:  World  Dynamics,  Urban 
Dynamics,  etc.  For  such  large  systems  an 
aggregated view seems necessary. At  such  high 
levels of aggregation ('the' pollution, 'the' world 
consumption) continuous models  may be  ade- 
quate:  flows,  rates,  exponential  delays.  How- 
ever,  at  the  level  of  MIS  we  prefer  more 
detailed.models with  discrete events, aperiodic 
decisions, and realistic representations of infor- 
mation attributes.  A technique permitting such 
a  level of detail  is  simulation  together with  its 
extension, gaming.  Note that  the  derivation  of 
adequate decision rules (policies) is  assisted  by 
SD modeling. Especially when the MIS is inte- 
grated  (e.g.  coupling  the  production  and  the 
inventory system),  then  large  systems  must  in- 
deed be managed. 
4. SIMULATION 
The  structure  of simulation  models  can  be 
different in  the  following respects.  The  system 
may be modeled by difference equations so that 
at  equispaced  points  of time  the  state  of  the 
system is computed. An example is a corporate 
simulation  model based  on  balance  and  profit 
&  loss  account  relationships.  Slightly  more 542  Kleijnen--Evaluation of Management Information Systems 
complicated  difference  equations  are  used  in 
System Dynamics: a  set  of recursive difference 
equations  with  variable  coefficients (rates).  In 
discrete  event  simulation  the  model  is  a  most 
detailed  representation  of the  real-world  sys- 
tem  in  so  far  as  individual  events  are  repre- 
sented.  Many  languages  have  been  developed 
for this approach, e.g.  GPSS and SIMSCRIPT. 
In  this  section  the  term  'simulation'  will 
exclude SD simulation models. 
Simulation permits the most realistic models. 
Moreover,  the  user--the  client  of  the  model 
builder---can  better  understand  a  simulation 
model.  A  disadvantage  is  that  it  may  take 
much  effort  to  build  and  run  a  simulation 
model,  while  the  responses  are,  strictly,  valid 
only for the  specified values of the  input  vari- 
ables  and  parameters.  In  Kleijnen  [6],  how- 
ever,  we  have  explained  how  statistical  tech- 
niques  can  be  used  to  generalize  simulation 
results.  Next we shall  present  an example of a 
simulation study on the value of information. 
Boyd &  Krasnow [2]  performed an interest- 
ing discrete-event  simulation study on the time- 
liness  of  information  in  the  context  of  a 
hypothetical manufacturing company. They in- 
vestigated  periodic  decision-making,  tactical 
decisions being made less frequently than oper- 
ational decisions. They also varied the process- 
ing  delay  (turnaround  time).  The  delays 
affected the  speed  with  which  decisions  could 
react  to  changes  in  the  environment  such  as 
changes in demand. This lead to fluctuations in 
both  profit and  physical  variables  like  inven- 
tory.  Reductions  in  the  various  delays  im- 
proved profit significantly. 
Note  that  Welke  [15,]  criticized  the  ad  hoc 
character  of simulation  studies  such  as  made 
by Boyd &  Kranow  1-2,]. He  proposed  a  gen- 
eral framework for the simulation  modeling of 
information systems. His framework supports a 
user's view, as opposed to the  traditional  tech- 
nical data processing view. The framework can 
be  applied  to  a  particular  system  in  order  to 
generate  a  descriptive  model,  which  can  next 
be evaluated  using a  discrete-event simulation 
package.  We  refer  to  Welke  [15,]  for  more 
details. 
human  participants  or "players'.  Let  us  briefly 
compare gaming  to  pure computer  simulation 
including  System  Dynamics  simulation.  The 
relative advantage  of gaming  is  its  behavioral 
realism:  Decision-makers may have multiple-- 
possibly  vague--goals.  They  may  use  heuris- 
tic-possibly  inconsistent---decision  rules. 
They may form competing or colluding teams. 
They  may  show  learning  behavior  or  fatigue 
and boredom, and so on. The disadvantages of 
gaming  relative  to  pure  machine  simulation, 
are  lack  of  experimental  control  and  certain 
practical nuisances.  If the purpose of the game 
is  to reach general  conclusions  rather  than  to 
train  people, then  human  participation  creates 
much  noise.  For  instance,  a  player  can  show 
inferior performance not because he is supplied 
with  poor  information  but  because  he  lacks 
'feeling', gets bored, is confronted with superior 
opponents,  etc.  General  conclusions  require 
repeated runs with the game. But repeated runs 
are  expensive  or  may  even  be  impossible 
because of lack  of players and computer time. 
Moreover,  repeated  runs  are  not  completely 
comparable  since  the  players  show  learning, 
fatigue etc. 
This brief discussion was not meant to decry 
gaming as a  useful research  instrument,  but  to 
warn  against  possible  pitfalls.  A  general  refer- 
ence on gaming is Shubik  [I 1]. 
The  'Minnesota'  gaming  experiments  were 
performed  at  the  University  of  Minnesota's 
MIS  Research  Center  during  the  period 
1970-1975.  In  these  experiments relationships 
were investigated among 
(i)  the information system itself; 
(ii)  the decision-maker:  psychological charac- 
teristics, experience; 
(iii)  the  decision  environment:  operational 
versus strategic, etc. 
Dickson  et  al.  I-4,] summarized  nine  different 
gaming  experiments.  The  experiments  taken 
together,  showed  that  effects  did  occur  when 
changing the experimental factors, i.e. informa- 
tion system, decision-maker, environment.  Un- 
fortunately,  no  simple  general  relationships 
could be formulated. 
5. GAMING 
In  gaming  or  man-machine  simulation  one 
or  more  input  variables  is  determined  by 
6. CONCLUSION 
The evaluation  of the  economic  benefits  of 
MIS  forms an  underdeveloped  area.  In  order Omega,  Vol.  7,  No. 6  543 
to make progress  in  this area,  we  need  theory 
plus tools. Theories  provide  framework,  clues, 
guidelines, ideas,  etc.  Tools  provide  numerical 
results.  In the context of MIS relevant theories 
seem to be: Control Theory, System Dynamics, 
Bayesian Information Economics. Useful tools 
are simulation, gaming, and--more generally-- 
mathematical modeling and  statistics.  A  new 
management style,  fostered  by  more  scientifi- 
cally trained managers, might stimulate the ac- 
ceptance  of  formal  models.  This  paper  sur- 
veyed a number of formal theories, models, and 
techniques. Their practical application remains 
as the major task. 
REFERENCES 
1.  ANSOFF HI  &  SLEVIN DP  (1968)  An  appreciation  of 
industrial dynamics. Mgmt Sci.  14(7), 383-397. 
2.  BoYo DF & KRASNOW HS (1963) Economic evaluation 
of management information systems.  IBM  Systems  Jl 
2, 2-23. 
3.  BRAAT JJM  {1973) The  IPSO  control  system.  Int.  Jl 
Prod.  Res.  11(4), 417-436. 
4.  DICKSON GW,  SENN  JA  &  CHERVAN'Y NL  (1977) 
Research  in  management  information  systems:  the 
Minnesota experiments. Mgmt Sci. 23(9), 913-923. 
5.  HOLT CC,  MODIGLIANI  F,  MUTH JF  &  SIMON HA 
(1960)  Plannin  9  Production,  Inventories  and  Work 
Force.  Prentice-Hall,  Englewood-Cliffs,  New  Jersey, 
USA. 
6.  KLEIJNEN JPC (1980)  Computers and  Profizs: Quantify- 
ing  Financial Benefits of Information.  Addison-Wesley, 
Reading. Massachusetts, USA. 
7.  KOBAYASHI H  (1978)  Modeling and  Analysis:  an  Intro- 
duction  to  System  Performance  Evaluation  Methodo- 
logy.  Addison-Wesley, Reading, Massachusetts, USA. 
8.  NORDHAUS WD (1973) World dynamics: measurement 
without data. Econ Jl 83,  1156-I 183. 
9.  POLtTZER JP &  WtL,',,IES P  (1977)  Optimal  Design  or" 
Management  Information  Systems  through  Control 
Theory.  Working  Paper  77-5,  Facult6  d'Adminis- 
tration. University of Sherbrooke, Que.bec, Canada. 
10,  RIETMULLER CE  &  SCHREIBER BD  (1977)  Delays  in 
socio-economical systems dynamics models. In Simula- 
tion  '77,  Proceedings  of the  International  Symposium 
(Ed.  HAMZA MH),  Acta  Press,  Anaheim,  California, 
USA. 
11.  SHUBIK M  (1975)  The  Uses  and  Methods  of Gamin 9, 
Elsevier, New York, USA. 
12.  SWANSON CV  (1971)  Evaluating the  Quality  of  Man- 
agement Information. Working Paper, Sloan School of 
Management, Cambridge, Massachusetts. USA. 
13.  TrtlSSEN WAH  (I978)  Investigations into  the  Club  of 
Rome's  World  3  Model;  Lessons  for  Understanding 
Complicated Models.  Doctoral dissertation, Technical 
University, Eindhoven, Netherlands. 
14.  VAN AKEN JE (1978)  On the  Control of Complex  In- 
dustrial  Organizations.  Martinus  Nijhoff  Social 
Sciences Division, Leiden, Netherlands. 
15.  WELKE RJ  (1975)  Information  System  Evaluation--a 
"Non-Bayesian"  Approach.  Faculty  of  Business, 
McMaster University, Hamilton, Ontario. Canada. 
ADDRESS  FOR  CORRESPONDENCE:  Dr  JPC  Kleijnen,  Eco- 
nomische  Faculteit,  Katholieke  Hogeschool,  (Tilburg 
University),  Postbus 90135,  5000  LE  Tilburg,  Nether- 
lands. 
O~F.  7  6-- ] 

(I978) Investigations into the Club of Rome's World 3 Model; Lessons for Understanding Complicated Models. Doctoral dissertation,

An appreciation of industrial dynamics.

Computers and Profizs: Quantifying Financial Benefits of Information.

Economic evaluation of management information systems.

Evaluating the Quality of Management Information. Working Paper, Sloan School of Management,

Information System Evaluation--a &quot;Non-Bayesian&quot; Approach. Faculty of Business,

Modeling and Analysis: an Introduction to System Performance Evaluation Methodology.

On the Control of Complex Industrial Organizations. Martinus Nijhoff Social Sciences Division,

Optimal Design or&quot; Management Information Systems through Control Theory. Working Paper 77-5, Facult6 d'Administration.

Research in management information systems: the Minnesota experiments.

The IPSO control system.

World dynamics: measurement without data.

Evaluation of management information systems.

Evaluation of management information systems.

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics