Reiter, R. W., Jr.

English

NASA Technical Reports Server

September 18, 1978, SEL Workshop Position Paper 9/30/78 1
INVESTIGATING SOFTWARE DEVELOPMENT APPROACHES: A SYNOPSIS *
Robert W. Reiter, Jr.
Department of Computer Science
University of Maryland
College Park, Maryland 20742
INTRODUCTION
The paper reports on research comparing various approaches, or methodologies, for software devel-
opment. The study focuses on the quantitative analysis of the application of certain methodo-
logies in an experimental environment, in order to further understand their effects and better demon-
strate their advantages in a controlled environment. A series of statistical experiments were conducted
The paper reports on research comparing various approaches, or methodologies, for software devel-
opment. The study focuses ontthe quantitative analysis of the application of certain methodologies
in an experimental environment, in order to further understand their effects and better demonstrate
their advantages in a controlled environment. A series of statistical experiments were conducted,
comparing programming teams which used a disciplined methodology (consisting of top-down
design, process design language usage, structured programming, chief programmer teams, and code
reading) with programming teams and individual programmers which employed their own ad hoc
approach. Specific details of the experimental setting, the investigative approach (used to plan,
execute, and analyze the experiments), and some of the results of the experiments are discussed.
The purpose of the research was to develop an investigative methodology for experimentally
studying and quantitatively characterizing the effect of methodologies and programming environ-
ments on software development. It involves the quantitative measurement and analysis of both
the process and the product of software development, in manner which is minimally obstrusive
(to those developing the software), very objective, and highly automatable. The basic premise is
that distinctions among the groups exist both in the process and in the product.
SPECIFICS
Nineteen units (teams or individuals) each performed the same software development task, but
under controlled and slightly varied conditions. Two programming factors, size of programming
team and degree of methodological discipline, each with two levels (single individual, and three-
person team; the ad hoc approach, and the disciplined methodology), were chosen as the indepen-
dent variables and formed the experimental treatments. The dependent variables to be observed
and measured were a large set (over 125) of programming aspects. The teams and individuals were
*Research supported in part by the Air Force Office of Scientific Research grant AFOSR-77-
3181A to the University of Maryland. Computer time supported in part through the facilities
of the Computer Science Center of the University of Maryland.
©Copyright 1978 by R.W. Reiter, Jr.
.
 J
 Robert W. Reiter, Jr.
University of Maryland
74 Page 1 of 8
https://ntrs.nasa.gov/search.jsp?R=19820068870 2020-03-21T07:01:41+00:00Z
September 18, 1978, SEL Workshop Position Paper 9/30/78 2
placed into three treatment groups, designated A, B and C (of 6, 6 and 7 units, respectively), each
operating under a certain combination of factor-levels:
A — individuals, ad hoc approach;
B — three-person teams, ad hoc approach;
C — three-person teams, disciplined methodology.
The time and place for the experiment was Spring, 1976, in conjunction with two academic courses
at the University of Maryland. The particular project or application to be developed was compiler
for a small high-level language and a simple stack machine. This task was roughly a two man-month
effort, and the resulting software systems averaged about 1200 source lines or 600 executable state-
ments, in high-level structured-language code. The participants were advanced undergraduates and
graduate students in the Computer Science Department. The implementation language was the
high-level structured-programming language SIMPL-T [Basil! and Turner 76], which is used exten-
sivly in course work at the University and has string-processing capabilities similar to PL/1.
Data collection for the experiment was automated on-line, with essentially no interference to the
programmer's normal pattern of actions during computer sessions. Special module compilation and
program execution processors created an historical data base of source code and test data accumu-
lated throughout the project development. Scores corresponding to each of the programming
aspects were extracted directly and algorithmically from this data base.
The programming aspects represent specific automatically isolatable and observable features of the
programming phenomenon, related to either the product or the process of software development.
Product aspects are based on the syntactic content and organization of the symbolic source code
which represents the complete final product developed. Process aspects are related to characteris-
tics of the development process itself, in particular, the cost and required effort as reflected in the
number of computer job steps (or runs) and the amount of textual revision of source code during
development. Major headings for the particular programming aspects reported on in this study are
listed in the accompanying table, with qualifying subcategories mentioned in square brackets.
APPROACH
The investigative methodology was designed and developed as a scientific and empirical solution
to the problem of comparing software development efforts under various conditions. It was used
to guide the planning, execution, and analysis of the set of experiments which comprise this study.
The approach consists of eleven steps or elements, as shown in the accompanying schmatic diagram
which charts the general flow (solid lines) and some of the interrelationships (dashed lines) among
these elements.
Robert W. Reiter, Jr.
University of Maryland
75 Page 2 of 8
September 18, 1978, SEL Workshop Position Paper 9/30/78 3
The methodology begins with Questions of Interest, which are turned into Research Hypotheses and
Statistical Hypotheses. The Statistical Model is very important since it governs the Experimental
Design and several other elements. Statistical Results, corresponding directly to the Statistical Hypo-
theses, are determined by the Colledted Data via the Statistical Test Procedures. Research Frame-
work(s) are necessary to organize the large volume of hypotheses and results into a smaller, more
managable form as Statistical Conclusions and Research Interpretations.
RESULTS
The methodology provides that the study's results be separated into statistical conclusions, represent-
ing factual findings, and research interpretations, representing intuitive judgements.
For each aspect there is one statistical conclusion which states any differences observed among the
three programming environments represented by the groups A, B, and C. These outcomes are ex-
pressed in the form of "equations"; e.g., A<B=C means that the average score for the individual
programmers was appreciably lower than the average scores for the ad hoc teams and the discip-
lined teams which both had about the same average score. In addition to the null outcome (A=B=C)
of no observed differences, there were twelve other possible outcomes, as noted in the accom-
panying table. The table simply lists all the non-null conclusions, arranged by outcome. The values
in the "error" column state the risk, as a probability value, of erroneously making that conclusion
and indicate how strongly pronounced the differences were in the data. Although there is much
fascinating material in these findings, space permits only a few particularly interesting conclusions
to be pointed out.
The A<B=C outcome was quite pronounced for the SEGMENTS aspect, indicating that the indi-
viduals built their systems with fewer routines on the average than either the ad hoc teams or the
disciplined teams, which used about the same number of routines. According to the A<B=C and
B=C<A outcomes, the individuals had noticably less global variables and more local variables than
both types of teams. The C=A<B outcomes for IF statements and DECISIONS indicate aspects
where the disciplined teams behaved like the individuals and both were different than the ad hoc
teams. For the number of COMPUTER RUNS (JOB STEPS), and several sub categories, the C < A
=B outcomes have very low error risks and indicate that the disciplined teams out-performed both
the individuals and the ad hoc teams in these aspects. On the number of PROGRAM CHANGES —
a measure of the amount of cummulative textual revision of the program source code during de-
velopment, which has been shown to correlate well with total error occurrences [Dunsmore and
Gannon 77] — the same data scores which support the C<A<B conclusion at a high risk of error
(0.185) also support the C < A = B conclusion at a very low risk of error (0.004), indicating a strong
distinction in terms of error-prone-ness in favor of the disciplined teams.
One framework for the interpretation of these conclusions is the concept of how the disciplined
methodology actually impacts the software development process and product. Prior to conducting
the experiment, certain general beliefs (see details on accompanying slide) about the impact had
Robert W. Reiter, Jr.
University of Maryland
76 Page 3 of 8
September 18, 1978, SEL Workshop Position Paper 9/30/78 4
been formulated. Certain basic suppositions (a priori expectations), for how the experiments should
turn out if the beliefs were true, were constructed from the general beliefs. Examination of how the
conclusions stack up against the suppositions (how true the beliefs are) shows that none of the con-
clusions for any of the observed programming aspects contravene the basic suppositions. Thus, the
study's results may be interpreted as strong experimental evidence in favor of these general beliefs.
SUMMARY
A practical methodology was designed and developed for experimentally and quantitatively investi-
gating the software development phenomenon. It was employed to compare three particular soft-
ware development environments and to evaluate the relative impact of a particular disciplined
methodology (made up of so-called modern programming practices). The experiments were suc-
cessful in measuring differences among programming environments and the results support the claim
that disciplined methodology effectively improves both the process and product of software develop-
ment. The results will be used to guide further experiments and will act as a basis for analysis of
software development products and processes in the Software Engineering Laboratory at NASA/
GSFC [Easili et al. 77]. The intention is to persue this type of research, especially extending the
study to include more sophisticated and promising programming aspects, such as Halstead's soft-
ware science quantities [Halstead 77] and othersoftware complexity metrics [McCabe 76].
REFERENCES
1. [Basili and Reiter 78] V.R. Basili and R.W. Reiter, Jr. Investigating Software Development
Approaches. Technical Report TR-688, Department of Computer Science, University of
Maryland, August, 1978.
2. [Basili and Turner 76] V.R. Basili and A.J. Turner. SIMPL-T, A Structured Programming
Language. Paladin House Publishers, Geneva, Illinois. 1976.
3. [Basili et al. 77] V.R. Basili, M.V. Zelkowitz, F.E. McGarry, R.W. Reiter, Jr., W.F.
Truszkowski, and D.L. Weiss. The Software Engineering Laboratory. Technical Report
TR-535, Department of Computer Science, University of Maryland. May, 1977.
4. [Dunsmore and Gannon 77] P.E. Dunsmore and J.D. Gannon. Experimental Investigation of
Programming Complexity. Proceedings of ACM-NES Sixteenth Annual Technical Symposium:
Systems and Software (June 1977), Washington, D.C., pp. 117-125.
5. [Halstead 77] M. Halstead. Elements of Software Science. Elsevier Computer Science Library.
1977.
6. [McCabe 76] T.J. McCabe. A Complexity Measure. IEEE Transactions on Software Engi-
neering, Vol. 2, No. 4 (December 1976), pp. 308-320.
Robert W. Reiter, Jr.
University of Maryland
77 page 4 of 8
^oa
.
^^60CEE-M
.
OCu
*************###**#*****
****************************
*
00ZPJH
t/5
-»
-»
OG
.
ont«CDO2Development Pi
STATEMENTERAGE TOKENS PER
>
OMPUTER RUNS (JOB STEPS)CJ
LOCATIONS**•
~
WO<l>CCOA\[compilations, executions, miscf
SSENTIAL RUNS (JOB STEPS)tu
"
 s
 
"
n-intrinsic, intrinsi
(CALLING) SEG1
n-intrinsic, intrinsi
function, procedure; no
G INVOCATIONS PER
function, procedure; no
b__l
 
^
^
 
l_
_
l
^W-1D
w
 Q
D
 S
Q
 
rjj
O
 Z
S
 0
D
H
 
^
-
w
 
z<J
AVERAGE UNIQUE COMPILATIO1
IAX UNIQUE COMPILATIONS FO
ROGRAM CHANGES
**
•
 
•«
 CM
H(CALLED) SEGMG INVOCATIONS PER
function, procedure]
>
 
"
******#***#**#*#:
 *
#
 
<
u
*
 
ex
*
 
«
»
*
 
•*£
*
 
+j
*
 
0
****************************
Final Produ
*
HW
W
 
O
J
 
W
p
 
o
o
Q
 
CKJ
oo
 
O
 
:>>
 W
g
 
2ls
S
 
£
£
3
s
 §
 £fg
§
 
1
 
S|2
0
 
g
 
«
 §
<
U
 
P
L
,
 
<
 
C
 
>
W
 
=T
 W
 ^
 {
*
 ^T
 >->
fe
 ??
 fe
 3
 <
 
.a
 <
O
 
g
O
 
0
 
>
 G
 CQ
u
 
—
„
 u
 i
 
,
 
'-3
 O
,.,00
 
i-,o
o
 i-iH
 o
_j
.
 
.
w
w
^
w
^
<
 E
O
^
_)
 J
 
g
J
 
g
W
^
^
 
o
CQCQ
 SCQ
 §9
 
9
o
 
—
^
^
 S3
 Id^i
 s
-
2g«
 
^5
 
^
«1«
^
>
>
^
>
!3|3
l
<
<
^
<
^
D
i
 
gftJ
 c3
H
H
 
s
o
H
 M
W
 
e
w
 
e
x
^
^
 
^f
 
^
^
 '
 
"
^
 
'"
"
'
 
^
^
 
1
~
~
'
 ^
^
 ^
"
"^
Q
 Q
 
Q
 
<
 
<
D
1ODULES
EGMENTS
EGMENT TYPE COUNTS
[function, procedure] :
SEGMENT TYPE PERCENTAGES
[function, procedure]
AVERAGE SEGMENTS PER MOD
LINES
•a
 
oo
 oo
00O^
 
'&
««•'&
g
 
oo
 
-S
 oi
 
-g
w
 
£
^
 «
 3
 
u
O
 
•<
 
iTC
u
 bT
C
*
 
P
H
 &
m
 
&
S
 
g
||l
s
 
<
 1
^
 §
£
 
3
«
^
«
ElAMETER PASSAGE 1
value, reference]
G,GLOBAL) ACTUAL
modified, not modified;
G,GLOBAL) POSSIBLE
modified, not modified ;
^
~J
 i—
i
 
prj
 
t
—
 '
 |"Tj
 
"
—
i
S
 
52
-
 52
-
i
—
 ,
i
—
 i
 
|2
5
 
2
OH
 
p
P
 
H
W
 
OS7
H
-f
 
3
<
 
^
 
^
't?
 
O
 W
 
m
Q
 
l-(
 
^J
 
^
0<
m
 
^
O
 
n
STATEMENTS
STATEMENT TYPE COUNTS
[:=, IF, CASE.WHILE, EXIT, 1
STATEMENT TYPE PERCENTAG]
[:=, IF, .CASE, WHILE, .EXIT,
AVERAGE STATEMENTS PER SE
AVERAGE STATEMENT NESTIN
;NTAGESELATIVE PERCE
; non-entry, entry]
G,GLOBAL) USAGE R
modified, not modified;
["T
 j
 
<
—
 
^
00
DECISIONS
\ BINDINGSG.GLOBAL.SEG) DAT,woo^FUNCTION CALLS
|: ***********
percentage]
l:*************^
actual; possible; relative
:******************;
i
—
 i
 
*#*
[non-intrinsic, intrinsic]
78
Robert
 W
.
 Reiter
,
 Jr
.
University
 of
 M
aryland
Page
 5
 of
 8
COa
n§
82V) Q
.
CO
 
f
-
t
.
 0
>
-
*
 
^
-
 
^
*
.C
 
1
-
 
^
o
 
o
 
-
*
^
.
 
t
-
 3
 
-
-
o>
 
^
 
^
 §
 
~
~
"
 
-
»
.
tfl
 
CO
 
CO
 
»
•
0
 
<
0
 I
-
 
-^
0
 
-H
 
t
-
 
*
-
•
 
•
 
~
~
^
 
x
r
f
 
~
~
o
 x
 t
f
 
,
X
 
/
x
X
X
 
/
x
 
/
'
 
/
X
X
 
/
X
 
/
^
 
•
 
"
-
 
-
•
 
'
 
!
0)5f
*
\
 
•
o
.•
x
 
-
 
/
 
-
-U
 
•
'
 
^H
to
 
n
 
0}
 
co
 co
C
 
Hi
 
.C
 
CO
 
O
 CO
O
 
L
 
O
 
C
O
 
-rt
 
C
O
 
•
-
 
t-
«
 a
>
 
.
C
e
o
 
_
4j<
u
M
 
^
 
if)
 
"C
 
'
'c
O
o
J
 
^
^
-
J
4
J
V
 
0
)
 
'
r
>
 
c
o
-
^
io)
 
c
o
o
o
)
 
J
-
>
O
N
-
 
•?
a
 
3
v
>
a
>
o
.
i
J
c
a
Q
.
x
 
<
A
 
-
2
*
 
_
 
0
-Q
-
,
 
<
4
 
!
.
>
,
<
»
 
j
_
>
>
,
\
.
 
0
 
S
p
7
7
o
\
<
*
 
£
^
*
 
c
o
.C
 
<
n
 
x
^
A
poV
10I^Jc
-
-^
r
-l
 
C
O
3S
"•-t
 
«f-l
X
 
4J
 C
O
^
 
m
 
^
'
 
•
•
-
*
 
«
-H
4
J
 
0
CO
 
C
4
J
 
O
0)
 
O
A
Q
)^CCOCDVO
.
-10
 C
O
-H
 4
J
4J
 
^H
^
 
m
 2
T
-l
 
C
O
jj
 C
O
7
 s
 <
-
CO
.•"•••••
o
•
 
-r-i
-P
.
 
<
U
•
 
,9CO
"
 
.C
•
 
o
•
 e. a
.
a
.
•
 
^
(!^
0)a)v
\
 
'
\
 
'
\
 
'
\
 
I
\
 
'
\
 
I
3j_>
 
a>
to
 
-o
-
tla•o
•a*i•^i
CO£
A
a)•O
C?
,
i
 .;
a)O
BO(TJ
CO1—I
CO
i
 
.a
-3
 
x
Q
-
 
•
 C
d
79
R
obert
 W
.
 R
eiter
,
 Jr
.
U
niversity
 of
 M
aryland
Page
 6
 of
 8
Non-Null Conclusions, arranged by outcome
outcome
A < B = C
B = C < A
B < C = A
C = A < B
C < A = B
A = B < C
error fVeq
9
0.0634
0.0698
0.1476
0.1614
0.2015
0.1271
0.15070.1748
0.1227
5
0.1706
0.1699
0.1699
0.1936
0.1090
3
0.2195
0.2364
0.1546
11
0.2134
0.2321
0.0780
0.1732
0.0196
0.1038
0.2065
0.1468
0.1732
0.0435
0.1861
8
0.0036
0.0223
0.0110
0.0221
0.1445
0.0037
0.0883
0.1180
0
programming aspect
SEGMENTS
DATA VARIABLES
DATA VARIABLE- SCOPE .COUNTS Y GLOBAL
DATA VARIABLE SCOPE COUNTS \ GLOBAL \ MODIFIED
DATA VARIABLE SCOPE COUNTS \ NON-GLOBAL
DATA VARIABLE SCOPE COUNTS A NON -GLOB ALA PARAMETER
DATA VARIABLE SCOPE PERCENTAGES \ NON-GLOBAL \ PARAMETER
AVERAGE NON-GLOBAL VARIABLES PER SEGMENT \ PARAMETER
(SEG, GLOBAL) POSSIBLE USAGE PAIRS
AVERAGE STATEMENTS PER SEGMENT
AVG INVOCATIONS PER (CALLING) SEGMENT \ NON-INTRINSIC
AVG INVOCATIONS PER (CALLED) SEGMENT
AVG INVOCATIONS PER (CALLED) SEGMENT \ FUNCTION
DATA VARIABLE SCOPE PERCENTAGES \ NON-GLOBAL \ LOCAL
STATEMENT TYPE PERCENTAGES \ CASE
(SEG, GLOBAL) USAGE RELATIVE PERCENTAGES
(SEG, GLOBAL) USAGE RELATIVE PERCENTAGES \ NOT MODIFIED \ NON-ENTRYi
SEGMENT TYPE COUNTS \ FUNCTION
STATEMENTS
STATEMENT TYPE COUNTS \ IF
STATEMENT TYPE COUNTS \ (PROC)CALL \ INTRINSIC
STATEMENT TYPE COUNTS \ RETURN ,
STATEMENT TYPE PERCENTAGES \ IF
STATEMENT TYPE PERCENTAGES \ RETURN
DECISIONS
INVOCATIONS \ PROCEDURE \ INTRINSIC
INVOCATIONS \ INTRINSIC
(SEG .GLOBAL, SEG) DATA BINDINGS \ POSSIBLE
COMPUTER RUNS (JOB STEPS)
COMPUTER RUNS (JOB STEPS) \ MODULE COMPILATIONS
COMPUTER RUNS (JOB STEPS) \ MODULE COMPILATIONS \ UNIQUE
COMPUTER RUNS (JOB STEPS) \ PROGRAM EXECUTIONS
COMPUTER RUNS (JOB STEPS) \ MISCELLANEOUS
ESSENTIAL RUNS (JOB STEPS)
AVERAGE UNIQUE COMPILATIONS PER MODULE
MAX UNIQUE COMPILATIONS FOR ANY ONE MODULE
A < B < C 0
A < C < B . 1 - " '
0.1194 LINES
B < C < A 2
0.1232 (SEG,GLOBAL) USAGE RELATIVE PERCENTAGES \ MODIFIED \ ENTRY
0.1173 (SEG,GLOBAL) USAGE RELATIVE PERCENTAGES \ ENTRY
B < A < C 0
C < A < B 1
0.1848 PROGRAM CHANGES
C < B < A 0
Robert W. Reiter, Jr.
University of Maryland
80 Page 7 of 8
Research Interpretations
General Beliefs:
— The disciplined methodology reduces the average cost and complexity of the process.
— The disciplined methodology can enable a programming team to compensate for their in-
herent coordination overhead and behave more like an individual programmer in terms of
designing and building the product.
Basic Suppositions:
— on process aspects: C < A , B
— on product aspects: A < C < B o r B < C < A
Support from the conclusions:
— process: C < A = B on 8 aspects
C < A < B on 1 aspect
A = B = C on 1 aspect
product: A < B = C on 9 aspects
A = C < A on 5 aspects
B < C = A on 3 aspects
C = A < B on 11 aspects
A < C < B on 1 aspect
B < C < A on 2 aspects
A = B = C on 96 aspects
None of the conclusions for any of the observed programming aspects contravene these basic
suppositions.
Thus, the study's results may be interpreted as strong experimental evidence in favor of these
general beliefs.
Robert W. Reiter, Jr.
Univeisity of Maryland
81 Page 8 of 8


Investigating software development approaches: A synopsis

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server