Early experiences building a software quality prediction model are discussed. The overall research objective is to establish a capability to project a software system's quality from an analysis of its design. The technical approach is to build multivariate models for estimating reliability and maintainability. Data from 21 Ada subsystems were analyzed to test hypotheses about various design structures leading to failure-prone or unmaintainable systems. Current design variables highlight the interconnectivity and visibility of compilation units. Other model variables provide for the effects of reusability and software changes. Reported results are preliminary because additional project data is being obtained and new hypotheses are being developed and tested. Current multivariate regression models are encouraging, explaining 60 to 80 percent of the variation in error density of the subsystems

Agresti, W. W.

Evanco, W. M.

Smith, M. C.

English

NASA Technical Reports Server

BUILDING A SOFTWARE Q U m  PREDICIION MODEL, 
W. W. Agresti, W. M. Evanco, M. C. Smith 
4 - t,. ' / 
. I : 
f r f ' D  The h m  Corporation l j  \ J ;  
Abstract 
Early experiences building a software quality prediction model are 
discussed. The overall research objective is to establish a capability to project 
a sohware system's quality from an analysis of its design, The tdmica l  
approach is to build multivariate models far estimating reliability and 
maintainability. Data fnm twentyone Ada subsystems have been analyzed 
thus far to test hypotheses about various design structures leading to failure- 
prone or unmaintainable systems Current design variables highlight the 
interconnectivity and visibility cf compilation units. Other model variables 
provide for the effects of reusability and software changes. Reported results 
are preliminary because dditional project data is being obtained and new 
hypotheses dre being developed and tested Current multivariate regression 
models are encouraging, explaining 60-80% of the variation in error density 
of the subsystems. 
Introduction 
A typical shortooming of large-scale software development is the 
uncertainty concerning the consequences of design decisions until much Iater 
in the  development process. Greater capability is needed during the design 
activity to assess the design itself for indications that, when implemented, the 
resulting system will have particular quality characteristics. This paper 
discusses the early experiences in a research project to evaluate the quality of 
Ada designs. 
The research objec!ive is to test the hypothesis that Ada software 
quality factors can be predicted d e g  deign ThepFtuucal approach is build 
[I , . ,  ( :.\.!<: . 8: 
v ,..- 
- This research was sp011sored by The h R E  ~orporadon under the Mission 
Oriented tnves tiga tion and Experhen ta tion (MOIE) program 
- Authors' Address: The h4lTRE Corporatiat, 7525 Colshire Drive, Mdnan, 
Virginia 22102 
Proceedings of the Weenth Annual Software Engineering Workshop, 
National Aeronautics and Space Administration, Goddard Space n'rght Center, 
Greenbelt, Maryland, November 1990 
https://ntrs.nasa.gov/search.jsp?R=19920010187 2020-03-17T12:43:53+00:00Z
multivariate models to estimate reliability and maintainability using 
characteristics of the design. The orientation to Ada is due to its prevalence 
in rnission-critical systems under development and its ability to serve as a 
notation for software design. This role for Ada as a design language has been 
recognized as American National Standards Institute (ANSI)/Institute of 
Electrical and Electronics Engineers (IEEE) Standard 990-1987. 
Previous research has established relationships between design or code 
characteristics and quality factors [I, 21. A recent system-lwel dgign 
measure, incorporating both control flow and data flow in FORTRAN 
systems, shows a strong correlation with reliability (31. The Constructive 
QUAlity Model (COQUAMO! is being developed to estimate software quality, 
basing its estimate on the observed quality of previous projects [4!. 
Qualitv Estimation Models 
Building the estimation models depends on having access to three 
classes of project data. 
- Design, expressed in Ada, from which design characteristics can be 
extracted 
- Environmental factors that influence the quality of the software but 
cannot be deduced from the design artifact itself - for example, level of reuse 
or volatility of changes to the software 
- Data characterizing what resulted when the design was implemented, 
tested, and fielded - for example, reported errors and effort to maintain the 
software 
The basic form of the estimation models is shown in Figure 3. 
Independent, explanatory variables in the models represent architectural 
design characteristics. Additional explanatory variables account for the effects 
of the organization and its development process. By the error term in the 
model, we will learn if we have been successful in explaining the variation in 
quality factors by using design characteristics and environmental factors. 
Ada Design - Representation 
One of the first issues we faced was developing an effective 
representation from which we could extract design characteristics. Our 
interest was in the static architecture of units in subsystems, not in the 
arrangement of statements within a unit. We viewed the subsystem as being 
composed of design units and relations as illustrated in Figure 4- Our analysis 
of Ada identified several candidates to serve as design units in our structure: 
program units, compilation units, and library units. All three units have 
participated in our model building, but compilation units have been 
particularly useful as a structural unit because they also serve as the unit of 
observation for reporting errors and changes. 
Our Ada analysis identified fifteen kinds of Ada compilation units 
generic package speafication, generic padrage body, and so on as shown in 
Figure 5. The compilation units are further divided into library units and 
secondary units (see Figure 5) and serve as the design unit nodes in the 
graphical Ada design structures in Figure 4. The nodes are related to aw 
another by the design relations of context ooupling, sp&cation/body, 
parent/subunit, and generic template/i.nstantiation. These &sign units and 
design relations comprise our representation of static Ada architecture 
Ada design representation is discussed further in [S]. 
Software Proiect Data 
Project data used in the analysis is summarized in Figure 6. The 
twenty-one subsystems included 2,143 compilation units. Dedarations are 
listed in Figure 6 because they play a key role in the hypotheses we are 
examining. One of our underlying themes is that a developer does not 
declare objects, types, subprograms, etc unless they are needed. Thus, the 
number and distribution of these declaraticns is of interest to us in 
characterizing designs. 
Our models attempt to explain variation in quality, and Figure 6 shows 
our project data exhibits significant variation. The data was obtained from 
the National Aeronautics =d Space Administration NASA)/Goddard Space 
Flight Center (GSFC) Software Engineering Laboratory (SEL). Reliability is 
measured as error density and varies in the range 1.4 - 17.0 erron per 
thousand source lines of code for the twezty-onesubsystems. MaintainaEIity 
varied aaoss the subsystems as 26 - 89% error cow-tions requiring less h 
or equal to one hour to complete. 
Hwotheses About Desim - StruW~tre 
We are pursuing simple hypotheses about design decision making, &e 
resulting design artifact, and the influenoes of design on reliability and 
maintainability . Figure 7 o u t h  an example of a general hypothesis tha 
excessive context coupling contributes to errors. The rationale is that greakr 
arc density in the directed graph in Figure 7 increases the likelihood of 
introducing an error, because a greater number of relationships must be 
understood. 
Figure 8 expands on library unit B of Figure 7. We have found that a 
liirary unit aggregation - a library unit and its declarative scope - to be a 
effective unit of granularity for our analysis of Ada designs. Figure 8 shows a 
second level of design decision mairing that occurs inside a library unit 
aggregation. We are interested in whether the designer has macie any effcrt 
to manage the visibility of the 103 declarations that have been imported inao 
unit B. By having 100 of the declarations brought in (via a "with" clause) to 
the specification, they are visible throughout the other units in the library 
unit aggregation, cascading through the structure. We don't know which of 
these declarations are used by each unit, but we want to reaord their visibility 
to the other units in the library unit aggregation. The measure of cascaded 
imports in Figure 8 takes visibility into account 100 of the imports are visible 
to five units ( => 500 cascaded imports) and three of the imports are visible to 
two units ( => 6 cascaded imports), for a total of 506 cascaded imports 
. . 
urunarv Results of Staa- . . 
Figure 9 summarizes the variables that have been introduced into our 
models thus far. Context couphg and visibility follow the example in Figure 
8, while import origin records the fraction of declarations imported from 
within the subsystem. Two environmental factors have been analpxi to 
date: volatility captures the relative number of changes that have been made 
in the subsystem, and custom code is the percentage of new and extensively 
modified code used in the subsystem. Custom code is essentially the opposite 
of reuse. 
The preliminary model explaining variation in total error density 
(Rgure 10) includes the explanatory variables of context coupling, visibility, 
and volatility. In this model and other similar regression analyses we have 
conducted, the coeffiaents have the expected signs: error densities inaease as 
coatext coupling, visibility, and volatility increase. 
Because of our interest in architectural design decisions, we conducted 
additional regression analyses which concentrated on errors occurring during 
system and acceptance testing. Our rationale was that, by eliminating errors 
reported during unit testing (and, therefore, more likely to be errors in 
implementing a single unit), we were reflecting more strongly the 
architectural (inter-unit) relationships. Figure 11 summarizes a model to 
estimate errors reported during system and acceptance testing. Again, context 
coupling and visibility are included as explanatory variables. Now, however, 
custom code is a significant factor in exphhhg the variation in error density- 
The explanatory power (as indicated by the coefficient of determination) is 
stranger for the model in Figure 11. 
Summary 
Early results in building estimation models for reliabsity and 
maintainability are encouraging. We have developed representations for the 
static structure of Ada systems using compilation units and library unit 
aggregations, allowing us to test hypotheses about the effects of different 
structures on reliability and maintainability . C o n k t  coupling measures 
cansistently figure strongly in the multivariate regression analyses we have 
conducted. Visibility and import origin measures provide further 
refinement. The models show strong effects of volatility and custom code on 
reliability . 
We stress the preliminary nature of the quantitative resub, based as 
they are on twenty-one Ada subsystems. We look forward to coatirming to 
explore hypotheses with additional data, leading to the development of mom 
robust models that can be subpded to validation 
We acknowledge the cooperation of Mr. Frank McC;any and Mr. Jar 
Valett of the NASA/GSFC SEL in allowing us to use SEL data for this 
research. 
References 
1. T. J. McCabe and C. W. Butler, "Design Complexity hhsurement and 
Testing," Communications of the ACM, December 1989, pp. 1415-1425. 
2 S. M. Henry and C A Selig, "Predicting S o d o d e  Complexity at the 
Design Stage," IEEE Software, March 1990, pp. 36-44. 
3. D. N. Card and W. W. Agresti, "Measuring Software Design Complexity," 
Journal of Systems and Software, March 1988, pp. 185197. 
4. B. Kitchenham, "Measuring SoftwareQuality," Roedings of the-Fi 
Annual Software Quality Workshop, Rochester, New York, 1989. 
5. W. W. Agresti, W. M Evanco, and M. C. Smith, "An Approach to Software 
Quality Prediction from Ada Designs," Fnxeedings of the Second Annual 
Software Quality Workshop, Rochester, New York, 1990. 
Early Experiences 
Building a Software 
Quality Prediction Model 
Research Project Overview 
a Objecitve: 
- 7- hypomds that M a  aaRwan qusllty tacton can be 
predicted durlng deJgn 
TecbnicalAp9loach: 
- 8ulld fn~lU~arb!8 models to atbnate rdWllW and 
maintslna#ilty 
- ~ ~ e r t r t l c s o f t h r ~ d . l f g n a p l u n d ~ n ~ ~ n  
-!Ww 
Basic Form of the Estimation Models 
R m  .f,Wl ,DC2,-,EF, ,qsuIl l . f  , - I + ?  
W n O m M l l t y ~ t ~ ~ ~ , W ~ , - ~ ~ ~ ~ ~ - I b ~ ~ b ~ ~ - I + @ ~  
wtmm - 
DC, : d o s 4 w - m  
EF, :envtrwnwnOltrdot- 
a , 9 4  
:moddpamwms 
: e m w m  (moxWndvrtltkrr) 
Representing Ada Design Structure 
ADA DESIGN 
DESKiN 
UNITS 
'QIIBCIIDYCII 
"Parts" in Ada Static Structure 
15 Compilatton UniS in Ada 
8s Ubrary Unib (L) or Secondary Units (S) 
Profile of Current Project Data 
a Twenty-one subsystemt from NASAIGSFC SEL. 
- Interactive, ground support sMwam lor flight dynamics and 
t e l e m e v Y w n g  Wkabm 
- 183 K IWrrcomment, rorrbtsnk #urcr Un8S Of Abs (KSLOC) 
- 601 UbraryUlIftS 
Exploring Simple Hypotheses 
About Design Structure 
Inside a Library Unit Aggregation to Show 
Imported and Exported Declarations 
Model Variables 
~ssfgr! Chsraztdatks: 
-  con',^ rap!lr.g: # lmpofts l d q w t a  
- v r ~ ~ b i ~ ~ t y :  ~ ~ ) m p o r t s ~ r ~ m p o r t s  
- Impon Qrlsln: ~ l r n r r n d ~ / # ~  
a EnvlronmantalF&ctais: 
- vom1iny: #&~'~guI#f lbnyunlm 
- CLutmCode: % new and extmslvoty modllkd axjo 
Preliminary Model-for Reliability: 
Total Errors (entot) 
Dependent varlabb: TOTERRSL = efrtoll KSLOC 
In (TOTERRSL) = .65 + 27 In (XI) + .0!5 In [X2j + 27 h (X3) 
(.=I- (-11) (.I 6) (-1 
XI P context COUpllng 
x P  = VISI~ I IR~ R'=.R 
X 3  = volatlllty 
' S t m n d u d d r v l . t b n d t h . p u M w t w ~  
MITE 
Preliminary Model for Reliability: 
Svstem and Acceotance Testina Erron (errsa) 
XI = context coupling 
X 2  = vlslblltty 
x 3 =  customcode 
SmnQrd drvlatkm d the prnrntw m . t l r m t ,  
Current Research Activity 
Contlnue to develop process mod& and rbout dedgn 
decblorrmaklng and dedgn ttnrchrms - and their rebtlonshlp to 
reltablllty and malntalnablllty 
8 Explore classikatlon trees and other alternathre analytlclrl methods 
"CaB for Ada Project Data" - to test hypotheses a d  dlbrate 
mrrlthrarlate models 
VIF4VGRAPH MATERIALS 
FOR THE 
W. AGRESTI PRESENTATION 
Early Experiences 
Building a Software 
Quality Prediction Model 
W. W. Agresti, W. M. Evanco, M. C. Smith 
MITRE Washington Software Engineering Center 
28 November 1990 
MITRE 
Research Project Overview 
Objective: 
- Test the hypothesis that Ada software quality factors can be 
predicted during design 
Technical Approach: 
- Build multivariate models to estimate reliability and 
maintainability 
- Use characteristics of the software design captured in Ada design 
language 
MITRE 
Basic Form of the Estimation Models 
Reliability = f (DC , DC , ... , EF ,EF ,...la , a  ,... ) + e  
1 1 2 1 2 1 2  1 
Maintainability = f (DC , DC , ... , EF ,EF ,...I b , b  ,... ) + e  
2 1 2 1 2 1 2  2 
where - 
DC 
I 
EF i 
: design characteristic variable 
: environmental factor variable 
a, ' 9 : model parameters 
e 
I 
: error term (unexplained variation) 
1 Figure 3 1 
Representing Ada Design Structure 
ADA DESIGN 
DESIGN 
UNITS 
"PARTS" BIN "CONNECTIONS" BIN 
DESIGN 
RELATIONS 
"Parts" in Ada Static Structure 
15 Compilation Units in Ada 
as Library units (L) or Secondary Units (S) 
Specification Body Subunit Instantiation 
MITRE 
I 
Generic 
Package 
Package 
Generic 
Subprogram 
Subprogram 
Task 
L S S L 
L S S N/A 
L S S L 
L L/S S NIA 
N/A N/A S N/ A 
A 
Profile of Current Project Data 
Twenty-one subsystems from NASAIGSFC SEL: 
- Interactive, ground support software for flight dynamics and 
telemetry processing applications 
- 183 K non-comment, non-blank source lines of Ada (KSLOC) 
- 601 Library Units 
- 2,143 Compilation Units 
- 29,849 Declarations 
Variation in dependent variables: 
: ie - Reliability range: 1.4 - 17.0 errorslKSLOC 
a - Maintainability range: 26 - 89 % "easy" fixes (requiring 5 I hour) 
Exploring Simple Hypotheses 
- 
About Design Structure 
1 Example of a general hypothesis: Excessive context coupling 
contributes to complexity which, in turn, contributes to errors 
Example of context coupling to access the resources of library units: 
Notation: 
MITRE 
Library unit 
A imports 
20 declarations Example: 
from B 
with B; 
package A is 
B exports 
20 declarations end A; 
to A 
Inside a Library Unit Aggregation to Show 
lm~orted and Exported Declarations 
A Library Unit Aggregation 
!I{ 8 * number of declarations 
I3 MITRE 
Statlc Measures: 
# imports = 103 
# exports = 20 
# cascaded imports = 506 
Model Variables 
Design Characteristics: 
- Context Coupling: # imports I # exports 
- Visibility: # cascaded imports I # imports 
- Import Origin: # internal imports I # imports 
Environmental Factors: 
- Volatility: # changes I # library units 
- Custom Code: % new and extensively modified code 
Preliminary Model for Reliability: 
Total Errors (errtot) 
0 Dependent variable: TOTERRSL = errtot I KSLOC 
In (TOTERRSL) = .65 + 2 7  In (X ) + -05 In (X ) + -27 In (X ) 
1 2 3 
(.36)* (.11) 6) (=OW 
X =: context coupling 
1 
X P  = visibility 
X , = volatility 
ti 
* Standard deviation of the parameter estimate 
2 
adjusted R = .72 
MITRE 
Preliminary Model for Reliability: 
System and Acceptance Testing Errors (errsa) 
Dependent variable: SYACERRSL = errse I KSLOC 
In (SYACERRSL) = .77 + . I9 In (X ) + .07 In (X ) + -97 In (X ) 
I 2 3 
X = context coupling 
1 
X S  = visibility 
X , = custom code 
ti 
* Standard deviation of the parameter estimate 
2 
adjusted R = -78 
, 4  Ic- 
e 
Current Research Activity 
" , ;' 
I. ,' !., 
. . 
I I '  
, , 
,$, I,\ , Continue to dewelop process models and hypotheses about design 
. i. I 
. . I t  
. .  1, :: decision-making and design structures -- and their relationships to 
: I  
a . I .  :; 
l : \  reliability and maintainability 
' , < '  \ 
I .  . (  
I *I I i,, 
' \. 
Explore classification trees and other alternative analytical methods 
. * 
: ; \ 
: 8 
I (  "Call for Ada Project Dataw -- to test hypotheses and calibrate 
1 :  multivariate models 
MITRE 


Early experiences building a software quality prediction model

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server