One approach to reducing software cost and increasing reliability is the use of an independent verification and validation (IV & V) methodology. The Software Engineering Laboratory (SEL) applied the IV & V methodology to two medium-size flight dynamics software development projects. Then, to measure the effectiveness of the IV & V approach, the SEL compared these two projects with two similar past projects, using measures like productivity, reliability, and maintain ablilty. Results indicated that the use of the IV & V methodology did not help the overall process nor improve the product in these cases

Card, David N.

Mcgarry, Frank E.

Page, Gerald

English

NASA Technical Reports Server

A PRACTICAL EXPERIENCE WITH INDEPENDENT VERIFICATION AND VALIDATION 
Gerald Page', Frank E. McGarry.., and David N. Card. 
Computer Sciences Corporation, Silver Spring, Maryland 20910 
National Aeronautics and Space Administration, Greenbelt, Maryland 20771 
ABSTRACT 
One approach to reducing software cost and in- 
creasing reliability is the use of an independent 
Verification and validation (IVLV) methodology. 
The Software Engineering Laboratory (SEL) applied 
the IVLV methodology to two medium-sized flight 
dynamics software development projects. Then, to 
measure the effectiveness of the IVLV approach, 
the SEL colapared these two projects with two simi- 
lar past projects, using measures like productiv- 
ity, reliability, and maintainability. Results 
indicated that the use of the IVLV methodology did 
not help the overall process nor improve the prod- 
uct in these cases. 
Independent verification and validation (IVSV) is 
the systematic evaluation of software by an inde- 
pendent organization, i.e., not the development 
organization. During the software development 
process, the IVLV team-provides an objective ap- 
praisal of the development process and product to 
Detect problems earlier and consequently solve 
them earlier 
Ensure that all defined requirements are 
addressed at each development stage 
Provide managers with better visibility of the 
development process 
These improvements should leaa to more reliable 
-software with better cost and schedule contro1.l 
However, the cost effectiveness of the IVLV meth- 
odology, when compared to more traditional devel- 
opment approaches, has not been fully demonstrated. 
This paper describes an attempt to assess the 
benefits and limitations of the application of 
IVLV in one particular development environment. 
SOPPWARE ENGINEERING LABORATORY 
The environment of the IVLV experiment was the 
National Aeronautics and Space Administration/ 
453 
Goddard Space Flight Center's (NASA/GSFC's) flight 
dynamics area. There, software is developed for 
such spacecraft problems as attitude determination 
and control, mission planning, and maneuver con- 
trol. This software development environment has 
been studied for 7 years by the Software Engineer- 
ing Laboratory* (SEL), a research project spon- 
sored by NASA/GSPC and supported by the Computer 
Sciences Department at the University of Maryland 
and by Computer Sciences Corporation. 
The SEL's goals are to understand and improve the 
overall software development process. 
end, the SEL conducts experiments with.production 
software projects and measures the effect of the 
techniques applied, identifying and adopting bene- 
ficial methodologies for future projects. 
During the past 7 years, the SEL has studied more 
than 45 software development projects totaling 
more than 2 million lines of source code. Most 
flight dynamics projects are developed on a group 
of IBH mainframe computers using FORTRAN and as- 
sembler programming languages. The applications 
projects monitored by the SEL are largely scien- 
tific and mathematical in nature with moderate 
reliability requirements but with severe develop- 
ment time constraints that are imposed by a fixed 
spacecraft launch date. The development process 
typically takes between 18 and 24 months from the 
beginning of preliminary design to the end of 
software acceptance testing. Depending on mission 
characteristics, the size of a system ranges from 
30,000 to 120,000 lines of source code, with an 
average of 30 percent reused from previous similar 
projects . 
A n  IVhV methodology was applied to two typical 
flight dynamics development projects in an attempt 
to determine the benefits of the approach. Each 
project was in development €or 2 years and was 
approximately 65,000 lines of source code in size. 
To this 
THE IVLV METHODOLOGY 
The major functions o€ an IVSV team are (1) to 
ensure that the product of each phase of the soft- 
ware life cycle is consistent with the product of 
the previous phase (i.e., verification) and (2) to 
ensure that the product of each phase accurately 
responds to the original software requirements 
0730-3 l57/ 84/ ooOO/O453SO I .OO @ 1984 IEEE 
3-54 
https://ntrs.nasa.gov/search.jsp?R=19870016347 2020-03-20T10:52:03+00:00Z
(i.e., validation). The benefits of using this 
methodology are claimed to be 
Earlier detection of errors 
Increased software reliability 
Improved software maintainability 
The additional product review activities of the 
IVbV team, especially in the early life cycle 
phases, should produce more complete and earlier 
error detection, as.well as a more easily main- 
tained product. Increased visibility and reduced 
cost are potential secondary benefits. The IVCV 
team provides an independent and impartial assess- 
ment of project status, thus increasing its visi- 
bility to management. Life cycle cost is reduced 
if fewer errors are propagated through to main- 
tenance and operations,. especially in environments 
where maintenance is a large part of the total 
life cycle cost. In addition, the system develop- 
ment cost should be reduced by earlier error de- 
tection and correction. 
A study3 of large TRW, GTE, and IBM software 
development projects suggests that the difficulty 
of correcting an error increases the longer the 
error remains in the system. Figure 1 (from that 
study) shows that the cost of correcting an error 
doubles for each life cycle phase that it remains 
in the system. Data collected.by the Software 
Engineering Laboratory (SEL) support the existence 
of a trend of increasing cost. By finding errors 
soon after they enter the system, the IVbV team 
should reduce the cost of subsequent error cor- 
rection during development. 
IVbV lVbV PAST 
1- 2. 1 
CHARACTERISTICS 
/ 
PAST 
2 
I # 
94 
4.9 
7.2 
5 12 t 0 18 
81 
5.3 
5.9 
/ 
PHASE WWERC ERROR DETECTEO 
Figure 1. Cost of Correcting Software 
Errors (TRW, GTE, IBM Data3) 
THE IVhV EXPERIMENT 
The goal of this study4 was to determine which 
of the potential benefits (previously described) 
would be obtained by applying the IVLV method- 
ology to flight dynamics projects. To p.rform 
this evaluation, the SEL developrd an IVLV,plan 
and applied it to two typical attitude projects. 
The reaults of these projects **..I. rorap.red to the 
results of t m  previous attitude projects (that 
did not use IVbV) with respect to'seven perform- 
ance measures. 
PROJECTS STUDIED 
Each of the experimental projects was initially 
estimated to require 5 to 7 staff-years of effort 
to produce approximately 55,000 lines of source 
code, exclusive of the IVbV effort. The IVhV team 
was expected to expend an additional 15 percent of 
the development team's effort. 
The two past projects with which the IVhV projects 
were compared actually required 9.1 and 9.2 staff- 
years of effort to produce 85,000 and 90,000 lines 
of source code, respectively. These projects were 
completed and operational in 1979. Both IVhV 
projects were completed and operational in 1981. 
All four projects were developed on the same IBM 
mainframe computers in FORTRAN. Table 1 describes 
some other characteristics of the projects. 
Table 1. Characteristics of Projects Studied 
I I PROJECTS 
SIZE ITHOUSANDS OF DELIV- 
ERED UNES OF SOURCE CODE) 
-rr--.. I C - . - -  ..,..l-..C, c r run  I I* I -.Tr-RIuI. I n*r 
DURATION IWEEKS) 
AVERAGE STAFF IFULL-TIME 
EQUIVALENT) 
APPLICATION EXPERIENCE 
(AVERAGE YEARS) 
68 
,,e 2.0 
ea 
5.2 
6.2 
w 
2.2 
70 
6.1 
7 .0  
'EXCLUDES lVbV TEAM 
Data were collected both manually and automatically 
from all four projects during and after develop- 
ment. Errors detected and hours charged were re- 
ported on forms filled out by the development and 
IVCV teams. These data were quality assured and 
stored in a computer data base for easy retrieval. 
FWCI"CTONS OF THE IVbV TEAM 
The SEL developed an IVbV plan based on the rele- 
vant software engineering literature. An IVhV 
team was selected and trained to 
. Validate requirements to ensure completeness 
and correctness 
Verify design to ensure that it is a complete 
and correct translation of the requirements 
Perform independent system testinq to ensure 
that the software satisfies the requirements 
Verify consistency from cequirements to soft- 
ware to operation 
Fix nothinp 
Report all discrepancies and other findings 
The application and overall experience of the IVbV 
technical staff was similar to that of the average 
454 
3-55 
development team: the managers, however, had more 
experience. The IVhV team shared resources with 
developers and operations personnel. 
team was expected to use about 15 percent of the 
development resources to perform these functions. 
The IVhV 
PERFORMANCE MEASURE 
OPERATIONAL RELIABILITY [ERRORS PER 
KSLOCI 
MAINTENANCE COST [STAFF-HOURS PER 
ERROR1 
REOUIREMENTS SPECIFICATION IERRORS 
PER KSLOCI 
SYSTEM DESIGN [ERRORS PER KSLOCI 
ACCEPTANCE TESTING [ERRORS PER KSLOCI 
TESTING EFFORT 1%) 
OEVELOPMENT COST [STAFF-MONTHS PER 
KSLOCI 
PERFORMANCE MEASURES 
lV&V PAST 
PROJECTS PROJECTS 
0.9 0.6 
14 13 
0.8 0.5 
2.1 2.2 
1.9 1 9  
34 38 
2.2 1.4 
One approach6 to assessing the benefit of using 
an IVhV team is to count the number of faults 
found by the team, to determine the probable cost 
of those errors had they become latent errors 
through later :tages of the life cycle, and, from 
this, to compute a dollar benefit of using the 
team. This procedure ignores the possibility that 
some (perhaps all) of the faults found by IVhV 
might have been found by the development team in 
the absence of the IVhV team. 
The approach adopted by the SEL was to define 
relevant performance measures and then to compare 
the overall performance of a development team 
working with an IVhV team to the performance of a 
development team working alone. 
seven measures were defined, covering every phase 
of the software life cycle: 
The following 
Operational Reliability--Errors discovered 
during operation per thousand lines of source 
code developed 
Maintenance Cost--Effort (staff-hours) per 
error corrected during maintenance 
Requirements Specification Quality--Errors per 
thousand lines of source code (found during 
code and test) that were attributed to the 
requirements specification 
System Design Quality--Errors per thousand 
lines of source code (found during code and 
test) that were attributed to the system design 
Software Implementation Quality--Errors per 
thousand lines of source code detected during 
acceptance testing 
Testing Effort--Percent of development effort 
required for system and acceptance testing 
Development Cost--Effort (staff-months) per 
thousand lines of source code developed 
These measures indicate the degree to which earlier 
error detection was achieved, reliability was in- 
creased, and maintainability was improved (the 
purported benefits of IVhV). Values of these 
measures for two IVSV-assisted projects were com- 
pared with values from two similar (but non-IVhV) 
past projects. This enabled the overall effec- 
tiveness of the IVhV methodology to be evaluated 
as well as its effect on each life cycle phase. 
EXPERIMENTAL RESULTS 
The IVhV process for the experimental projects 
lasted 14 to 16 months and required an additional 
effort of 16 to 18 percent of the development 
effort (close to the initial 15 percent esti- 
mate). IVCV staffing averaged 1.1 persons and 
peaked at 3.0 persons (full-time equivalents). 
Six different individuals were involved in the 
IVhV team, including technical managers. 
Both development teams expended 9.8 staff-years Of 
effort. One project produced 66,000 and the other 
67,000 lines of source code. Both size and cost 
substantially exceeded the initial estimates. 
Table 2 shows the values obtained for the seven 
performance measures from the IVhV and past 
projects. 
The following sections discuss the performance 
results obtained from the experiment in more de- 
tail. 
OPERATIONAL RELIABILITY 
The additional testing and verification effort 
provided by the IVhV team should increase the 
operational reliability of the delivered soft- 
ware. However, the error rates achieved by the 
two IVCV projects were greater than the error 
rates for the two past projects. The average 
error rate for the two past projects was 0.6 error 
per thousand lines of developed code: both IVCV 
projects had higher error rates (_an average of 
0.9).  
improve the quality of the software put into 
operation. 
The use of an IVSV methodology did not 
MAINTENANCE COST 
Early detection of errors by the IVCV team should 
reduce the cost of correcting errors during main- 
tenance. That is, relatively few requirements, 
design, or serious coding errors should remain in 
the system when it enters the maintenance and 
operations phase. Those errors found should be 
trivial and few. However, the average effort re- 
quired to correct an error (during maintenance) 
for the IVCV projects (14 hours) was about the 
same as that for the non-IVhV projects (13 hours). 
Given that the operational reliability of the IVCV 
projects was not any better, the relative mainte- 
nance cost of the IVCV projects was greater than 
that of the non-IVCV projects. 
455 
3-56 
REQUIREplENTS SPECIFICATION QUALITY 
One effect of an IVLV team should be to reduce the 
number of requirements errors that are propagated 
through to the coding phase, yet the requirements 
error rates for the IVCV and non-IVLV projects 
were about equal (0.6 and 0.5 error per thousand 
lines of source code, respectively). Rowever, 
very few requirements remained unspecified in the 
later stages of development. Hence, there were 
very few late surprises in teras of requirements 
problems as compared with the past projects. 
use of an IVCV methodology did decrease the impact 
of requirements errors, ambiguities, and misinter- 
pretations. Nevertheless, becau8e requirements 
errors are not a major problem in this environment 
(few requirements errors occur), the effect on the 
overall developent process was minor. 
SYSTEM DESIGN QUALITY 
Another effect of an IVLV team should be to reduce 
the number of design errors that are propagated 
through to the coding phase. 
the design error rate for the IVCV projects (2.1 
errors par thousand source lines of code) was 
approximately equal to that of the past projects 
(2.2 errors per thousand source lines of code). 
The ase of an IVLV methodology did not produce any 
reduction in design errors reaching coding. 
The 
In this experiment, 
SOFIWARE IMPLEMENTATION QUALITY 
The additional system testing and review performed 
by the IVLV team should result in fewer develop- 
ment errors being uncovered during acceptance 
testing. Both IVLV and non-IVCV projects demon- 
strated an acceptance testing error rate of 1.9 
errors per thousand lines of source code. The use 
of an IVLV methodology produced no change in the 
reliability of the software entering acceptance 
testing. 
TESTING EFFORT 
Early error discovery should facilitate system and 
acceptance testing, thereby reducing the effort 
required. Since testing can account for 40 per- 
cent of the development cost in this environment, 
there is the potential for substantial cost sav- 
ings in this area. In this experiment, the IVLV 
projects demonstrated only a marginal reduction in 
the effort required for system and acceptance 
testing relative to the past projects (from 35.9 
to 34.3 percent). 
DEVELOPMENT COST 
It was expected that the cost of using the IVLV 
methodology would be confined to the additional 
cost of owrating the IVLV team. However, the two 
IVcV projects (2.6 staff-months per thousand lines 
of source code) were about 85 percent more 
expensive than the twO past projects (1.4 staff- 
months per thousand lines of source code). Since 
the operational reliability of the software was 
not any better, an 85-peccent increase in cost for 
the same product is a very expensive penalty to 
pay. The cost of the development part of the IVLV 
projects alone (2.2 staff-months per thousand 
lines of source code) was approximately 55 percent 
higher than the past development cost. The cost 
of using IVCV is high. 
CONCLUSIONS AND RECOMMENDATIONS 
In summary, the performance measures indicate that 
the first application of an IVCV methodology in 
the flight dynamics environment 
Did- not produce a more reliable product 
Did not detect errors earlier 
Did not improve maintainability 
The overall conclusion is that IVLV is not cost 
effective for NASA/GSPC's flight dynamics proj- 
ects, especially since other software techniques 
have been shown to improve software qualit 
this environment at little or no net cost. 
Despite these results, it may be possible to 
better integrate the IVLV methodology into the 
software development process to make IVCV more 
cost effective in the flight dynamics environment 
for 
4 in 
The right sire  effort 
The right reliability requirement 
Uost around-based flight dynamics  prnjocts req~ire 
8 2 4 staff-years of effort. 
may be cost effective for larger projects. For 
onboard (flight) systems with a more stringent 
reliability requirement, an IVCV methodology may 
be cost effective for 5- to 6-staff-year efforts. 
In both these cases, an IVcV effort of approxi- 
mately 15 percent of the development effort should 
be sufficient in the flight dynamics environment. 
Software developers should keep in mind, however, 
that no software engineering methodology can re- 
place technical and managerial expertise. It may 
be best to regard IVCV as an insurance policy8: 
an additional premium that should be paid when the 
consequences of failure are great. 
An IVcV methodology 
REFERENCES 
S. B. Mohanty, "On Software Verification 5 
Validation,. Proceedinqs of the Software Veri- 
fication and Validation Symposium, June 1981 
Software Engineering Laboratory, SEL-81-104, 
The Software Enqineerinq Laboratory, 
D. N. Card, F. E. McGarry, G. Page, et al., 
February 1982 
B. W. Boehm, "Software Engineering RCD Trends 
and Defense Needs,' Research Directions in 
Software Technolx. MIT Press: 
Massachusetts. 1979 
Software Engineering Laboratory, SEL-81-110, 
Evaluation of an Independent Verification and 
validation (IVCV) Methodoloqy for Pliqht 
Dvnamics, G. Page and F. E. McGarry, to be 
pub1 ished 
456 
3-57  
5. Michael S. Dcutsch, 'Verification and Valida- 7. D. N. Card, P. E. McGarry, and G. Page, "Eval- 
tion,. Software Enqineerinq. Prentice-Hall, uating Software Engineering Technologies in 
Inc.: New Jersey, 1979 the SEL," Proceedings of the Eighth Annual 
Software Engineering Workshoe, November 1983 
6. M. S. Fujii, "Five Major IVLV Projects: A 
Quantitative Analysis," Proceedings of the 8. J. B. Munson, "Acquisition Management Of 
Software Verification and Validation S m s -  - ium, June 1981 Embedded Computer Systems and the Role Of IVLV," Proceedings of the Software Verifica- 
tion and Validation Symposium, June 1981 
. .  
451 
3-58 


A practical experience with independent verification and validation

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server