A viewgraph presentation describing the NASA Software Assurance Research Program (SARP) project, with a focus on full life-cycle defect management, is provided. The topics include: defect classification, data set and algorithm mapping, inspection guidelines, and tool support

Bechtel, Andre

Feldmann, Raimund L.

Godfrey, Sally

Regardie, Myrna

Seaman, Carolyn

Shull, Forrest

NASA Technical Reports Server

09/2008 Technical Presentation  1SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Dr. Forrest Shull (PI)
Ms. Sally Godfrey (NASA POC)
Mr. Andre Bechtel 
Mr. Raimund L. Feldmann 
Ms. Myrna Regardie 
Dr. Carolyn Seaman
Making Use of a Decade of Widely 
Varying Historical Data
SARP project
“Full Life-cycle Defect Management”
https://ntrs.nasa.gov/search.jsp?R=20080044833 2019-08-30T05:36:21+00:00Z
09/2008 Technical Presentation  2SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
 We are in the second year of our initiative and studying
¾ Parameters that affect the results of inspection
¾ The relation between V&V effectiveness in early lifecycle (e.g., 
inspection) and late (testing)
 We are using this information to provide feedback and decision 
support to NASA projects, on questions such as:
¾ Can I get guidance on how to plan 
my inspections based on results
from projects like my own?
¾ Based on my inspection results,
what are the implications for the effort
required to be spent on other non-optional
activities, like system testing?
Problem we are addressing
09/2008 Technical Presentation  3SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Our approach
Literature Recommendations
Historical Baseline Models
Users:
Projects
SEPG
Inspection Planners
Researchers
Current Model Formulation
Outputs:
Automated feedback
What if Analysis
Experience Bases
Trends
…
Inputs from:
GSFC
GRC
JPL
JSC
MSFC
More to come…
09/2008 Technical Presentation  4SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
First year results
 Collected more than 2,529 inspection records in our database
¾ Evaluated old classification schema 
¾ Developed new classification based on existing standards and the 
collected data
¾ Mapped data into new classification schema
 Developed prototype tool to support planning and reporting
¾ Incorporated latest analyses and models based on the data
¾ Designed capabilities for accepting data from various forms (e.g., 
JPL forms) as well as various databases 
¾ Gained feedback on usability and possible enhancements
 Created central inspection experience base
¾ Provides materials necessary for applying inspections in various
contexts: e.g., defect type definitions, mapping to various 
taxonomies, checklists, forms, …
09/2008 Technical Presentation  5SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Unifying different defect classifications
 Motivation: Valuable defect data has been collected over the 
years across many Centers and projects
 Issue: Different defect classifications used in historic and 
contemporary data sets, as well as across and within Centers
 Action: Define a unified defect classification schema along with 
a mapping to existing data sets
 Benefits:
¾ Leverages data required by NPR 7150.2 for analysis and feedback 
to teams
¾ Enables monitoring and validation of existing guidelines
¾ Unified classification schema is applicable to inspections and testing
09/2008 Technical Presentation  6SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Mapping the different data sets
historic data sets contemporary dataactions
A7
cat.
1
cat.
2
cat.
3
cat.
4
A B Y Z
A B’ W Y’ Z
A C W’ Y Z
Select candidate defect categorization scheme 
(e.g., ODC)
ODC
cat.
X
cat.
Y
cat.
Z
Analyze historical categories; 
(e.g., keep A & Z; combine Y&Y’; exclude C; partition all others)
ODC
cat.
X
cat.
Y
cat.
Z
A B Y Z
A B’ W Y’ Z
A C W’ Y Z
initial ODC-based new schema
Define initial new categorization schema
(i.e., mix of historic and common categorization schema) cat.
X’
cat.
Y
cat.
Z
cat.
A
A B Y Z
A B’ W Y’ Z
A C W’ Y Z
initial ODC-based new schema
Map historical data to new categorization, for categories 
that exist in both.
cat.
X’
cat.
Y
cat.
Z
cat.
A
A B Y Z
A B’ W Y’ Z
A C W’ Y Z
Partition remaining historical data set categories; 
refine new schema if needed
ODC-based new schema
cat.
X’
cat.
Y
cat.
A
A B Y Z
A B’ W Y’ Z
A C W’ Y Z
cat.
Z’
Review new categorization and mapping
09/2008 Technical Presentation  7SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Mapping algorithm
Any 
historical
datasets 
left?
end
begin
no Any defectcategories 
left in X?
Choose 
historical 
dataset X
no
Choose defect 
category CX
Remove category CX
Assign all historical 
defects in CX into this 
category
Use the assignment algorithm 
to distribute defects in CX
Add CX to S, 
along with all 
its defects.
Is CX
already in S? 
(under 
another 
name)?
yes
yes
yes
Can all 
defects in CX 
be included 
under any 
single existing 
category in 
S?
Assign all historical 
defects in CX into 
this category
yes
Can all 
defects in CX 
be included 
under multiple 
(existing) 
categories in 
S?
yes
nono
Remove X from historical datasets
Select new 
categorization 
schema S 
(e.g., ODC)
no
09/2008 Technical Presentation  8SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Updating existing inspection guidelines
 Motivation: NASA guidelines for effective inspections 
(e.g., 3 points of control) were formulated in early 1990’s
 Issue: Development procedures (e.g., standards, languages, 
etc.) have changed over time; 
Æ New factors must be considered
 Action:
¾ Validate guidelines based on a wider set of recent data; 
¾ Refine the guidelines if needed (e.g., by adding more variables, 
tailoring to different domains, etc.)
¾ Integrate them into an inspection support tool and training courses
 Benefits: Refined guidelines will increase effectiveness of 
inspections and provide better user guidance
09/2008 Technical Presentation  9SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
User guidance based on heuristics
optimum
acceptable border
line
problematicborder
line
problematic
defines
 



 
09/2008 Technical Presentation  10SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Average Team Size
0
1
2
3
4
5
6
7
8
9
requirements design source code test other
Target team size: optimal is 4 to 6; borderline is 3 or 7
Example: Comparison of team size
09/2008 Technical Presentation  11SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Comparing test and inspection data
 Motivation: Better knowledge of inspection’s strengths & weaknesses 
could be used to better allocate resources among V&V activities.
 Issue: Defects that slip through inspections aren’t found until much later; 
different defect type descriptors mean they often are hard to compare.
 Action: Compare test and inspection defect profiles (on the same 
projects or within the same domain)
 Benefits: Past knowledge about recurring defect types can be used to 
select the right overall strategy for optimal V&V planning
Research Questions:
 What defects types are typically removed by inspections vs. testing?
 What project characteristics (size, language, software domain, new 
development/enhancements) influence the types of defects found?
 What percent of logic errors can be expected to be removed by 
inspections?
 Can test results be used for post-mortem analysis of
inspection performance?
09/2008 Technical Presentation  12SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Overview: Inspections vs. testing
09/2008 Technical Presentation  13SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Initial results: Across projects
Research Question: What defect types are typically removed by 
inspections vs. testing? In this domain:
Æ Overall the defect removal profile seems similar, but
Æ Inspections found on average 64% of the total system defects
09/2008 Technical Presentation  14SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Initial results: Within a project
Research Question: What defect types are typically removed by 
inspections vs. testing? Specifically, for a maintenance project: 
Æ Many more internal interface defects were found by inspections
09/2008 Technical Presentation  15SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Improving tool support
 Motivation: Data and resources from across NASA, that use 
different taxonomies, cannot easily be leveraged without 
centralized tool support.
 Issue: Need to do mappings and analysis without requiring extra 
steps from the user, and to seamlessly integrate the results.
 Action: 
¾ Centralize existing materials and resources Æ Experience Base; 
¾ Integrate Experience Base and results data into a combined 
dashboard
 Benefits: Integrating real-time feedback into normal engineering 
activities, for:
¾ The planning of inspections, 
¾ Collection of data, 
¾ Analysis and building of up-to-date baselines, 
¾ Feedback and improvement.
09/2008 Technical Presentation  16SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Providing an inspection experience base
http://fc-md.umd.edu/EB/
09/2008 Technical Presentation  17SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Improving tool support for inspections
built-in 
experience 
base 
access
reporting 
and 
evaluation 
capabilities
file import 
capability
7
09/2008 Technical Presentation  18SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Future work
 Refine the test and inspection data comparison
¾ Obtain additional data sets for testing and refining our preliminary conclusions 
¾ Integrate test results into inspection tool
 Initial deployment of tool 
¾ Obtain additional feedback on usability and future deployment
¾ Pursue expansion of the Experience Base with testing-related materials
Æ a centralized site for V&V resources
 Integrating with other existing inspection data forms and tool 
support
¾ Especially eRoom-based tool available through Kevin Carmichael / GRC
09/2008 Technical Presentation  19SAS_08_Full Life-cycle Defect Management_inspections & tests_Shull© 2008 Fraunhofer USA Inc.
Inspection Experience Base on-line at: 
http://fc-md.umd.edu/EB
Forrest Shull (PI)
fshull@fc-md.umd.edu
301-403-8970
Contact information
Sally Godfrey (NASA POC)
sara.godfrey@nasa.gov
301-286-5706
Myrna Regardie
mregardie@fc-md.umd.edu
301-403-2050


Making Use of a Decade of Widely Varying Historical Data: SARP Project - "Full Life-Cycle Defect Management"

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server