A long-term program is in progress at JPL to reduce cost and risk of flight mission operations through a defect prevention/error management program. The main thrust of this program is to create an environment in which the performance of the total system, both the human operator and the computer system, is optimized. To this end, 1580 Incident Surprise Anomaly reports (ISA's) from 1977-1991 were analyzed from the Voyager and Magellan projects. A Pareto analysis revealed that 38 percent of the errors were classified as human errors. A preliminary cluster analysis based on the Magellan human errors (204 ISA's) is presented here. The resulting clusters described the underlying relationships among the ISA's. Initial models of human error in flight mission operations are presented. Next, the Voyager ISA's will be scored and included in the analysis. Eventually, these relationships will be used to derive a theoretically motivated and empirically validated model of human error in flight mission operations. Ultimately, this analysis will be used to make continuous process improvements continuous process improvements to end-user applications and training requirements. This Total Quality Management approach will enable the management and prevention of errors in the future

Barnes, G. Michael

Bruno, Kristin J.

Sherif, Josef

Welz, Linda L.

English

NASA Technical Reports Server

N94-11534
ANALYZING HUMAN ERRORS IN FLIGHT MISSION
OPERATIONS
Kristin J. Bruno
Linda L. Welz G. Michael Barnes Josef Sherif
Jet Propulsion Laboratory,
California Institute of Technology
4800 Oak Grove Drive
M/S 125-233
Pasadena, California 91101
(818) 354-7891
email: kbruno@ spa1.jpl.nasa.gov
Computer Science Department (COMS)
California State University
Northridge, California 91324
(818) 885-2299
email: renzx_ms.secs.csun.edu
Jet Propulsion Laboratory,
California Institute of Technology
4800 Oak Grove Drive
M/S 125-233
Pasadena, California 91101
(818) 354-8365
Abstract
A long-term program is in progress at JPL to reduce cost and risk of flight mission operations through a defect
prevention/error management program. The main thrust of this program is to create an environment in which the
performance of the total system, both the human operator and the computer system, is optimized. To this end, 1580
Incident Surprise Anomaly reports (ISAs) from 1977-1991 were analyzed from the Voyager and Magellan projects.
A Pareto analysis revealed that 38% of the errors were classified as human errors. A preliminary cluster analysis
based on the Magellan human errors (204 IS&s) is presented here. The resulting clusters described the underlying
relationships among the ISAs. Initial models of human error in flight mission operations are presented. Next, the
Voyager ISAs will be scored and included in the analysis. Eventually, these relationships will be used to derive a
theoretically motivated and empirically validated model of human error in flight mission operations. Ultimately,
this analysis will be used to make continuous process improvements to end-user applications and training
requirements. This Total Quality Management approach will enable the management and prevention of errors in the
future.
Introduction
A long-term program is in progress at JPL to reduce
cost and risk of flight mission operations through a
defect prevention/error management program. Flight
mission operations require systems that place human
operators in a demanding, high risk environment. This
applies not only to the mission controllers in the "dark
room", but also to the mission planners and flight
teams developing sequences, to the Deep Space
Network (DSN) operators configuring and monitoring
the DSN, and to the engineering teams who must
analyze spacecraft performance. This environment
generally requires operators to make rapid, critical
decisions and solve problems based on limited
information, while following standard procedures
closely. The mission operations environment is,
therefore, inherently risky because each decision that a
human operator makes is potentially mission critical,
and in a high-demand environment, human errors occur
frequently. Given the high risk in such an
environment, these human errors Can have grave
financial (e.g., the Soviet loss of PHOBOS) or loss-of-
life (in manned space fligh0 consequences.
To contain this risk at J'PL, flight mission operations
procedures include intensive human reviews. In
addition, when an error does occur, rapid rework is
499
required to ensure mission success. This strategy has
worked well to reduce risk and has ensured the success
of JPL missions. However, the large human labor
investment in these reviews and rework has contributed
substantially to the cost of flight mission operations.
Prevention of such errors would reduce both cost and
risk of flight projects. The motivation of this program
is that risk can be contained more cost effectively by
preventing human errors rather than reworking them.
The goal of this program is the management, reduction
and prevention of errors. The key facet of this program
is to create an environment in whLch the performance
of both the human operator and the computer system is
optimized. Systems must be designed to enhance
normal human performance (e.g., as described in Card,
Moran, & Newell, 1983); training programs must be
designed to alleviate likely errors; and functions that are
human-error prone should be automated. Thus, to
design and implement a successful defect/error
prevention program requires a theoretically motivated
model of human problem solving and decision making
based on current theoriesofknowledge representation,
the structure of memory, schemas, and mental models
(e.g., Anderson & Bower, 1973; Norman, 1988).
further, such a model must be data-validated to ensure
its ultimate applicability to the flight mission
operations environment. Principles of cognitive
psychology, human-computer interaction, and Total
Quality Management (TQM) are used to analyze past
https://ntrs.nasa.gov/search.jsp?R=19940007062 2020-06-16T20:46:43+00:00Z
errorsand make changes to end-user applications and
training requirements and task policies and procedures
to prevent or manage these errors in the future.
Method and Results
The process developed for this program can be viewed
as a continuos process improvement loop consisting of
five steps:
1. Institute the Mission Operations and Command
Assurance (M0&CA) function on JPL flight
projects.
2. Analyze Incident Surprise Anomaly (ISA) data for
causes of errors and patterns of causes.
3. Develop a prototype of a human process model of
the underlying factors causing cognitive errors
during flight operations based on the ISA dam.
4. Develop a defect prevention/error management
methodology based on the flight operations human
process model.
5. Insert the methodology into Flight Mission
Operations system development and training via
system requirements and training prototypes and
into policies and procedures via MO&CA.
Thus far in the program Step 1 has been successfully
completed. MO&CA teams have been installed on
flight projects to help _UCe cost and risk. The:main
benefits of these teams are realized from collecting and
analyzing error data in the form of ISA Reports. Based
on these reports MO&CA teams make
recommendations for subsequent changes to flight
operations procedures, and work with the flight
mission operations teams to incorporate the
recommendations. The work of these teams is ongoing
on several projects. The current work, reported in this
paper, consists of extended analysis of error data (IS.M)
to determine patterns of causes and develop a prototype
human process model (Steps 2 and 3). Currently,
error data with cause codes is available for three flight
projects over a 14 year period.
The goal of Step 2 was to reduce the data to a
meaningful subset of the most frequent causes of errors
based on the TQM principle of investigating the most
prevalent problems first in a defect prevention/error
management program. ISA reports from two projects,
Voyager and Magellan, were classified in one of 12
cause code categories, Each project used a slightly
different taxonomy of detailed cause categories within
the high-level cause. Thus, the detailed analysis
entailed developing a composite cause category
taxonomy of dam for both projects making die detailed
cause category analysis equivalent. The categories used
were developed by MO&CA teams based on major
functions in the flight mission operations
environment. An early Pareto analysis was performed
to determine the most frequent high-level causes of
errors. The analysis showed that, of the 1580 ISA
reports recorded, the three cause categories of Human
(38%), Software (20%), and Documentation (10%)
accounted for 68% of the errors ('Figure 1).
%
40.
O
f 30.
T
O
t 20.
It
I
I I0,
S
A
S
0 ,
g
it|
S_- Software
Docum - Decumentatlon
Human. Human
Equip- Equipment
Schedul -Schedule
S/C - Spacecraft
Revw - Review Process
Forms -Forms
proJ - project Policy
Approl - Approval Process
Product - S/W or Data Product
Unkwn - Unknown
ISA Cause Category - Voyager (1977-1989) & Magellan (1989-1991) [
Figure 1
Voyager and Magellan ISAs - By Cause Code
500
Based on the high number of human errors, subsequent
analysis of more specific cause codes was restricted to
ISA reports for the Magellan proj_t t_t _wereclassified
as Human Error. The taxonomy of cause codes was
then used to score the 204 Magellan Human ISAs.
Each ISA was read by a team of 2 investigators who
assigned as many cause codes as was appropriate. In
addition, each cause code was assigned a value of 0, 1,
or 2. These values were assigned as follows: "2" was
assigned if this cause alone caused the anomaly to occur
and the ISA to be written; a "1" was assigned if this
cause was an ancillary cause which contributed to the
anomaly, but would not by itself have caused it; a "0"
was assigned if this cause did not apply to the ISA.
Thus, for the 204 Magellan Human Error ISAs
examined, 269 cause codes were assigned.
Next, the Magellan human error data was subjected to a
cluster analysis to identify clusters of cause code
patterns. Interpretation of these clusters was expected
to reveal the underlying factors causing cognitive
errors. BMDP's cluster analysis, a multidimensional
scaling technique, was used. The program groups the
pair of cases (in this case ISAs), with the shortest
Euclidean distance (the square root of the sum of
squares of the difference between the values of the
variables for two cases). In a step-wise manner, two
cases or clus_rs are grouped such that initially each
case is an individual cluster and at the end all cases are
in one cluster. In the present analysis, 25 clusters Were
formed first at distance 0. Thus the internal distance
among ISAs in each of those 25 clusters was 0; that is,
the ISAs were scored identically. Figure 2 shows a
Pareto chart of the size of the first 25 clusters. The 4
largest clusters contained 25, 21, 19, and 15 ISAs
respectively, followed by a gap. The next cluster, of
size 9, was the cluster of ISAs of unknown cause.
Thus, only the 4 largest clusters were selected for
interpretation. These 4 clusters consisted of ISAs with
only one cause rated "2". They were Oversight (12%),
Lack of Communication (10%), Edit Error in Product
(9%), and Omission of Action (7%), respectively, and
accounted for 39% of the 204 ISAs (Figure 3). At the
next major level of clustering, distance 3.3, these 4
clusters joined, along with others, to account for 52%
of the total ISAs. Finally, at the third major level of
clustering, distance 6.6,95% of the ISAs joined. The
final 5% of the ISAs did not join until distance 14.8
and these errorswere rare, dissimilar cases.
Size of the Initial 25 Clusters
(Number of ISAs per
Cluster)
9.0
O
t-
@
O"
tL
8.0
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
A.
3 5 7 9 11 13 15 17 19 21 23 25
Cluster Size
Figure 2
Magellan Human Error ISAs - By Cause Code
501
Figure3
MagellanHumanErrorISA Cluster Analysis
In order to model the underlying causes of the ISAs
within the four initial clusters, each ISA within a
cluster was reexamined for its specific characteristics.
Characteristics common to all ISAs in the cluster were
compared to known cognitive phenomena, particularly
with human error theory. Several taxonomies of
human error have been proposed (e.g., Norman, 1981,
1983; Reason, 1990). However, there is no general
agreement on a single taxonomy. Thus, it has been
suggested that a taxonomy must be tailored to a given
environment (Senders & Moray, 1991). The taxonomy
adopted here, in Appendix A, is tailored and simplified
from Reason (1990). Figure 4 shows the cognitive
mechanisms in the taxonomy. Tasks are divided into a
planning and an execution component. The error types
are Oversight (generally known as a slip), Omission of
Action (generally known as a lapse), a mistake, and a
violation. This general taxonomy was then used to
model the common characteristics within each duster.
The four highest frequency clusters that joined at
distance 3.3 exhibited at least two common cognitive
elements, omission of action and oversight. In
omission of action, a goal was acquired, but a subgoal
was not executed for some reason, typically cognitive
capture or a distraction. Cognitive capture generally
refers to a psychological phenomenon in which a well-
rehearsed action takes control of a less familiar action.
This is particularly true when attention is drawn
elsewhere. For example, one ISA (8508) documenvxt a
ease in which DSN station operators did not notice for
three days a special condition in the Sequence of Events
(SOE) during Magellan support. The problem was
caused by the fact that Magellan support had become
routine and the SOE rarely changed. Thus, this routine
support "captured"the processing of the changed SOE
so that some new steps were omitted. This error
502
Error Type.
Oversight
(slip)
Omission of
Action (lapse)
Planning
OK
OK
Task Type
Execution
Followed plan, but performed
a wrong action
Followed plan, but skipped
an action
Mistake
Violation
Faulty Plan
OK
Followed faulty plan
Intentionally deviated fi'om plan
Figure 4
, Human Error
resulted in a loss of data. The second common
cognitive element in this cluster was oversight, in
which the status of the task is not evident at any
given point in time. Thus, an incorrect action (i.e.,
inappropriate at this point in the task) may be
performed, or an incorrect object may be used. For
example, another ISA (1973) documented an error in
which a file was created using an old version of the
required software. The problem was traced to the fact
that, as new versions of the fide generation software
became available, they were simply installed on the
appropriate machines. As time passed, multiple
versions of the software were available. To the
operator generating the file, it was not clear which
was the correct version of the software. Thus, the
task status was not evident, and a wrong object (the
old software) was used. The lack of distinction
between the current software and previous releases
generated a description error. There were no salient
attributes to facilitate the use of the correct software
release. This error resulted in loss of time, since the
problem had to be researched and the file regenerated,
thus increasing operations costs. In addition, risk
increased since an incorrect fde v)as generated.
As these common cognitive elements were uncovered,
it became clear that the single common element
underlying this cluster was that all the ISAs were
execution errors. Thus, this cluster, at distance 3.3,
was labeled "Execution Errors"(Figure 3). Finally, at
the third major level of clustering, at distance 6.6,
planning errors joined the execution errors, thus
suggesting a label of "Execution and Planning
Errors."
Preliminary Conclusions
Although this defect prevention and error management
program is in its infancy, some preliminary
conclusions can be drawn from the initial analysis.
1. Flight mission operations human error
data is amenable to interpretation via
human error theory. JPL currently has a large
volume of ISA data. While this data may be locally
analyzed within a project during operations,
particularly during a major anomaly, the analysis is
typically ad hoc and localized to that one project.
This preliminary work demonstrates that by modeling
error data, underlying causes can be investigated in a
systematic way, and classes of errors in this
environment (such as execution errors) can be
uncovered. In addition, this general information can
then be shared across projects.
2. In JPL flight mission operations, a
significant portion of human ISAs are
errors in executing a task. The results of this
study showed that 52% of the 204 Magellan human
errorsanalyzed were execution errors. This provides a
focus for possible solutions on execution problems.
It is also speculated that execution errors will be
found to be preventable or manageable.
503
3. System requirements, policies and
procedurescanbe written to preventknown
cognitiveerrors. Aswas previously mentioned,
through this systematic analysis, classes of errors
will be uncovered. In this way, solutions to manage
errors that do occur, or solutions to prevent them
from occurring can be generated. For example, to
prevent errors like the one documented in ISA 8508,
special conditions in a file can be highlighted to avoid
capture in routine tasks. To avoid errors such as ISA
1973, proper configuration management policies and
procedures should be written and enforced in
operations. In this ease, archiving old versions of the
file-generation software off-line would eliminate
operator confusion about which software to use in
generating files and thus prevent this oversight or
slip.
In summary, a method for analyzing human errors in
flight mission operations has been presented.
Although in a preliminary phase, it is clear that such
a method in which error data is subjected to a cluster
analysis, the resulting clusters are examined for
common cognitive elements, and these elements are
modeled using cognitive psychological theory, can
lead to an understanding of the causes of errors and
typical classes of errors. Using TQM principles,
these findings can then be used at the beginning of a
project's life cycle to improve system requirements,
project policies and procedures, and operator training
to manage errors that do occur, or prevent them from
occurring at all. It is only through such a systematic
analysis method that cost and risk can be reduced in
flight mission operations. Finally, it is clear that
this analysis has wide applicability to other errors. It
is currently planned that this program will eventually
expand to include analysis of other errors in Figure 1
such as software and documentation, and to other
environments such as the DSN and system
developmenL
Acknowledgement
The research described in this paper was carried out by
the let Propulsion Laboratory, Cal'ffomia Institute of
Technology, under a contract with the National
Aeronautics and Space Administration.
References
Anderson, J. R. & Bower, G. H., Human Associative
_, V. H. Winston, Washington,
D.C., 1973.
Card, S., Moran, T., & Newell, A., :]J_cA_Y.gho]gg_
of human-computer interaction, Erlbaum
Associates, HiUsdale, NJ., 1983.
Norman, D. A., "Categorization of action slips,"
Psychological Review, 88, 1981, 1-15.
Norman, D. A., "Design rules based on analyses of
human error, "Communications of the
ACM, 26 (4), 1983, 254-258.
Norman, D. A., The desi_ of everyday things.
Doubleday, New York, 1988.
Reason, J., Human error, Cambridge University
Press, New York, 1990.
Senders, J. W. and Moray, N. P., _,
Lawrence Edbaum Associates, HiUsdale,
N.J., 1991.
504
APPENDIXA
Taxonomyof HumanError CauseCodes
HUM
HUM1
-1
-2
,,, i
-3
-4
-5
-6
-7
-8
-9
-10
-11
HUM2
-1
-2
-3
HUMANS
-4
-5
-6
-7
HUM4
-I
-2
Inadequate Knowledge/
Inex_tience
Procedures
Policies/guidelines/
requirements
S/C characteristics
Command procedures
Ground Operations
S/C Status
Constraints
Schedule change
Anticipated command effect
HUM5
S/W Flight
S/W Ground
Violation of
Cons_'ain_Procedure
Procedures
Rule/
Policies
Guidelines
Flight ( Mission Rules
S/C Compatibility
Ground Operations -
Compatibility
Operational Requirement
Error
Wrong Plan - Mistake
Plan OK - Error Unknown
-3 Plan OK - Omission of Action
-4
-1
-2
-3
-4
Plan OK - Oversight
Product Interface
Error in Copying
Error in calculation
Data entry error
Edit Error
HUM6 Communication
-1 Lack of communication
-2 Miscommunication
ii
Error made due to inexperience or lack of knowledge if person is
expenencect
Error made due to an intentional deviation from plan
Error made due to an unintentional deviation from plan
Plan is wrong, but was executed correctly
Plan is correct) error is unknown
Plan is correct t but an action was omitted during execution
Plan is correct) but an action during execution was wrong.
Error made while pr_ucing a product
Errorin original data entr7
Error in editing an existing product
505

Session H3: HUMAN PERFORMANCE MEASUREMENT III--
OPERATION SIMULATION
Session Chair: Capt. James Whiteley



Analyzing human errors in flight mission operations

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server