A multidimensional bipolar-adjective rating scale is employed as a subjective measure of operator workload in the performance of a one-axis tracking task. The rating scale addressed several dimensions of workload, including cognitive, physical, and perceptual task loading as well as fatigue and stress effects. Eight subjects performed a one-axis tracking task (with six levels of difficulty) and rated these tasks on several workload dimensions. Performance measures were tracking error RMS (root-mean square) and the standard deviation of control stick output. Significant relationships were observed between these performance measures and skill required, task complexity, attention level, task difficulty, task demands, and stress level

Bird, K. L.

English

NASA Technical Reports Server

=N82-1366,
4
SUBJECTIVE RATING SCALES AS A WORKLOAD i
ASSESSMENT TECHNIQUE
Kathleen L. Bird*
\
Virginia Polytechnic Institute
SUMMARY
Any investigation of the task workload inherent in flying must address
many dimensions including cognitive, perceptual, and psychomotor. The pre-
sent study employs a multidimensional bipolar-adjective rating scale as a
subjective measure of operator workload in the performance of a one-axis
tracking task. The rating scale addressed several dimensions of workload,
including cognitive, physical, and perceptual task loading as well as
fatigue and stress effects. Eight subjects performed a one-axis tracking
task (with six levels of difficulty) and rated these tasks on several
workload dimensions. Performance measures were tracking error RMS (root-
mean square) and the standard deviation of control stick output. Signifi-
cant relationships were observed between these performance measures and
skill required, task complexity, attention level, task difficulty, task
demands, and stress level.
INTRODUCTION
There is little agreement among scientists in how they conceive work-
load. To arrive at a functional, accurate, definition of workload several
questions must be addressed. Does workload refer to the task demands im-
posed or. the operator, or is workload the operator effort required to sat-
isfy those task demands? What role, if any, does operator fatigue, physi-
cal and mental, as well as emotional stress play in the operator's assess-
ment of workload? Is an individual's assessment of workload level really
an assessment of a combination of all these factor_? Most of the current
workload definitions focus on a sh_gle facet of this multidimensional area.
Jahns (1973) defines workload as "...the extent to which an operator is
occupied by a task" (reference i). Focusing on task performance measure-
ments, Levison, Elkind, and Ward (1973) define workload as "...the frac-
tion of the controller's capacity that is required for him to perform a
given task to some specified or criterion level of performance" (refer-
ence 2). In an attempt to express the multidimensional aspects of work-
load, Tennstedt (1973) defines it as "...a summation of such processes as
perception, evaluation, decision-making and actions taken to eccomodate
those needs generated by influences originating within or without the air-
craft" (reference 3). While Tennstedt's workload definition addresses
several workload dimensions, it falls short of addressing those questions
previously posed.
* This research was conducted at NASA-Ames Research Center and was sponsored
by NASA _rant NAG-217 to Virginia Polytechnic Institute and State Univ.
-33- ,...,,., ,,,T_L_,,:,,",
1982005792-046
https://ntrs.nasa.gov/search.jsp?R=19820005796 2020-03-21T15:56:13+00:00Z
The multidimensional aspect of workload is demonstrable in the flight
duties of a pilot. A pilot's flight duties may encompass facets of cogni-
tive (e.g., in-flight computations, fuel consumption management), percep-
tual (e.g., monitoring instruments, kinesthetic cues), and psychomotor
(e.g., manual control of the yoke, rudder pedals) aspects. The pilot must
also encounter, and cope with, the effects of fatigue and stress (mental\
and physical).
To investigate the multidimensional aspects of flying, wor_load
assessment techniques (both behavioral and physiological) should address
the cognitive, perceptual and psychomotor dimensions, as well as measure
operator fatigue, and stress. Wierwille (1979) has suggested that a fruit-
ful area of research would combine the best of physiologic_l measures with
behavioral measures in a multivariate analysis as a function of workload
(reference 4). One or more workload assessment techniques need to be de-
velop_d that can reliably measure the multiple dimensions of workload.
A subjective rating scale may offer a promising behavioral workload
assessment technique. Hicks and Wierwille (i_79) compared workload
measurements obtained fro_. rating scales with thos_ obtained from primary
task performance, secondary task performance, occlusion, and physiological
measures (reference 5). Specifically, the rating scale proved to be a
sensitive measure of operator workload in the performance of an automobile
driving simulation task. Jenney, Older, and C/meron (1972) reported
"...encouraging findings as to the usefulness and validity of subjective
magnitude estimates" (reference 6). They recorded hourly subjecti@e esti-
mates of fatigue, tension, and task difficulty in assessing workload levels
involved in performing an information p£ocassing task. Borg (1971) employed
a simple rating scale and reported good agreement between perceived ex-
ertion, and difficulty, and physiological indicators of efforh (stress)
(reference 7).
The purpose of this study was to develop and validate a multidimension-
al rating scale to assess pilot workl_ad. Several dimensions of workload
were addressed, including cognitive, physical, and perceptual task loading
as well as fatigue and stross effects.
Subjective Rating Scale
The multidimensional rating scale included 15 bipolar adjective
pairs, one or more pairs addressing each of the several wor_load dimensions.
These bipolar adjective pairs dichotomized: i. skill required (no skill-
much skill), 2. task complexity (simple - complex), 3. attention level
(extremely low - extremely high),. 4. monitoring (none - constant), 5. task
difficulty (easy - difficult), 6. controlability (easy - difficult), 7. my
performance (unsatisfactory - satisfactory), 8. instructions (clear - con-
fusing), 9. task demands (undemanding - demanding), i0. energy level (lazy -
energetic), II. stress level (l,_w stress - high stress), 12. activity level
(idle - bl]sy), 13. fatigue (tired - refreshed). 14. task stability (predic-
table - unpredictable), and 15. J,.terest level (bored - interested).
1982005792-047
The scales appeared one at a time on a CRT (cathode ray tube). The
adjectives were positioned at opposing ends of a vertical line, with the
descriptor [e.g., skill required, attention level) positioned below the
scale. Subjects assigned a subjective rating (scale 1 to i00) to the
tasks by positioning a cursor along the vertical line.
To validate the multidimensional rating scale, it is necessary to have
subjects perform a battery of tasks which concentrate on different aspects
of workload and examine whether the rating scale accurately measures these
aspects. Future studies will emplo I a battery of six to eight primary tasks
(similiar to the Civil Aeromedical Institutes Multiple Task Performance Bat-
tery, MTPB) which will include cognitive, perceptual, and psychomotor com-
ponents. The primary tasks selected will closely approximate tasks demanded
in flying.
The present study examines the psychomotor aspects of workload. Sub-
jects performed a one-axis compensatory tracking task with six level_ of
difficulty. They rated the six tracking tasks for degree of workload using
the multidimensional rating scale
Tracking Task
The task was a one-axis compensatory tracking task with a K/S plant.
A va,_dom number generator provided a rectangular distribution of frequencies
(bandwidth of 1.0, 1.5, 2.0 rad/sec) filtered through a second-order filter
to produce the forcing function. The filtered output produced the movement
of the cursor.
Difficulty was manipulated by varying the standard deviation (SD of
32, 64) and bandwidth (I.0, 1.5, 2.0). The following tasks were presented:
I. task 1 (bw 1.0, SD 32), 2. task 2 (bw 1.5, SD 32), 3. task 3 (bw 2.0,
SD 32), 4. task 4 (bw 1.0, SD 64), 5. task 5 (bw 1.5, SD 64), 6. task 6
(bw 2.0, SD 64). Performance measures were the tracking error _MS (root-
mean square) and the standard deviation of the control stick output.
The tracking tasks were presented on a CRT. The six tracking tasks
consisted of a vertical line (5.56 cm) which randomly moved in a lateral
direction. Maximum displacement of the cursor was 12.70 cm. The subjects
task was to keep this cursor centered between two stationary vertical lines
(2.11 cm) by means of a control stick right and left.
METHOD
Subject s
Five males and three females (aged 18 to 42) served as paid volun-
teers. These subjects had bell previously screened for tracking ability
to gaurantee a minimum amount of psychomotor ability_ A pilot study
-35-
• ! t"
1982005792-048
yielded a criterion score which the subjects were required to achieve
before selection. All subjects were right handed.
Apparatus
This study was conducted in a small, sound-attenuated experimental
chamber. The subject was seated before a CRT. The control stick was
located on the right arm of the chair. The throttle was located on the
left arm. Data aquisition was recorded and task presentations were pro-
gramuned through a Digital Equipment Corporation PDP-12 computer.
Procedure
Subjects were told the purpose of the study, given a description of
the required tasks, and instructions for rating the tasks on the various
workload parameters. They were told these tasks would vary in degree of
difficulty. The importance of maintaining an equally high standard of per-
formance across all tasks was stressed. To familiarize the subjects, tasks _ _
were presented (in order of ascending difficulty) &nd the subjects were
permitted to track each task for one minute. During the experimental
session the tasks were not presented in order of ascending difficulty but
rather in random order. The subjects were given a one-minute practice
session prior to each experimental session. The experimental session
(for each tracking task) immediately followed the practice session for a
duration of four minutes. After completing each tracking task subjects
gave a rating for each of the 15 bipolar adjective pairs. As the scales
appeared on the CRT subjects would move the throttle to position a cursor
along the vertical line to indicate their rating. When they were satisfied
with their rating, they pressed a response button. Immediately, a second
scale would be displayed.
Subjects were required to perform each tracking task for a d,lr_tion
of four minutes. Standard deviation of the tracking error, output error
RMS, standard deviation of control stick output, and stick output RMS were
sampled every 30 milliseconds. The following analyses were performed on
the data collected during the final two minutes of e_ch experimental track-
ing session.
RESULTS
Effect of task difficult_ on error RMS
A 2 (standard deviation) x 3 (bandwidth) analysis of variance was
computed on the error RMS to determine if a significant difference in error
RMS would appear as a result of mantDulatin_ the bandwidth and standard
deviation. Results indicated a significant difference attri_-table to the
bandwidth factor, F (2,14) - 49.30, R_ .01 and the standarJ devi_4on
-36-
1982005792-049
factor, F (1,7) = 311.59, p (.01. In addition, there was a significant
interaction between the bandwidth and standard deviation factors, F (2,14) =
10.b0, £< .01.
Effect of task difficult_ on the standard deviation of stick output
A 2 (standard deviation) x 3 (bandwidth) analysis cf variance was
computed on the standard deviation of stick output to determine whether
manipulating the bandwidth or standard deviation _'ould produce a signifi-
cant difference. There was a significant differelce attributable to the
bandwidth factor, [ (2,14) = 8.25, _ .01 and the task standard devia-
tion factor, [ (1,7) = 59.81, _C.01. There was no significant inter-
action between these factors, [ (2,14) = 2.24, _ >.05.
Eff____eecto__ftas____kdifficult_,o .nnbipola[ adjective ratings
Fifteen 2 (standard deviation) x 3 (bandwidth) analyses of variance
were computed on the ratings for each of the 15 bipolar adjective scales.
There was a significant difference attributable to the bandwidth factor
for the following scales: skill required (_ (2,14) = 9.62, £ (.01);
monitoring ([ (2,14) = 5.05, _ 4.05); task difficulty ([(2.14) = 12.59,
£ 4.01); my performance (}_ (2,14) = 8.71, £< .01); task demands ([ (2,14) =
4.46, £_.05); and stress level ([ (2,14) = 5.90, £(.05). There was a
significant difference _n the ratings attributable to the standard deviation
factor for the following scales: skill required ([(1,7) = 55.86, _( .01);
task complexity ([(1,7) = 28.84, £(.01); task difficulty ([ (1,7) = i8.04,
£ _ .01); controlability ([(1.71 = 13.93, £ (.01); my performance ([(1.7) =
7.67, £_ .01); task demands ([(1,7) = 6.90, £4.05); stress level (P(I,7) =
7.94, £_ .05); and fatigue (F(l,7 = 20.87, £ f.01). No significant effect
attributable to either the ban0 dth or standard deviation factoEs were
found for the following scales: attention level; instructions; energy
level; activity level; task stability; or interest level (_).05). There
were also no significant interactions behween the bandwidth and standard
deviation factu_s for any of these analyses (with the majority of [ values
less than one, £).05).
Relationship between error RMS and bipolar adjective ratings
To determine if a significant relationship exists between the error
RMS and the scale ratings, Pearson product-moment correlations were com-
puted (df = 47}. The following significa,.; correl_tions were derived be-
tween scale ratings and error RMS scores: skill required (r = +.55, _4.01);
task complexity (r = +.40, _<.01); attention level (r = +.42, £<.01);
monitoring (r = +.44, £ (.01); task difficulty (r = +.57, _f .01); con-
trolability {r = +.61, £( .01); task demands (r = +.51, _(.01); stress
(r = _.28, _<.05); and task stability (r = +.32, _<.05). The remaining
scales were not significantly correlated (at or above _ ).05) with error
RMS: my performance, instructions, energy level, activity level, fatigue,
\
-37-
1982005792-050
iand interest level.
Relationship between stick output standard deviation and scale ratings
To determine if a significant re] tionship exists between the stick
' output standard deviations and the _cale ratings, Pearson product-moment
correlations were computed (df = 47). The following bipolar adjective
scales were found to be significantly correlated with stick output stan-
dard deviations: skill required (r = +.55, _<. Jl); task complexity (r =
+.53, £(.01); attention level (r = +.29, _( .0,); task difficulty (r =
+.55, _(.01); my performance (r = -.45, _ <.01); task demands (r = +.46,
(.01); stress (r = +.40, £(.05); and activity level (z = +.29, _(.05).
The following scales were not significantly correlated with stick sta:idard
deviation: monitoring, controlability, instructions, energy level, fatigue,
task stability, and interest level.
The correlation between error RMS and standard deviation vf the stick
output was significant (r = +.44, _(.01).
DISCUSSION
The relationship between increasing task demands and task performance
(output error RMS) was examined. As the task demands increase (with an in-
crease in input bandwidth and input standard deviation) subjects' aDility
to reduce this error decreased. Increasing the input standard deviations
from 32 to 64 produces an increase in output error RMS. This result might
be expected considerinq the relative amount of error the s,'bject is asked to
reduce. Increasing the bandwidth _i.9. 1.5, 2.6) produced an increase in
output error RMS (Mean = 25.8, 36.2, 43.5). Manipulating the task _nd%idth
and standard deviation h_d a significant effect on the standard dev_atiJn
of stick output. Doubling the input standard deviation (from 32 to 64) also
doubled the mean standard deviation of the stick output (Mesn= 12.3 (SD 32),
Mean = 24.9 (SD 64)). A similiar increase in mean stick _tandard deviation
coul_ be attributed to an increase in the bandwidth (I.0 (mean = 15.7),
1.5 (mean = 19.7), 2.0 (mean = 20.6)). In summa[ _)n, as the task demand
increases, a degradation of task performance occ_ s. As task demand in-
creases, subject effort (as measured by stick ouzput standard deviation)
also increases.
Theoretically, an increase in task difficulty should be reflected in
the subject's evaluation of task workload level. Several rating scales are
strongly related to output error RMS scores. Apparently performance degra-
dation was stron_!y reflected in evaluation of skill required, task difficulty,
controlability, and task demands. An increase in subject effort was strongly
reflected in evaluation of skill requi_ed, task complexity, task difficulty,
and task demands. TheJe scales are most promising as indicators of certain
workload dimensions a_ should be investigated further with other flight
relate& _oks.
-38-
1982005792-051
? !
REFERENCES
i. Jahns, D.W. Operator workload: What is it and how should it be meas-
ured? In K.D. Cross and J.J. McGrath (Eds.) Crew System Design_.
Santa Barbara, California: Anacapa Sciences, July, 1973.
2. Levison, W.H., Elkind, J.I. and Ward, J.L. Studies of multivariable
manual control systems: A model for task interference.
NASA-CR-1746, 1971.
3. Wierwille, W.W. Pysiological measures of aircrew mental workload.
Human Factors, 1979, 21, pp. 575-593.
4. Hicks, T.G. and Wierwille, W.W. Comparison of five mental workload
assessment procedures in a moving-base driving simulator.
Human Factors, 1979, 21(2), pp.129-143.
5. Jenney,L.L., Older, H.J., and Cameron, B.J. Measurement of operator
workload in an information processing task. NASA-CR-2150, 1972.
6. Borg, G. Psychological and physiological studies of physical wor_.
In Singleton, W.T., et al. (Ed.) Measurement of Man at Work.
London: Taylor and Francis, 1971, pp. 121-128.
-39-
1982005792-052


Subjective rating scales as a workload

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server