One mission of the Paul M. Fitts Human Engineering Division of Armstrong Laboratory is to improve the user interface for complex systems through user-centered exploratory development and research activities. In support of this goal, many current projects attempt to advance and exploit user-interface concepts made possible by virtual reality (VR) technologies. Virtual environments may be used as a general purpose interface medium, an alternative display/control method, a data visualization and analysis tool, or a graphically based performance assessment tool. An overview is given of research projects within the division on prototype interface hardware/software development, integrated interface concept development, interface design and evaluation tool development, and user and mission performance evaluation tool development

Eggleston, Robert G.

English

NASA Technical Reports Server

MULTI-MODAL VIRTUAL ENVIRONMENT RESEARCH
AT ARMSTRONG LABORATORY
Robert G. Eggleston
Paul M. Fitts Human Engineering Division
Crew Systems Directorate
Armstrong Laboratory
Wright-Patterson AFB, OH
INTRODUCTION
One mission of the Paul M. Fitts Human Engineering Division is to improve the user interface
for complex systems through user-centered exploratory development and research activities. In
support of this goal, many current projects attempt to advance and exploit user-interface concepts made
possible by so-called Virtual Reality (VR) technologies. Virtual environments may be used as a general
purpose interface medium, an alternative display/control method, a data visualization and analysis tool,
or a graphically based performance assessment tool. All of these uses of VR may be exploited in the
development of new user-interface prototypes and supporting design tools. As a result, the Division
has several active R&D efforts in these areas, as they pertain to user-machine interfaces.
The purpose of this presentation is to provide a brief overview of the range of R&D projects
within the Division that involve VR technology. For the purpose of discussion, research projects are
clustered into four categories:
• Prototype Interface Hardware/Software Development
• Integrated Interface Concept Development
• Interface Design and Evaluation Tool Development
• User and Mission Performance Evaluation Tool Development.
125
https://ntrs.nasa.gov/search.jsp?R=19960026481 2020-06-16T03:56:38+00:00Z
PROTOTYPE INTERFACE HARDWARE SOFTWARE DEVELOPMENT:
HELMET-MOUNTED DISPLAY SYSTEMS
The Division has been a pioneer in the development of visually coupled systems (VCS)
technology which centers around the design of helmet-mounted display (HMD) devices. Beginning in
1966, a line of HMDs has been produced, ranging from small field-of-view (FOV) monocular systems
to wide FOV stereoscopic ones. R&D activities address the development of high luminance miniature
CRTs, relay optical systems, head gear, militarized image generators, trackers, and all technological
and engineering aspects required to produce fully integrated systems for use in airborne platforms. The
technical advancements made in this work over the years have provided the foundation from which the
virtual reality industry has emerged. For example, the need for a method to reliably track head position
without restricting operator movement or emergency egress stimulated the development of the magnetic
tracking concept now used in many VR systems. The first successful implementation of such a device
was sponsored by the Division and achieved in collaboration with our staff.
Current VCS development efforts are concentrated on:
• standardized, plug compatible VCS systems for integration into military platforms
• color CRT that can operate in high (luminance) ambient environments
• system and component performance measurement methods and standards
• psychophysical performance assessment.
Helmet-Mounted
Display & Visually
Coupled Systems
Technology
Laboratory
Figure 1
126
VCS CONTROL/DISPLAY CONCEPTS
In addition to developing the hardware and software technology infrastructure for the VCS
system, we also develop advanced interface concepts around the technology. One example is the
design of an interface device that supports head aiming (to select and track targets) and head-referenced
instrument displays (flight and weapons) that are always available to the operator. This work involves
the design of display formats, symbology, and control methods. It also includes a wide range of
human performance research to insure compatibility of the concept to both the user and the expected
real world task conditions. Several concepts have been developed and evaluated in simulated and
actual flight tests.
Figure 2
127
THREE-DIMENSIONAL AUDIO DISPLAYS
Similar to the VCS R&D efforts, Armstrong Laboratory has also been a leader in the
development of 3-D audio systems, including hardware concepts, psychophysical analysis, application
systems, and flight testing. Our first localized display was designed and fabricated in-house in the late
1980's. New systems compatible with operational avionics systems are being developed and
evaluated.
The geodesic dome shown below is housed in an anechoic chamber located in the laboratory.
This is the only available facility in the country that can support the full range of technical studies
needed to advance this technology.
A study was recently completed that addressed target acquisition performance with and without
the aid of a 3-D sound localizer. A summary of the results of that study is shown in the next two
charts. Each graph depicts target search times as color zones (shown here as shapes of gray) over the
search space defined in terms of azimuth and elevation angle. The first graph displays times when
search was completed visually without the aid of localized sound cueing. The next graph shows the
search time map when localize sound cueing was used. The improvement with sound cueing is
obvious from visual inspection of the graphs.
3-D Audio
Displays
1989
1992
1996?
Figure 3
128
3-D AUDIO DISPLAYS
100
8O
60
40
20
Visual Only Search
(Time in msec)
SIMULATED
I-1._N'_ 3,Nmi _ TO
iJ TO 1.500
_TO IJOmO
Figure 4
100
80
60
w_ 4O
z 20
m
.1
w -20
-4O
-60
-8O
-2OO
Combined 3-D Audio/Visual Search
i I I II . _ . IIIIIIIW I
i t t, ii| i i _ ' I _._ i...t_ l.J,_i l,,,i I , , , t|, _,,
-1SO -100 -60 0 SO 100 lSO ZOO
,I_ZmUTX(Oqm)
(Time in msec)
SIMULATED
I-_.ro 3,qlmIjieelo _i,._t0"lvt}1JiN
ijINlro t_
NIITO IJN0
111 iii
Figure 5
129
ALTERNATIVE CONTROL TECHNOLOGIES
Several projects in the Division are involved with developing and exploring the feasibility of
novel interface methods as alternative I/O devices for use in a wide variety of application domains.
These include the development of a gesture recognition system that combines a data glove which senses
figure postures and gestures (coupled with a position/orientation tracker) with a neural net classifier. A
demonstration system is being constructed that recognizes American sign language. The device may be
used as an input mechanism to a system/computer or as a means to support nonverbal communication
between people. The Veterans Administration has expressed interest in this work as an assistive
technology.
A facial sensor testbed has been developed to determine if continuous speech recognition
accuracy can be improved by "reading" facial expressions during speech. The basic idea is to use facial
patterns as a basis for disambiguating classification uncertainties found with current hidden Markov
model speech recognition systems.
Perhaps the most unusual alternative control technology under study is called Brain Actuated
Control (BAC). As the name suggests, we are exploring ways brain states can be detected and used as
a control decision which is then executed by the use of standard servomechanism principles in a closed-
loop system. We have demonstrated reliable two-state based brain control, with the operator
voluntarily enhancing and suppressing the magnitude of a certain EEG frequency. Continuous control
has also been demonstrated through exploitation of a composite voluntarily produced EEG-EMG
signal. This is another area where our work has spin-off potential for use with disadvantaged
individuals.
Facial Sensor Test Bed Hand Gesture Recognition
Figure 6
130
Brain Actuated Flil;ht Simulator
--o
SIMULATOR
DISPLAY
VISUALEVOKED RESPONSE
SIMULATOR
;>
:Z
_H
o
I-"
INTEGRATED INTERFACE CONCEPT DEVELOPMENT
Beginning with the Visually Coupled Airborne Simulation System (VCASS), which became
operational in the early 1980's, we have had the ability to produce virtual interfaces or virtual cockpit
concepts for investigation. This system probably marked the beginning of the new era of VCS
technology that has become known as Virtual Reality, Virtual Environment, or Synthetic Environment
technology. VCASS provides a 120 degree (horizontal) by 60 degree (vertical) instantaneous FOV in a
head-mounted stereoscopic display system. We have coupled it over the years with data gloves, a
simple force reflecting device, and 3-D audio to produce various forms of virtual cockpits.
Unfortunately, limitations in computer graphics and system time delays have interfered with our ability
to adequately assess the value of these innovative interfaces.
We have recently opened a new facility called the Synthesized Immersion Research
Environment (SIRE), that provides a full capability immersive VR system. It integrates haptic displays
with 3-D sound and VCS technology and also includes a large screen projection system. SIRE
supports a wide range of human performance research, including investigations of augmented cockpit
concepts which couple VR concepts and alternative control concepts with more conventional panel
mounted displays.
132
INTEGRATED INTERFACE CONCEPT DEVELOPMENT
Figure 8
133
INTEGRATED INTERFACE CONCEPT DEVELOPMENT
Z
.<
Z
Z
L::"_:-E:"-:"]
:::::::::::::::::::::
ilIiti_N
134
Figure 9
INTEGRATED INTERFACE CONCEPT DEVELOPMENT
Figure 10
135
INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT
Historically, it has often proven difficult to transition user-centered interface ideas and concepts
into operational systems. There are many reasons for this breakdown in the development process.
Most agree that one reason is the lack of adequate interface design and evaluation tools that fit properly
into the systems engineering process used in advanced design of large-scale systems. In an effort to
overcome this difficulty, we have been developing tools and concepts to support the design and
evaluation process.
One area of research important to this goal is engineering anthropometry. In this area, we
perform studies that allow us to develop relevant anthopometric data bases. For example, we assess
the.ability of different body types (size, strength, etc.) to perform different actions often required in
maantenance tasks. Data bases like these have been combined with computer graphics systems to
produce interactive visualization and analysis tools for assessing system designs (computer
representations). VR has recently been used to allow the designer to assume different body sizes
during the analysis.
136
INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT
Figure 11
137
INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT
138
Figure 12
INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT
Figure 13
139
HUMAN CENTERED DESIGN ON THE CAD/CAE PIPELINE
Another effort has concentrated on the development of active information using multi-media
technology. The goal is to be able to place human performance data into the CAD/CAE environment
used in design. This research has addressed ways to improve access to information, ways to help the
designer understand what questions they need to ask, and ways to facilitate opportunistic searches. In
addition, we have invested in the production of a new tool called a perception and performance
prototyper. This software product allows the designer to transition from equations and graphical
presentations and analysis to experiential methods that allow the designer to experience the perceptual
and performance consequences of their design decisions.
Figure 14
140
:Z
:Z
0
:Z
F-'
I,
C_
v.-,
"' Perception & Performance Prototyper
Vertical Sinusuoidal Vibration i Display Image
YibraUo n So urce
0 o_y
(_ Display & ObserYer
O obserY_
Display Vibration versus Observer Vibmtlon EITects :_-: r ..........
i i i iii
Fuel L_tel: Low
Vibration Frequency (I
Vibration Magnitude (ms 2
I!Iltl t t!¥fll I!IW
Click START to demonstrate the effectof verticalvibrationon the
appearance of the Displa! e. View from a distance of 0.75
:3
N
:Z
:=
¢3
USER PERFORMANCE AND VIRTUAL SYSTEMS
It is clear that the potential applications for Virtual Technology are enormous. DoD, for
example, is moving ahead with the use of this technology as both a tool and an end product in the
development of advanced systems. Distributive Interactive Simulation (DIS) is being considered as a
test bed to investigate advanced concepts under operational like conditions. Some of these systems
may be virtual; others physical. In this and all other work involving VR technology, we suffer today
from a lack of adequate understanding of how user performance interacts with properties of the VR
technology itself. Without information like this we will not be able to determine to what degree user
performance in, say, a DIS environment, reflects what can be expected in the real world, and what is
due to characteristics of the VR simulations. Information of this nature is also needed to aid in design
decision making, and to provide performance-based bench marks or requirements to guide VR
technologists in advancing the state of the art.
We have undertaken a research project that focuses on this issue. One goal of this effort is to
establish a quantitative relation between VR system properties and user performance. Our plan is to do
this with a series of standardized bench mark tasks that cover a wide range of perception, perceptual-
motor, and cognitive activities. To date we have completed studies that demonstrate the cost of VR
system time delay on manual tracking and control and aimed movement control. The effects of several
VR system variables on size-distance judgments in a virtual environment have also been studied.
The next chart depicts the fact that user performance based design data requires a link to be
made between properties of user performance with properties of VR systems. The user performance
properties must, in turn, be linked to specific application domains in order to map user performance to
task performance. One goal of our research is to provide the desired user-performance data and the
tools needed to complete the mapping to a wide variety of task domains.
Figures 18 and 19 provide a brief summary of the results from some experiments that
investigated simple aimed movement performance which was accomplished in a virtual environment
(VE). In general, the data follow Fitts Law over an ID range of 2 to 6, but show departure from
linearity around an ID of 7. In comparison with physical aimed movement, performance by highly
practiced subjects was degraded by about 35-50% in a VE (see Fig. 19). Further, the effect of time
delay in the VR system caused movement times to increase substantially under ID 5 conditions,
somewhat for ID 6 conditions, and very little at other ID levels (see Fig. 19). Data like these provide
the basis for constructing design trade-off nomographs like the example shown in Fig. 20.
A design trade-off nomograph (see Fig. 20) is a form of data representation that highlights the
relation between a measure of user performance and two or more measures of VR system properties.
This form of representation allows a VR system designer to easily see how user performance interacts
with system properties. It can be used for defining system requirements and for performing trade-off
analyses to select between alternative design options that can meet a requirement. Thus, it aids the
designer in seeing the connection between user performance in a way that is clearly connected to
system properties, but at the same time allows the designer to bring in technology risk, cost, and
schedule factors in the process of deciding how to meet a performance goal.
143
USER PERFORMANCE AND VIRTUAL SYSTEMS
144
Figure 17
O0
Mean Movement Time by ID and Study
(secs)
2.0
1.8.6
1.4
_) 1.2
E
1.0
0 o.8
¢U 0.4
@
0.2
0.0
T I _ t I 1
_ -
/
_J/
- i- _ (lcr-oesID - .0211
1 2 3 4 5 6 7 8
Study
et el. ('Jo_,q
.<
Index of Difficulty
_n
o_
Reciprocal Tapping Task
Mean Movement Time by ID and Study
(secs)
2.0
1.81.6
1.4
a 1.2
E
• 1.0
>
0 o.8
_; 0.6
m 0.4
0
:! 0.2
0.0
l ! ! ....! !
_J
1 2 3 4 5 6 7 8
_tudy
o _ _a _.I_
• Pm_ rwmm a _ Ira4)
Pky_J _ & lab (_m)
El ewN ele (mid - e_ (_0 Hz)
<
.<
C_
Index of Difficulty
USER PERFORMANCE AND VIRTUAL SYSTEMS
Figure 20
147
CLOSING REMARKS
As you can see, Armstrong Laboratory is engaged in a broad range of R&D research that
involves both the development and exploitation of VR technology. We have a long history with the
technology which has allowed us to achieve a high degree of understanding of the technical issues and
challenges that need to be solved to insure its effective use in different application domains. We are
very sensitive to the fact that VR technology by itself is neither desirable or undesirable. Rather, its
value depends on how well we can unite it with the human user to produce successful products.
DETERMINING  QUlREMENT$ AND
CAPABILITIES OF VIRTUAL FI   &CES
Figure 21
148


Multi-modal virtual environment research at Armstrong Laboratory

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server