The principal objective is to develop a laboratory testbed that will provide a unique capability to elicit, control, record, and analyze the relationship of operator task loading, operator eye movement, and operator brain wave data in a computer system environment. The ramifications of an integrated eye/brain monitor to the man machine interface are staggering. The success of such a system would benefit users of space and defense, paraplegics, and the monitoring of boring screens (nuclear power plants, air defense, etc.

Harrington, Nora

Villarreal, James

English

NASA Technical Reports Server

N 8  8- 112 1 9  
DEVELOPMENT AND EXPERIMENTATION OF AN 
EYE/BRmRASK TESTBED 
Nora Hanington, Ph.D., Analytics, Willow Grove, PA 
James Villameal, NASNJSC, Houston, TX 
An SBIR (Small Business Innovative Research) 
Phase I was awarded to Analytics to investigate the 
feasibility of an innovative concept that uses an operator's 
brain waves as a control mechanism for computer systems. 
The Phase I reported that the present brain wave recording 
technology is incapable of using these signals for direct data 
transmission. But the development of such technologies as 
super conductive materials at near room temperature and 
biomagnetism is advancing rapidly. A direct application 
from conventional MEG or EEG sensing systems could 
determine an operator's state of awareness. 
The principal objective of Phase I1 is to develop a 
laboratory testbed that will provide a unique capability to 
elicit, control, record, and analyze the relationship of 
operator task loading, operator eye movement, and operator 
brain wave data in a computer system environment. The 
ramifications of an integrated eyebrain monitor to the man 
and machine interface are staggering. The success of such a 
system would benefit users of space and defense, 
paraplegics, and the monitoring of boring screens (FAA, 
nuclear plants, Air Defense, etc.) 
INTRODUCTION 
A variety of man-machine interface concepts have 
been developed in recent years in an attempt to: (1) increase 
the flow of relevant information between the system and the 
operator, and (2) alleviate the need for complex programmer- 
oriented input protocols. These concepts have concentrated 
on the presentation, selection, or display aspects of the 
interface. Another component of innovative interface design 
is control mechanisms for computer systems. Development 
of advanced hardware and software systems for mission 
planning and control is desirable to enhance the human 
operator's job performance, especially during periods of 
high workload. 
MErHODoLoGY 
The Phase I research was an investigation of the 
feasibility of using brain waves as a control input to a 
computer system. Currently, there are a number of devices, 
such as the mouse and the touch screen, that allow for more 
direct and intuitive control than do conventional keyboards. 
Use of these devices requires only simple software for 
managing hardware communication protocols, but the 
approach to controlling a system via brain waves requires 
more sophisticated software for the interpretation of 
encephalographic data. Although, in the absence of pilot 
studies, it is premature to assume that brain wave sensing is 
capable of conveying complex instructions to a computer, it 
seems plausible that brain waves are capable of conveying 
coarse information. 
In order to establish the feasibility of the concept of 
using brain wave sensing for computer control several 
research questions were addresses. A review of several 
technologies was undertaken in order to evaluate the relative 
merits of each technology to the application. 
Another issue considered in the Phase I research was 
the current status of hardware necessary for measuring brain 
waves. The field of neuromagnetometry is advancing 
rapidly, but is still in its early stages of development. It was 
clear that if conclusions were based on existing 
instrumentation and methods of data analysis the results of 
the Phase I feasibility study would have been that the control 
of systems through brain wave signals was not very 
practical. Therefore, the scope of the Phase I research was 
expanded to include an evaluation of anticipated future 
developments in instrumentation. 
The approach to the Phase I feasibility study 
involved several research techniques. Initially, an extensive 
literature search was conducted to determine the state-of-the- 
art in the application of MEG technology. The literature 
review revealed that MEG had advantages over the 
conventional EEG, however the scope of the MEG was 
limited. 
Electroencephalography (EEG) 
In the domain of the spontaneous EEG only a very 
limited band of EEG activity could be of possible use. 
Alpha activity is of large amplitude and it is strongly 
associated with activity of the visual cortex in the relaxed, 
wakeful but visually inattentive individual. In principle, the 
modulation of alpha by changes in the level of visual 
attention could be used for control of computers. However, 
this would be a very primitive level of control, as the 
changes in level of alpha activity are quite slow as compared 
https://ntrs.nasa.gov/search.jsp?R=19880007835 2020-03-20T07:34:01+00:00Z
to the speed with which a person can type instructions at a 
computer keyboard. To date, studies of the effects of 
changes in level of attention have been based on very simple 
experimental paradigms. The results of these studies do not 
provide conclusive evidence regarding the variables that 
affect the alpha wave form. Therefore, alpha and its 
modulation should be studied further using procedures that 
provide a much greater degree of experimental control over 
the amount and type of mental work being done by the 
subject. 
Magnetoencephalography (MEG) 
Magnetic recording techniques offer several 
advantages for monitoring specific neural activity in the brain 
with regard to computer control. However, some of the 
problems associated with the EEG are also present in MEG. 
Specifically, alpha activity is indeed the strongest MEG 
spontaneous signal, it is not certain precisely how it is 
affected by states of attention, as well as intentions, even 
when these states are under very good experimental control. 
Clearly the problem resides in the cognitive processes of the 
human operator, and not within the recording techniques. 
For example, subjects given a vigilance task in which they 
simply monitor an oscilloscopic display are required to make 
decisions based upon changes in the visual information. The 
relative frequencies with which such decisions must be 
made, the difficulty in making the "correct" decision, the 
properties of the display itself (e.g., spatial frequency 
content, flicker or temporal frequency, moving or static 
targets), and the duration of an experimental session are all 
factors that could affect MEG output. 
E y e B d a s k  Monitor Concept 
A system for monitoring operator task activity can 
now be built around the manual connol operations necessary 
to perform various task procedures. This is possible 
because a computer can easily be made aware of these 
events. A certain degree of task-level ambiguity is inherent 
in such operations but most of these could be resolved with a 
fair degree of certainty by reference to fixed task-domain 
knowledge. However, a more severe criticism can be 
leveled against such a system on the grounds that its results 
are of too coarse a grain. Decisive action is usually taken 
after a considerable period of "silent" mental activity -- 
analytical tasks performed by an operator are not likely to be 
identifiable in the sequence of manual control operations. 
Unfortunately it is just such tasks which are of critical 
importance to decision aiding and intelligent problem solving 
systems. Analytics believes that a system which records and 
correlates human ocular and brain wave activity can bridge 
the gap between isolated manual control operations. 
The examination of ocular activity can clarify what is 
going on during discrete control events. Eye data is 
ambiguous when used to identify an ongoing task: scanning 
out a straight line could as easily mean that the operator was 
tracking a moving object, estimating a path, or even briefly 
glancing from one point to another. Since the effect of 
visual attention and vigilance on brain wave activity is 
extremely robust, it is expected that components within 
specific wave bands can be used to disambiguate ocular 
behavior vis-a-vis operator performance of analytical tasks. 
This is not to say that brain wave data will not itself prove 
ambiguous, for it will undoubtedly do so. The entire task 
identification problem is characterized by the need to resolve 
potential ambiguities and conflicts in and between all the 
various levels and types of available information -- eye 
position, brain wave readings, manual events and fixed 
domain knowledge. 
In order to handle mutual disambiguation, an actual 
system must be capable of passing information both 
upwards and downwards between levels to achieve a "best 
fit" between the low-level information and the high-level task 
hypotheses. This type of processing has been applied 
successfully in the domain of speech recognition (for 
example, in the HEARSAY system), where low-level 
phoneme and word recognition is permitted to interact with 
higher-level notions of syntax and semantics. Errors and 
ambiguities in word recognition can be corrected by 
determining what "makes sense" in the context. This kind of 
approach is generally termed "hypothesize-and-test", since 
there are several independent knowledge sources and the 
interpretation of each can be evaluated against the 
interpretation of the other. Also termed "iterative guess- 
building", the reinterpretation ceases when some 
predetermined level of confidence has been attained for the 
interpretation system as a whole. For the eyebraidtask 
monitor it is expected that eye data, brain wave data, manual 
control data, and knowledge of the mission task domain (a 
task syntax) can be fused to build a continually updated task 
history which can be extended as needed for purposes of 
.prediction. In the context of this application, the feasibility 
of using brain wave information to contribute to computer 
system conml appears highly plausible. 
The successful application of brain wave data to 
intelligent systems revolves around a thorough 
understanding of the complex linkage of task structure, 
operator eye-movement, brain wave response, and task 
syntax. The definition of that linkage at a level sufficiently 
specific to provide the basis for distributed intelligence 
system algorithms requires that a testbed be developed that 
focuses specifically on the issue of eyebraidtask linkage 
(Figure 1). 
Analytics. under contract to NASA, has pioneered 
development of the application of eye-sensing technology to 
computer control and has successfully integrated an 
eydvoice controlled interface into a complex tasklscenario 
generator. This unique system, called OASIS, has been 
refined and demonstrated as a working prototype. OASIS 
will provide a baseline system that will be further refined 
with the addition of brain sensing capability into a functional 
prototype testbed that will focus directly on the issue of 
eye/brain/task linkage. 
TECHNICAL FEASIBILITY OF COMPUTER CONTROL 
In almost every field where computer hardware is 
employed, operator work stations are characterized by 
growing complexity and continuously increasing data flow. 
In general, two major issues are of prime concern: 1) the 
increased operator workload and 2) the reduced habitability 
which typically results when older control technologies are 
extended to support increased functionality. 
94 
Workload problems are believed to be responsible 
for operator errors in critical tasks and more generally for 
reductions in operator effectiveness or productivity. A 
competent workstation design attempts to reduce workload 
by efficiently organizing the entire suite of operator tasks. 
More recently, system developers have begun to focus on 
the possibility of creating additional channels for 
operatorlmachine communication and of redistributing 
workload amss  the resulting range of control options. This 
is of special interest when a continuous control task such as 
steering must be integrated with a variety of other 
operations. Offloading to new control channels os also of 
interest in the context of special environmental conditions 
such as high G forces where normal operator functioning 
may be highly restricted. The development of commercially 
available voice systems is the most obvious result of this 
approach to the issue of operator workload, although other 
technologies such as control by head position are already in 
use. Voice interaction has been of particular interest to 
developers in the aerospace industry where hands-busy and 
eyes-busy operation are common and where workload 
redistribution is an attractive solution. 
Humans and machines are rapidly becoming 
components in distributed intelligence systems where tasks 
are performed cooperatively. When tasks are complex, the 
passive role of "ready servant" requires that operator needs 
be anticipated, much as the nurse attending a surgeon must 
know what is likely to be requested before the request is 
made. As the computer begins to take on a more active role, 
the need for machine knowledge of operator activity and 
intentions becomes essential. Now the machine may need to 
query the operator regarding his actions or plans, as well as 
spontaneously criticize or offer alternative solutions. 
Ideally, a smart system would know when intervention was 
appropriate. By analogy to the situation of human 
cooperation, it is obvious that in all but the most critical 
situations the appropriateness of intervention is dependent on 
an understanding of what the operator is doing or is about to 
do. 
In order to cooperatively solve a problem, humans 
depend on shared knowledge regarding the problem domain 
and available courses of action. Techniques are already 
available for providing machines with this type of 
intelligence. However, humans also depend on observation 
of their partners, frequently utilizing subtle cues to assess the 
significance of more easily recognized actions. For 
example, facial expressions and posture are usually taken as 
indicators of a person's relative satisfaction with the results 
attained by specific goal directed activity. Efficient human 
cooperation requires just this kind of inference in order to 
continuously adjust individual strategies as a problem 
develops over time. Unfortunately, the cues used by 
humans themselves are by no means completely understood 
and many, such as facial expressions, would require major 
research efforts before the sensing equipment itself could be 
developed. In order to provide a machine with the inference 
capabilities required for efficient cooperation, all available 
resources will have to be focused on machine understanding 
of operator behavior. This will require a dedicated, focused 
laboratory facility such as the EBT testbed. 
specific to provide the basis for distributed intelligence 
system algorithms requires that a testbed be developed that 
focuses specifically on the issue of eyebraidtask linkage. 
NEXT STEPS 
The Phase I1 effort will concern itself with the 
development of a prototype EyeBrainask (EBT) testbed, 
and through applied research and development, the 
refinement and optimization of the system. The principal 
objective of the proposed Phase I1 effort is to develop a 
laboratory testbed that will provide a unique capability to 
elicit, control, record, and analyze the relationship of 
operator task loading, operator eye movement, and operator 
brain wave data in a computer system environment. 
Additionally, the testbed will have the capability to serve as 
the vehicle for demonstrating computer control using brain 
waves at a future time. 
ACKNOWLEDGEMENTS 
The research presented in this paper was sponsored 
by NASA under the Small Business Innovative Research 
(SBIR) program. In addition to the authors, major 
contributions to this effort were made by James Deimler and 
James Stokes. The authors would also like to acknowledge 
the contributions of Dr. Lloyd Kaufman and Dr. Samuel J. 
Williamson of New York University. At Analytics, special 
thanks are extended to Ms. Phyllis H. Martin for her 
technical assistance in the pparation of this paper. 
The successful application of brain wave data to 
intelligent systems revolves around a thorough 
understanding of the complex linkage of task structure. 
operator eye-movement, brain wave response, and task 
syntax. The definition of that linkage at a level sufficiently 
95 


Development and experimentation of an eye/brain/task testbed

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server