In certain space-restricted environments, many of the benefits resulting from computer technology have been foregone because of the size, weight, inconvenience, and lack of mobility associated with existing computer interface devices. Accordingly, an effort to develop a highly miniaturized and 'wearable' computer display and control interface device, referred to as the Sensory Integrated Data Interface (SIDI), is underway. The system incorporates a micro-video display that provides data display and ocular tracking on a lightweight headset. Software commands are implemented by conjunctive eye movement and voice commands of the operator. In this initial prototyping effort, various 'off-the-shelf' components have been integrated into a desktop computer and with a customized menu-tree software application to demonstrate feasibility and conceptual capabilities. When fully developed as a customized system, the interface device will allow mobile, 'hand-free' operation of portable computer equipment. It will thus allow integration of information technology applications into those restrictive environments, both military and industrial, that have not yet taken advantage of the computer revolution. This effort is Phase 1 of Small Business Innovative Research (SBIR) Topic number N90-331 sponsored by the Naval Undersea Warfare Center Division, Newport. The prime contractor is Foster-Miller, Inc. of Waltham, MA

Miller, James E.

English

NASA Technical Reports Server

N93--22Z72
MICRO-VIDEO DISPLAY WITH OCULAR TRACKING " "
AND INTERACTIVE VOICE CONTROL
James E. Miller
Naval Undersea Warfare Center Division, Newport
Missiles Division, Code 831
Newport, R! O2841
k
/ )
Mi,g'rRACT
In certain space-restricted environments, many of the benefits resulting from computer technology
have been foregone because of the size, weight, inconvenience, and lack of mobility associated with existing
computer interface devices. Accordingly, an effort to develop a highly miniaturized and "wearable" computer
display and control interface device, referred to as the Sensory Integrated Data Interface (SIDI), is
underway. The system incorporates a micro-video display that provides data display and ocular tracking on a
lightweight headset. Software commands are implemented by conjunctive eye movement and voice
commands of the operator. In this initial prototyping effort, various "off-the-shelf' components have been
integrated into a desktop computer and with a customized menu-tree software application to demonstrate
feasibility and conceptual capabilities. When fully developed as a customized system, the interface device will
allow mobile, "hands-free" operation of portable computer equipment. It will thus allow integration of
information technology applications into those restrictive environments, both military and industrial, that have
not yet taken advantage of the computer revolution. This effort is Phase I of Small Business Innovative
Research (SBIR) Topic #N90-331 sponsored by the Naval Undersea Warfare Center Division, Newport.
The prime contractor is Foster-Miller, Inc. of Waltham, MA.
INTRODUCTION
During the last decade, there has been a rapid development of technologies that support computer
miniaturization. These technologies have significantly reduced computer workstation size and, at the same
time, improved processor speed, memory capacity, and software utilization. It is interesting to note, however,
that functional improvements in the interface devices for computers have been relatively few. With the
exception of the computer mouse/trackball and new "pen-based" input devices, virtually all computer systems
still rely on large and cumbersome interfaces, namely CRT/LCD screen displays and keyboards. There has
been relatively little ergogenic improvement with these interfaces since their 1950s predecessors were
introduced.
Unfortunately, many of the benefits associated with computer technology have been foregone
because of this lack of appropriate interface devices. This is particularly true for certain space-restricted
environments such as aircraft, submarines, and spacecraft. The disadvantages imposed by size, weight, and
inconvenience of existing interfaces as well as the lack of mobility for the operators using them has resulted
in the avoidance of "computer solutions."
The development effort described below seeks to solve this problem by integrating several new "off-
the-shelf' technology products to produce a highly miniaturized, micro-video computer display and control
system which incorporates both interactive ocular tracking and voice control. The goal is to create a
"wearable" interface mounted on a portable headset. Further development will lead to the implementation of
highly miniaturized, portable alternatives to currently used cathode-ray tube/liquid crystal display and
keyboard control devices. The advantages of computer technology will thus become more accessible for
space restricted and operator-mobile applications and environments.
227
https://ntrs.nasa.gov/search.jsp?R=19930012983 2020-03-17T06:35:43+00:00Z
DISCUSSION
A number of emerging technologies are now available which can provide significant improvement for
the human-to-computer interface. Singularly, each one of these technologies has the ability to provide
improvement, but significant synergy is gained by combining them together into a single, integrated system.
The technologies applied in this effort are those of micro-video displays, ocular tracking systems, and
computer voice recognition. Based upon a capability verses cost analysis of commercially available products,
as described in reference (1), the component subsystems described below were selected for use in this
project. Since the interface concept relies on human sensory interaction, the system is referred to as the
Sensory Integrated Data Interface (SIDI)
Once the component subsystems were selected, the effort consisted of assembling them into a
functional interface. On an adjustable headset were mounted a micro-video display, an associated Infra-Red
(IR) illumination source and video camera, and a microphone. The micro-video display presents computer
data to the operator. Using eye-tracking hardware installed in a PC expansion slot, screen cursor movement
is caused to follow eye movement. Software menu selections are then implemented by gaze-f'Lxation timing
and voice command. An analogy to the "point and click" functionality of a computer mouse can be drawn.
The line-of-sight gaze is used to "point"to a menu option and the voice command "clicks on it" or activates
the selected item.
Figure 1 illustrates how the components were assembled on an operator headset. Note that the
display projector/eye-tracker is worn in a monocular arrangement in front of one eye. This allows the
operator to access computer data while operating other equipment or performing other tasks. Note also that
the use of the headset allows "hands-free" operation of the computer. Figure 2 illustrates a high level
component integration scheme for the desktop computer. A customized software application was developed
to demonstrate conceptual capabilities. As shown in Figure 3, it consists of a menu-tree listing of topics
which the operator activates by gazing and, when ocular lock is achieved, initiates with a verbal command.
Running underneath this application are the customized software device drivers for the system components.
Micro-Video Display
Data display for SIDI is provided by the "Private Eye" developed by Reflection Technology, Inc.
The "Private Eye" uses a new proprietary technology which uniquely combines conventional semiconductor
and optical techniques to create an image of a 12 inch monitor in a miniature package measuring only 1.1
inches X 1.2 inches X 3.2 inches. It weighs less than two ounces. The display uses emissive elements to
provide a high definition, high contrast, and fast responding image viewable in bright daylight.
Worn in a monocular position in front of one eye, the unit interfaces with a PC computer via an
interface card in an expansion slot. A virtual image of the computer CRT display is projected approximately
18 inches in front of the viewer's eye. The unit displays 720 X 280 pixels which can be formatted as 25 lines
with 80 characters per line or can be used to show bit-mapped graphics. The monochrome image appears to
float in space in front of the viewer with a quality and resolution matching that of a standard display. A lens
system allows image focusing to accommodate viewing with or without eye glasses.
Ocular Tracking System
The subsystem chosen to provide line-of-sight ocular tracking for the SIDI was the Eye Slaved
Pointing (ESP) System from ISCAN, Inc. It is available as a PC compatible, turnkey eye movement
monitoring system that serves as a hybrid interface device. Terminate and Stay Resident (TSR) software
developed for the ESP is designed to substitute an operator's line-of-sight gaze for standard computer
pointing devices. The system provides cursor positional data to software applications which permit or require
a pointing device.
228
The eye-tracker functions by using a custom set of integrated circuits and tracking algorithms to lock
on and track a point on the operator's eye in real-time. This is accomplished by measuring and tracking a
low-level Infra-Red (IR) light beam reflected from the eye surface. A miniature IR video camera is used to
acquire the pupil and corneal images needed by the tracking algorithms. The operator views a series of
positional data points on a display to calibrate the tracking function for the display. From that point on, the
tracking function converts eye movement into screen cursor movement. Since the display and eye-tracking
camera are positionally coupled by the headset, movement of the head does not affect the tracking function.
Both the ESP and Private Eye are mounted so as to be adjustable on the headset and allow variable
positioning of the components for different operators.
Voice Reco_tion Unit
The Voice Recognition Unit (VRU) chosen to provide voice control for the SIDI was the Voice
Master from Covox, Inc. The system consists of a control unit, speaker, microphone, and software which can
provide both speech recognition and generation capability for a computer using an available RS-232 port.
The operator trains the system to accept specific voice commands which implement menu options once the
ESP has achieved ocular lock on a menu option.
VRUs allow a computer to translate any spoken language into accurate, intelligent commands. It
does so by breaking down a spoken command into its frequency components and then comparing those
components with those of pre-stored commands. Most language requirements for a computer can be carried
out with a relatively small vocabulary when combined with gaze-directed screen interaction. Within 10 - 15
minutes, an experienced operator can fully train the voice unit to recognize the required sets of alpha-
numeric commands/inputs.
While voice input alone can be used to select items from a display menu, there are many situations
where it does not perform well. For example, voice commands are poor for most interactions with graphic
images, like pointing to a specific wire in a schematic or selecting a location to insert information in a text
display. Even with text applications, voice interaction is difficult when selecting one item from a display
which has many similar items, like a stock inventory sheet or a page of text. Eye-directed interaction in
conjunction with voice commands handles these situations quickly and naturally.
S0ftwar¢
A customized software application was developed to demonstrate functional and conceptual
capabilities of the SIDI. The software consists of both the user application and the underlying TSR that
handles hardware interrupts. As shown in Figure 3, the user application consists of a menu tree that allows
the operator to access various types of documentation for submarine system and subsystem components.
The operator makes a selection by gazing at a major topic item. When the SIDI achieves ocular lock-on, the
selection flashes at the operator signifying that verbal commands can be initiated. Lock-on is achieved
quickly enough so that the operator perceives that the system is operating concurrently with his thought
processes. By giving various commands such as _ACCEPT', "UP', "DOWN', and "BACK" the operator can
move around the menu to view desired data. The software was written in the C programming language with
some in-line assembly code.
SID! APPLICATIONS
There is a wide spectrum of applications for the SID! or a SID! derivative. They range from various
consumer entertainment products to business applications to military documentation applications. Any
current computer application that could benefit from operator mobility is a candidate SIDI application.
Figure 4 illustrates an application for a technician performing a maintenance routine on a cruise missile using
229
procedural documentation displayed by a SIDI system. A few of the many possible implementations that
could be envisioned are as follows:
Paperless technical manuals
Equipment diagnostic/repair systems
Personalized training systems
Entertainment electronics
Portable inventory management systems
Procedural documentation in space-restricted areas
Computer interface for the physically handicapped
FUTURE WORK
The goal of this project was to prototype existing commercial off-the-shelf hardware to demonstrate
the feasibility of the SIDI concept. That goal was achieved. A functional prototype system was developed
that successfully allows hands-free operation of a computer. The next step, as described in reference (2), is
to refine the concept by developing a more compact and optimized interface device that provides operator
mobility, requires less hardware adjustment to accommodate different operators, and is compatible with
commercial software applications.
To maximize operator mobility, it is proposed that the interface be developed to serve as a remote
terminal served by a computer base station. The user module will contain the minimum amount of
electronics required to drive the display, eye tracking camera, and microphone. The base station will provide
all the intelligence for these components via a data rink. The data link will be preferably an un-tethered
technology, such as an Infra-Red (IR) spacial optical link, thereby providing the operator unrestricted move-
ment.
To reduce the amount of component adjustment for each individual who operates the interface, the
headset hardware will be miniaturized and integrated into a single, easy-to-use, composite unit. The
hardware will be optimized for
Ease of alignment to obtain the eye image
Ease of calibration to the user's eye
Better tolerance to room lighting variations
Better display resolution
A separation of component functions to allow base station interaction
Finally, to preclude the need for customized software applications, the output of the eye tracking
system will be modified to reflect standard computer mouse and trackball conventions. This will allow the
use of standard software interface environments.
1.
.
REFERENCES
Kendrick, W. K. "Component Survey and System Definition Report," NAV-9371, Contract N66604-
91-C-0936, Foster-Miller, Inc. March 25, 1992.
Nappi, B. "Sensory Integrated Data Interface (SIDI)," Small Business Innovative Research (SBIR)
Phase II Proposal, Foster-Miller, Inc. August 21, 1992.
23O
EYE IMAGING
CAMERA
HEAD
ADJUSTABLE
MIRROR ARM
ADJUSTABLE
ILLUMINATOR
ARM
CAMERA
MOUNTING
TRACK
HEADBAND
ASSEMBLY
VIRTUAL
DISPLAY
IR
ILLUMINATOR
FIGURE 1
231
ISCAN
EYE TRACKER
PC BUS
I EYE SCANNER
I SOFTWARE I
1 RS232
COVOX
VOICE MASTER
I MICROPHONE
FIGURE 2
232
EYE <
SELECTIONS
VOICE <,
SEL EC TIONS I
REPAIR
TORPEDO RADIO MISSILE
1
REACTOR
I
SONAR
I
Manual Schematic PartsList
] Drawings Inventory
f IManual Schematic PartsList
Drawings Inventory
,[Manual Schematic
Drawings
i i I IManual Schematic PartsList Manual Schematic PartsList
Drawings Inventory Drawings Inventory
1
PartsList
Inventory
EYE <
SELECTIONS
VOICE <
SELECTIONS
m
1
Manual
TRAINING
L
I 1 I I I
TORPEDO RADIO MISSILE REACTOR SONAR
I[I,I ' 'Schematic PadsList Manual Schematic PartsList Manual
Drawings Inventory Drawings Inventory Drawings
I I I I IManual Schematic Parts List Manual Schematic Parts List
I
Drawings Inventory Drawings Inventory
I ISchematic PartsList
Inventory
FIGURE 3 233
FIGURE 4 234


Micro-video display with ocular tracking and interactive voice control

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server