Advanced Multimodal Solutions for Information Presentation

High-workload, fast-paced, and degraded sensory environments are the likeliest candidates to benefit from multimodal information presentation. For example, during EVA (Extra-Vehicular Activity) and telerobotic operations, the sensory restrictions associated with a space environment provide a major challenge to maintaining the situation awareness (SA) required for safe operations. Multimodal displays hold promise to enhance situation awareness and task performance by utilizing different sensory modalities and maximizing their effectiveness based on appropriate interaction between modalities. During EVA, the visual and auditory channels are likely to be the most utilized with tasks such as monitoring the visual environment, attending visual and auditory displays, and maintaining multichannel auditory communications. Previous studies have shown that compared to unimodal displays (spatial auditory or 2D visual), bimodal presentation of information can improve operator performance during simulated extravehicular activity on planetary surfaces for tasks as diverse as orientation, localization or docking, particularly when the visual environment is degraded or workload is increased. Tactile displays offer a third sensory channel that may both offload information processing effort and provide a means to capture attention when urgently required. For example, recent studies suggest that including tactile cues may result in increased orientation and alerting accuracy, improved task response time and decreased workload, as well as provide self-orientation cues in microgravity on the ISS (International Space Station). An important overall issue is that context-dependent factors like task complexity, sensory degradation, peripersonal vs. extrapersonal space operations, workload, experience level, and operator fatigue tend to vary greatly in complex real-world environments and it will be difficult to design a multimodal interface that performs well under all conditions. As a possible solution, adaptive systems have been proposed in which the information presented to the user changes as a function of taskcontext-dependent factors. However, this presupposes that adequate methods for detecting andor predicting such factors are developed. Further, research in adaptive systems for aviation suggests that they can sometimes serve to increase workload and reduce situational awareness. It will be critical to develop multimodal display guidelines that include consideration of smart systems that can select the best display method for a particular contextsituation.The scope of the current work is an analysis of potential multimodal display technologies for long duration missions and, in particular, will focus on their potential role in EVA activities. The review will address multimodal (combined visual, auditory andor tactile) displays investigated by NASA, industry, and DoD (Dept. of Defense). It also considers the need for adaptive information systems to accommodate a variety of operational contexts such as crew status (e.g., fatigue, workload level) and task environment (e.g., EVA, habitat, rover, spacecraft). Current approaches to guidelines and best practices for combining modalities for the most effective information displays are also reviewed. Potential issues in developing interface guidelines for the Exploration Information System (EIS) are briefly considered

Characterization of Pilot Technique

Skilled pilots often use pulse control when controlling higher order (i.e. acceleration-command) vehicle dynamics. Pulsing does not produce a stick response that resembles what the human Crossover Model predicts. The Crossover Model (CM) assumes the pilot provides compensation necessary (lead or lag) such that the suite of display-human-vehicle approximates an integrator in the region of crossover frequency. However, it is shown that the CM does appear to drive the pilots pulsing behavior in a very predictable manner. Roughly speaking, the pilot generates pulses such that the area under the pulse (pulse amplitude multiplied by pulse width) is approximately equal to area under the hypothetical CM output. This can allow a pilot to employ constant amplitude pulsing so that only the pulse duration (width) is modulated a drastic simplification over the demands of continuous tracking. A pilot pulse model is developed, with which the parameters of the pilots internally-generated CM can be computed in real time for pilot monitoring and display compensation. It is also demonstrated that pursuit tracking may be activated when pulse control is employed

Caution and Warning Alarm Design and Evaluation for NASA CEV Auditory Displays: SHFE Information Presentation Directed Research Project (DRPP) report 12.07

The design of caution-warning signals for NASA s Crew Exploration Vehicle (CEV) and other future spacecraft will be based on both best practices based on current research and evaluation of current alarms. A design approach is presented based upon cross-disciplinary examination of psychoacoustic research, human factors experience, aerospace practices, and acoustical engineering requirements. A listening test with thirteen participants was performed involving ranking and grading of current and newly developed caution-warning stimuli under three conditions: (1) alarm levels adjusted for compliance with ISO 7731, "Danger signals for work places - Auditory Danger Signals", (2) alarm levels adjusted to an overall 15 dBA s/n ratio and (3) simulated codec low-pass filtering. Questionnaire data yielded useful insights regarding cognitive associations with the sounds

Linking the Pilot Structural Model and Pilot Workload

Behavioral models are developed that closely reproduced pulsive control response of two pilots using markedly different control techniques while conducting a tracking task. An intriguing find was that the pilots appeared to: 1) produce a continuous, internally-generated stick signal that they integrated in time; 2) integrate the actual stick position; and 3) compare the two integrations to either issue or cease a pulse command. This suggests that the pilots utilized kinesthetic feedback in order to sense and integrate stick position, supporting the hypothesis that pilots can access and employ the proprioceptive inner feedback loop proposed by Hess's pilot Structural Model. A Pilot Cost Index was developed, whose elements include estimated workload, performance, and the degree to which the pilot employs kinesthetic feedback. Preliminary results suggest that a pilot's operating point (parameter values) may be based on control style and index minimization

Modeling Pilot Pulse Control

In this study, behavioral models are developed that closely reproduced pulsive control response of two pilots from the experimental pool using markedly different control techniques (styles) while conducting a tracking task. An intriguing find was that the pilots appeared to: 1) produce a continuous, internally-generated stick signal that they integrated in time; 2) integrate the actual stick position; and 3) compare the two integrations to issue and cease pulse commands. This suggests that the pilots utilized kinesthetic feedback in order to perceive and integrate stick position, supporting the hypothesis that pilots can access and employ the proprioceptive inner feedback loop proposed by Hess' pilot Structural Model. The Pulse Models used in conjunction with the pilot Structural Model closely recreated the pilot data both in the frequency and time domains during closed-loop simulation. This indicates that for the range of tasks and control styles encountered, the models captured the fundamental mechanisms governing pulsive and control processes. The pilot Pulse Models give important insight for the amount of remnant (stick output uncorrelated with the forcing function) that arises from nonlinear pilot technique, and for the remaining remnant arising from different sources unrelated to tracking control (i.e. neuromuscular tremor, reallocation of cognitive resources, etc.)

Speech Intelligibility Advantages using an Acoustic Beamformer Display

A speech intelligibility test conforming to the Modified Rhyme Test of ANSI S3.2 "Method for Measuring the Intelligibility of Speech Over Communication Systems" was conducted using a prototype 12-channel acoustic beamformer system. The target speech material (signal) was identified against speech babble (noise), with calculated signal-noise ratios of 0, 5 and 10 dB. The signal was delivered at a fixed beam orientation of 135 deg (re 90 deg as the frontal direction of the array) and the noise at 135 deg (co-located) and 0 deg (separated). A significant improvement in intelligibility from 57% to 73% was found for spatial separation for the same signal-noise ratio (0 dB). Significant effects for improved intelligibility due to spatial separation were also found for higher signal-noise ratios (5 and 10 dB)

Using Published HRTFS with Slab3D: Metric-Based Database Selection and Phenomena Observed

Presented at the 20th International Conference on Auditory Display (ICAD2014), June 22-25, 2014, New York, NY.In this paper, two publicly available head-related transfer function (HRTF) database collections are analyzed for use with the open-source slab3d rendering system. After conversion to the slab3d HRTF database format (SLH), a set of visualization tools and a five-step metric-based process are used to select a subset of databases for general use. The goal is to select a limited subset least likely to contain anomalous behavior or measurement error. The described set of open-source tools can be applied to any HRTF database converted to the slab3d format

Auditory Alarm Design for NASA CEV Applications

This monograph reviews current knowledge in the design of auditory caution and warning signals, and sets criteria for development of 'best practices' for designing new signals for NASA's Crew Exploration Vehicle (CEV) and other future spacecraft, as well as for extra-vehicular operations. A design approach is presented that is based upon cross-disciplinary examination of psychoacoustic research, human factors experience, aerospace practices, and acoustical engineering requirements. Existing alarms currently in use with the NASA Space Shuttle flight deck are analyzed and then alternative designs are proposed that are compliant with ISO 7731, ``Danger signals for work places – Auditory Danger Signals'', and that correspond to suggested methods in the literature to insure discrimination and audibility. Future development of auditory ``sonification'' techniques into the design of alarms will allow auditory signals to be extremely subtle, yet extremely useful for indicating trends or root causes of failures. A summary of `best practice' engineering guidelines is given, followed by results of an experiment involving subjective classification of alarms by ten subjects

Information Presentation

The goal of the Information Presentation Directed Research Project (DRP) is to address design questions related to the presentation of information to the crew on flight vehicles, surface landers and habitats, and during extra-vehicular activities (EVA). Designers of displays and controls for exploration missions must be prepared to select the text formats, label styles, alarms, electronic procedure designs, and cursor control devices that provide for optimal crew performance on exploration tasks. The major areas of work, or subtasks, within the Information Presentation DRP are: 1) Controls, 2) Displays, 3) Procedures, and 4) EVA Operations

Speech Synthesis for Data Link: a Study of Overall Quality and Comprehension Effort

This study investigated subjective preference for synthesized “spoken data link” messages to provide initial design guidance for communication displays in the context of NextGen (Next Generation Air Transport System) operations. Ratings of Overall Quality and Comprehension Effort were obtained as a function of voice type, synthesized speech rate, and sentence prosody. Rank-order data analyses showed that both Overall Quality and Comprehension Effort were affected by speech rate: under the “Fast Rate” condition (vs. “Default Rate”), Overall Quality decreases and Comprehension Effort increases. However, the introduction of “Prosodic Emphasis” (pitch and level changes for specific phrases) in Fast Rate sentences produced a relative improvement in both comprehension and quality ratings. For both speaking rates, the introduction of “Prosodic Emphasis” resulted in higher quality ratings and lower comprehension effort ratings. The data suggest that faster speaking rates, which may improve message throughput in a display, may be viable when combined with prosodic emphasis