Quantifying Cognitive Efficiency of Display in Human-Machine Systems

As a side effect of fast growing informational technology, information overload becomes prevalent in the operation of many human-machine systems. Overwhelming information can degrade operational performance because it imposes large mental workload on human operators. One way to address this issue is to improve the cognitive efficiency of display. A cognitively efficient display should be more informative while demanding less mental resources so that an operator can process larger displayed information using their limited working memory and achieve better performance. In order to quantitatively evaluate this display property, a Cognitive Efficiency (CE) metric is formulated as the ratio of the measures of two dimensions: display informativeness and required mental resources (each dimension can be affected by display, human, and contextual factors). The first segment of the dissertation discusses the available measurement techniques to construct the CE metric and initially validates the CE metric with basic discrete displays. The second segment demonstrates that displays showing higher cognitive efficiency improve multitask performance. This part also identifies the version of the CE metric that is the most predictive of multitask performance. The last segment of the dissertation applies the CE metric in driving scenarios to evaluate novel speedometer displays; however, it finds that the most efficient display may not better enhance concurrent tracking performance in driving. Although the findings of dissertation show several limitations, they provide valuable insight into the complicated relationship among display, human cognition, and multitask performance in human-machine systems

Haptic wearables as sensory replacement, sensory augmentation and trainer - a review

Sensory impairments decrease quality of life and can slow or hinder rehabilitation. Small, computationally powerful electronics have enabled the recent development of wearable systems aimed to improve function for individuals with sensory impairments. The purpose of this review is to synthesize current haptic wearable research for clinical applications involving sensory impairments. We define haptic wearables as untethered, ungrounded body worn devices that interact with skin directly or through clothing and can be used in natural environments outside a laboratory. Results of this review are categorized by degree of sensory impairment. Total impairment, such as in an amputee, blind, or deaf individual, involves haptics acting as sensory replacement; partial impairment, as is common in rehabilitation, involves haptics as sensory augmentation; and no impairment involves haptics as trainer. This review found that wearable haptic devices improved function for a variety of clinical applications including: rehabilitation, prosthetics, vestibular loss, osteoarthritis, vision loss and hearing loss. Future haptic wearables development should focus on clinical needs, intuitive and multimodal haptic displays, low energy demands, and biomechanical compliance for long-term usage

Vibrotactile Sensory Augmentation and Machine Learning Based Approaches for Balance Rehabilitation

Vestibular disorders and aging can negatively impact balance performance. Currently, the most effective approach for improving balance is exercise-based balance rehabilitation. Despite its effectiveness, balance rehabilitation does not always result in a full recovery of balance function. In this dissertation, vibrotactile sensory augmentation (SA) and machine learning (ML) were studied as approaches for further improving balance rehabilitation outcomes. Vibrotactile SA provides a form of haptic cues to complement and/or replace sensory information from the somatosensory, visual and vestibular sensory systems. Previous studies have shown that people can reduce their body sway when vibrotactile SA is provided; however, limited controlled studies have investigated the retention of balance improvements after training with SA has ceased. The primary aim of this research was to examine the effects of supervised balance rehabilitation with vibrotactile SA. Two studies were conducted among people with unilateral vestibular disorders and healthy older adults to explore the use of vibrotactile SA for therapeutic and preventative purposes, respectively. The study among people with unilateral vestibular disorders provided six weeks of supervised in-clinic balance training. The findings indicated that training with vibrotactile SA led to additional body sway reduction for balance exercises with head movements, and the improvements were retained for up to six months. Training with vibrotactile SA did not lead to significant additional improvements in the majority of the clinical outcomes except for the Activities-specific Balance Confidence scale. The study among older adults provided semi-supervised in-home balance rehabilitation training using a novel smartphone balance trainer. After completing eight weeks of balance training, participants who trained with vibrotactile SA showed significantly greater improvements in standing-related clinical outcomes, but not in gait-related clinical outcomes, compared with those who trained without SA. In addition to investigating the effects of long-term balance training with SA, we sought to study the effects of vibrotactile display design on people’s reaction times to vibrational cues. Among the various factors tested, the vibration frequency and tactor type had relatively small effects on reaction times, while stimulus location and secondary cognitive task had relatively large effects. Factors affected young and older adults’ reaction times in a similar manner, but with different magnitudes. Lastly, we explored the potential for ML to inform balance exercise progression for future applications of unsupervised balance training. We mapped body motion data measured by wearable inertial measurement units to balance assessment ratings provided by physical therapists. By training a multi-class classifier using the leave-one-participant-out cross-validation method, we found approximately 82% agreement among trained classifier and physical therapist assessments. The findings of this dissertation suggest that vibrotactile SA can be used as a rehabilitation tool to further improve a subset of clinical outcomes resulting from supervised balance rehabilitation training. Specifically, individuals who train with a SA device may have additional confidence in performing balance activities and greater postural stability, which could decrease their fear of falling and fall risk, and subsequently increase their quality of life. This research provides preliminary support for the hypothesized mechanism that SA promotes the central nervous system to reweight sensory inputs. The preliminary outcomes of this research also provide novel insights for unsupervised balance training that leverage wearable technology and ML techniques. By providing both SA and ML-based balance assessment ratings, the smart wearable device has the potential to improve individuals’ compliance and motivation for in-home balance training.PHDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/143901/1/baotian_1.pd

Design and Assessment of Vibrotactile Biofeedback and Instructional Systems for Balance Rehabilitation Applications.

Sensory augmentation, a type of biofeedback, is a technique for supplementing or reinforcing native sensory inputs. In the context of balance-related applications, it provides users with additional information about body motion, usually with respect to the gravito-inertial environment. Multiple studies have demonstrated that biofeedback, regardless of the feedback modality (i.e., vibrotactile, electrotactile, auditory), decreases body sway during real-time use within a laboratory setting. However, in their current laboratory-based form, existing vibrotactile biofeedback devices are not appropriate for use in clinical and/or home-based rehabilitation settings due to the expense, size, and operating complexity of the instrumentation required. This dissertation describes the design, development, and preliminary assessment of two technologies that support clinical and home-based balance rehabilitation training. The first system provides vibrotactile-based instructional motion cues to a trainee based on the measured difference between the expert’s and trainee’s motions. The design of the vibrotactile display is supported by a study that characterizes the non-volitional postural responses to vibrotactile stimulation applied to the torso. This study shows that vibration applied individually by tactors over the internal oblique and erector spinae muscles induces a postural shift of the order of one degree oriented in the direction of the stimulation. Furthermore, human performance is characterized both experimentally and theoretically when the expert–trainee error thresholds and nature of the control signal are varied. The results suggest that expert–subject cross-correlation values were maximized and position errors and time delays were minimized when the controller uses a 0.5 error threshold and proportional plus derivative feedback control signal, and that subject performance decreases as motion speed and complexity increase. The second system provides vibrotactile biofeedback about body motion using a cell phone. The system is capable of providing real-time vibrotactile cues that inform corrective trunk tilt responses. When feedback is available, both healthy subjects and those with vestibular involvement significantly reduce their anterior-posterior or medial-lateral root-mean-square body sway, have significantly smaller elliptical area fits to their sway trajectory, spend a significantly greater mean percentage time within the no feedback zone, and show a significantly greater A/P or M/L mean power frequency.Ph.D.Mechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/91546/1/channy_1.pd

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 13th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2022, held in Hamburg, Germany, in May 2022. The 36 regular papers included in this book were carefully reviewed and selected from 129 submissions. They were organized in topical sections as follows: haptic science; haptic technology; and haptic applications

Head-mounted Sensory Augmentation System for Navigation in Low Visibility Environments

Sensory augmentation can be used to assist in some tasks where sensory information is limited or sparse. This thesis focuses on the design and investigation of a head-mounted vibrotactile sensory augmentation interface to assist navigation in low visibility environments such as firefighters’ navigation or travel aids for visually impaired people. A novel head-mounted vibrotactile interface comprising a 1-by-7 vibrotactile display worn on the forehead is developed. A series of psychophysical studies is carried out with this display to (1) determine the vibrotactile absolute threshold, (2) investigate the accuracy of vibrotactile localization, and (3) evaluate the funneling illusion and apparent motion as sensory phenomena that could be used to communicate navigation signals. The results of these studies provide guidelines for the design of head-mounted interfaces. A 2nd generation head-mounted sensory augmentation interface called the Mark-II Tactile Helmet is developed for the application of firefighters’ navigation. It consists of a ring of ultrasound sensors mounted to the outside of a helmet, a microcontroller, two batteries and a refined vibrotactile display composed of seven vibration motors based on the results of the aforementioned psychophysical studies. A ‘tactile language’, that is, a set of distinguishable vibrotactile patterns, is developed for communicating navigation commands to the Mark-II Tactile Helmet. Four possible combinations of two command presentation modes (continuous, discrete) and two command types (recurring, single) are evaluated for their effectiveness in guiding users along a virtual wall in a structured environment. Continuous and discrete presentation modes use spatiotemporal patterns that induce the experience of apparent movement and discrete movement on the forehead, respectively. The recurring command type presents the tactile command repeatedly with an interval between patterns of 500 ms while the single command type presents the tactile command just once when there is a change in the command. The effectiveness of this tactile language is evaluated according to the objective measures of the users’ walking speed and the smoothness of their trajectory parallel to the virtual wall and subjective measures of utility and comfort employing Likert-type rating scales. The Recurring Continuous (RC) commands that exploit the phenomena of apparent motion are most effective in generating efficient routes and fast travel, and are most preferred. Finally, the optimal tactile language (RC) is compared with audio guidance using verbal instructions to investigate effectiveness in delivering navigation commands. The results show that haptic guidance leads to better performance as well as lower cognitive workload compared to auditory feedback. This research demonstrates that a head-mounted sensory augmentation interface can enhance spatial awareness in low visibility environments and could help firefighters’ navigation by providing them with supplementary sensory information

ISMCR 1994: Topical Workshop on Virtual Reality. Proceedings of the Fourth International Symposium on Measurement and Control in Robotics

This symposium on measurement and control in robotics included sessions on: (1) rendering, including tactile perception and applied virtual reality; (2) applications in simulated medical procedures and telerobotics; (3) tracking sensors in a virtual environment; (4) displays for virtual reality applications; (5) sensory feedback including a virtual environment application with partial gravity simulation; and (6) applications in education, entertainment, technical writing, and animation

Informative Vibrotactile Displays to Support Attention and Task Management in Anesthesiology.

The task set of an anesthesiologist, like that of operators in many complex, data-rich domains, requires effective management of attention, which must be divided among multiple tasks and task-relevant data sources. The inefficient allocation of attentional resources can lead to errors in monitoring a patient’s physiology, which constitute a significant portion of preventable medical errors. To better support attention management and multitasking performance without additionally loading the visual or auditory channels, this dissertation describes work to develop novel “continuously-informing” vibrotactile displays of physiological data. These displays use coded vibration patterns to communicate blood pressure and respiration data in real time. A theory-based approach was taken in the design of these displays to support the properties of “preattentive reference”: the signals can be processed in parallel without interfering with ongoing tasks, include partial information to support efficient task-switching, and can be processed in a mentally economical way. A series of research activities identified: 1) types of information that could best support anesthesiologists in task management decisions; 2) how to display this information via vibrotactile signals in ways that minimize perceptual interference from effects such as vibrotactile adaptation, masking, and tactile “change blindness”; 3) how to encode the information in vibrotactile patterns to minimize interference with concurrent tasks at cognitive processing stages; and 4) mappings between signal modulations and the represented data that best support economical processing. An evaluation study, set in a high-fidelity clinical simulation, showed substantial improvements in anesthesiologists’ multitasking performance, including faster detection and correction of serious health events, and fewer unnecessary interruptions of ongoing tasks with continuously-informing tactile displays, when compared to performance with traditional (visual/auditory) display configurations. This work contributes to theories and models of tactile and multimodal information processing, specifically concerning the performance effects of perceptual and cognitive interferences when information is processed via two or more sensory channels concurrently. It also demonstrates how a vibrotactile display designed to support properties of preattentive reference can improve attention management and multitask performance, thus showing promise for reducing the prevalence of monitoring errors and system awareness issues in anesthesiology and other complex, data rich domains.Ph.D.Industrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/78911/1/ferrist_1.pd