Passive Control Architectures for Collaborative Virtual Haptic Interaction and Bilateral Teleoperation over Unreliable Packet-Switched Digital Network

This PhD dissertation consists of two major parts: collaborative haptic interaction (CHI) and bilateral teleoperation over the Internet. For the CHI, we propose a novel hybrid peer-to-peer (P2P) architecture including the shared virtual environment (SVE) simulation, coupling between the haptic device and VE, and P2P synchronization control among all VE copies. This framework guarantees the interaction stability for all users with general unreliable packet-switched communication network which is the most challenging problem for CHI control framework design. This is achieved by enforcing our novel \emph{passivity condition} which fully considers time-varying non-uniform communication delays, random packet loss/swapping/duplication for each communication channel. The topology optimization method based on graph algebraic connectivity is also developed to achieve optimal performance under the communication bandwidth limitation. For validation, we implement a four-user collaborative haptic system with simulated unreliable packet-switched network connections. Both the hybrid P2P architecture design and the performance improvement due to the topology optimization are verified. In the second part, two novel hybrid passive bilateral teleoperation control architectures are proposed to address the challenging stability and performance issues caused by the general Internet communication unreliability (e.g. varying time delay, packet loss, data duplication, etc.). The first method--Direct PD Coupling (DPDC)--is an extension of traditional PD control to the hybrid teleoperation system. With the assumption that the Internet communication unreliability is upper bounded, the passive gain setting condition is derived and guarantees the interaction stability for the teleoperation system which interacts with unknown/unmodeled passive human and environment. However, the performance of DPDC degrades drastically when communication unreliability is severe because its feasible gain region is limited by the device viscous damping. The second method--Virtual Proxy Based PD Coupling (VPDC)--is proposed to improve the performance while providing the same interaction stability. Experimental and quantitative comparisons between DPDC and VPDC are conducted, and both interaction stability and performance difference are validated

Cyber security threats and challenges in collaborative mixed-reality

Collaborative Mixed-Reality (CMR) applications are gaining interest in a wide range of areas including games, social interaction, design and health-care. To date, the vast majority of published work has focused on display technology advancements, software, collaboration architectures and applications. However, the potential security concerns that affect collaborative platforms have received limited research attention. In this position paper, we investigate the challenges posed by cyber-security threats to CMR systems. We focus on how typical network architectures facilitating CMR and how their vulnerabilities can be exploited by attackers, and discuss the degree of potential social, monetary impacts, psychological and other harms that may result from such exploits. The main purpose of this paper is to provoke a discussion on CMR security concerns. We highlight insights from a cyber-security threat modelling perspective and also propose potential directions for research and development toward better mitigation strategies. We present a simple, systematic approach to understanding a CMR attack surface through an abstraction-based reasoning framework to identify potential attack vectors. Using this framework, security analysts, engineers, designers and users alike (stakeholders) can identify potential Indicators of Exposures (IoE) and Indicators of Compromise (IoC). Our framework allows stakeholders to reduce their CMR attack surface as well understand how Intrusion Detection System (IDS) approaches can be adopted for CMR systems. To demonstrate the validity to our framework, we illustrate several CMR attack surfaces through a set of use-cases. Finally, we also present a discussion on future directions this line of research should take

Human Performance and Cognitive Workload in Haptic, Audio and Visual Environments

The ability to efficiently perform a task in a human-in-the-loop system and in multi-sensory virtual environments is highly dependent on the type of sensory feedback the operator is receiving and the amount of workload the operator is exposed to. Despite the vast amount of research on Collaborative Virtual Environments (CVEs) and Human Machine Interactions (HMIs), little is known about what type of feedback increases the performance of a human operator and what type of sensory feedback minimizes the amount of workload the operator is exposed to. While individual differences influence human performance outcomes, the physiological processes a human being set the fundamental guidelines for assessing human performance. The purpose of this study is to evaluate the performance of participants for a combination of sensory two feedback modes (audio-visual, haptic-visual or audio-haptic) in a primary task to find the optimum feedback model for CVE and HMI applications. A concurrent secondary task is also designed to evaluate workload of each feedback mode (audio, haptic or visual) and the effect of different levels of workload on task completion time and task accuracy. For example, a car driver performs a primary task by steering the car in the correct direction. A secondary task, in the same context, would be monitoring the fuel level or checking the speed limit. In the primary task, participants are required to press a virtual button from a set of three (right button, left button or up button). The secondary task evaluates the amount of workload the participant is exposed to in three different feedback modes (haptic, audio or visual). Each participant is required to recognize a Morse code. In this study, participants perform three trials. In first trial, participants perform one task the primary task alone. In the second and third trials, participants perform the primary task and the secondary task concurrently. The primary task evaluates human performance and includes combined sensory modalities as a feedback mode (audio-visual, haptic-visual or audio-haptic). The time it takes the participant to press the virtual button (primary task response time), the number of correct button presses (primary task accuracy), the time it takes the participant to recognize the Morse code (secondary task response time) and the number of the correct codes (secondary task accuracy) are all collected. In addition, NASA Task Load Index (TLX) questionnaire is used after each trial to assess the subjective performance and subjective workload of participants. The data collected is tested for normality using Lilliefors test, filtered using Grubb’s test to eliminate outlying data and analyzed using one-way ANOVA and multiple two-sample t-tests. A Tukey HSD is also used to show the differences between experimental conditions. The results of this study indicate that the hypothesis that all combinations of feedback provide the same performance can be rejected for the primary task response time. For instance, the results show that the there is a difference in response time between the audio-haptic and the audio-visual feedback modes in the first, second and third trials. The results of this study also indicate that the hypothesis that all sensory feedback modes provide the same workload can be rejected for the secondary task accuracy. Results show that there is a difference between haptic and auditory conditions and shows that visual condition has a lower accuracy than the other feedback modes

Enhancing the E-Commerce Experience through Haptic Feedback Interaction

The sense of touch is important in our everyday lives and its absence makes it difficult to explore and manipulate everyday objects. Existing online shopping practice lacks the opportunity for physical evaluation, that people often use and value when making product choices. However, with recent advances in haptic research and technology, it is possible to simulate various physical properties such as heaviness, softness, deformation, and temperature. The research described here investigates the use of haptic feedback interaction to enhance e-commerce product evaluation, particularly haptic weight and texture evaluation. While other properties are equally important, besides being fundamental to the shopping experience of many online products, weight and texture can be simulated using cost-effective devices. Two initial psychophysical experiments were conducted using free motion haptic exploration in order to more closely resemble conventional shopping. One experiment was to measure weight force thresholds and another to measure texture force thresholds. The measurements can provide better understanding of haptic device limitation for online shopping in terms of the availability of different stimuli to represent physical products. The outcomes of the initial psychophysical experimental studies were then used to produce various absolute stimuli that were used in a comparative experimental study to evaluate user experience of haptic product evaluation. Although free haptic exploration was exercised on both psychophysical experiments, results were relatively consistent with previous work on haptic discrimination. The threshold for weight force discrimination represented as downward forces was 10 percent. The threshold for texture force discrimination represented as friction forces was 14.1 percent, when using dynamic coefficient of friction at any level of static coefficient of friction. On the other hand, the comparative experimental study to evaluate user experience of haptic product information indicated that haptic product evaluation does not change user performance significantly. However, although there was an increase in the time taken to complete the task, the number of button click actions tended to decrease. The results showed that haptic product evaluation could significantly increase the confidence of shopping decision. Nevertheless, the availability of haptic product evaluation does not necessarily impose different product choices but it complements other selection criteria such as price and appearance. The research findings from this work are a first step towards exploring haptic-based environments in e-commerce environments. The findings not only lay the foundation for designing online haptic shopping but also provide empirical support to research in this direction

Effective cooperative haptic interaction over the Internet

We present a system that enables, for the first time, effective transatlantic cooperative haptic manipulation of objects whose motion is computed using a physically-based model. We propose a technique for maintaining synchrony between simulations in a peer-to-peer system, while providing responsive direct manipulation for all users. The effectiveness of this approach is determined through extensive user trials involving concurrent haptic manipulation of a shared object. A CAD assembly task, using physically-based motion simulation and haptic feedback, was carried out between the USA and the UK with network latencies in the order of 120ms. We compare the effects of latency on synchrony between peers over the Internet with a low latency (0.5ms) local area network. Both quantitatively and qualitatively, when using our technique, the performance achieved over the Internet is comparable to that on a LAN. As such, this technique constitutes a significant step forward for distributed haptic collaboration