GazeSwitch : Automatic Eye-Head Mode Switching for Optimised Hands-Free Pointing

This paper contributes GazeSwitch, an ML-based technique that optimises the real-time switching between eye and head modes for fast and precise hands-free pointing. GazeSwitch reduces false positives from natural head movements and efficiently detects head gestures for input, resulting in an effective hands-free and adaptive technique for interaction. We conducted two user studies to evaluate its performance and user experience. Comparative analyses with baseline switching techniques, Eye+Head Pinpointing (manual) and BimodalGaze (threshold-based) revealed several trade-offs. We found that GazeSwitch provides a natural and effortless experience but trades off control and stability compared to manual mode switching, and requires less head movement compared to BimodalGaze. This work demonstrates the effectiveness of machine learning approach to learn and adapt to patterns in head movement, allowing us to better leverage the synergistic relation between eye and head input modalities for interaction in mixed and extended reality

Motion correlation: selecting objects by matching their movement

Selection is a canonical task in user interfaces, commonly supported by presenting objects for acquisition by pointing. In this article, we consider motion correlation as an alternative for selection. The principle is to represent available objects by motion in the interface, have users identify a target by mimicking its specific motion, and use the correlation between the system’s output with the user’s input to determine the selection. The resulting interaction has compelling properties, as users are guided by motion feedback, and only need to copy a presented motion. Motion correlation has been explored in earlier work but only recently begun to feature in holistic interface designs. We provide a first comprehensive review of the principle, and present an analysis of five previously published works, in which motion correlation underpinned the design of novel gaze and gesture interfaces for diverse application contexts. We derive guidelines for motion correlation algorithms, motion feedback, choice of modalities, overall design of motion correlation interfaces, and identify opportunities and challenges identified for future research and design

Classifying Head Movements to Separate Head-Gaze and Head Gestures as Distinct Modes of Input

Head movement is widely used as a uniform type of input for human-computer interaction. However, there are fundamental differences between head movements coupled with gaze in support of our visual system, and head movements performed as gestural expression. Both Head-Gaze and Head Gestures are of utility for interaction but differ in their affordances. To facilitate the treatment of Head-Gaze and Head Gestures as separate types of input, we developed HeadBoost as a novel classifier, achieving high accuracy in classifying gaze-driven versus gestural head movement (F1-Score: 0.89). We demonstrate the utility of the classifier with three applications: gestural input while avoiding unintentional input by Head-Gaze; target selection with Head-Gaze while avoiding Midas Touch by head gestures; and switching of cursor control between Head-Gaze for fast positioning and Head Gesture for refinement. The classification of Head-Gaze and Head Gesture allows for seamless head-based interaction while avoiding false activation

MEEC: Second workshop on momentary emotion elicitation and capture

Recognizing human emotions and responding appropriately has the potential to radically change the way we interact with technology. However, to train machines to sensibly detect and recognize human emotions, we need valid emotion ground truths. A fundamental challenge here is the momentary emotion elicitation and capture (MEEC) from individuals continuously and in real-time, without adversely affecting user experience nor breaching ethical standards. In this virtual half-day CHI 2021 workshop, we will (1) have participant talks and an inspirational keynote presentation (2) ideate elicitation, sensing, and annotation techniques (3) create mappings of when to apply an elicitation method

Gaze-Based Intention Recognition for Human-Agent Collaboration: Towards Nonverbal Communication in Human-AI Interaction

© 2020 Joshua Jiun Wei NewnHuman-agent collaboration has repeatedly been proposed over the decades as a way forward to leverage the strengths of artificial intelligence. As it has become common for humans to work and play alongside intelligent agents, it is increasingly imperative to improve the capacity of agents to interact with their human counterparts socially, naturally and effectively. However, current agents are still limited in their capacity to recognise nonverbal signals and cues, which in turn, limits their capabilities for natural interaction. This thesis addresses this limitation by investigating how artificial agents might support humans in real-time collaboration, given the increased capacity to recognise human intentions afforded by processing their gaze data in real-time. We hypothesise that a socially interactive agent with an increased capacity to recognise intentions can drastically improve its interactive capability, such as by adapting its recommendations to their anticipated intentions as well as to the intentions of others. Using a scenario-based based design approach, we designed five studies to inform and evaluate the different capabilities of a collaborative gaze-enabled intention-aware artificial agent. In Studies 1 and 2, we first evaluate the capacity of human subjects to perform intention recognition using gaze visualisation and its corresponding effects in a competitive gameplay setting. The findings showed that humans players could improve their capacity to infer their intentions of their opponent when shown a live visualisation of their opponent's gaze throughout the game. However, this capacity can be hampered when the opponent was aware that their gaze was being watched. The findings further indicate that humans have a limited capacity in performing gaze-based intention recognition, suggesting that the task may be more suitable for an artificial agent that is trained to process the rich multimodal information available in our setting. In Study 3, we present the implementation details and evaluation of a gaze-enabled intention-aware artificial agent, developed as part of this thesis, that incorporates gaze into its intention recognition process. The evaluation, which uses the data from the previous two studies, demonstrates that incorporating gaze into the agent's planning process not only increases the agent's capacity to recognise intentions but also that it performs better overall than human subjects. In Studies 4 and 5, we operationalise the artificial agent by first giving the agent both the ability to communicate intentions of their opponent to its human collaborator and to explain its reasoning process if required. Subsequently, we evaluated the experience of the players playing the game with and without the assistance of the agent, which ultimately provided insights into how we can further improve the interaction between the human and an intention-aware artificial agent. The findings in this thesis resulted in three contributions towards the understanding of how artificial agents can support human-agent collaboration, given the ability increased capacity to recognise intentions with eye-tracking. The findings from Studies 1, 2 and 3 extend the relationship between gaze awareness and intention, by demonstrating that gaze when tracked over time, can lead to the detection of distal intentions (i.e. long-term intentions that often require several steps to be fulfilled). Following, Studies 3, 4 and 5 contribute to the design of a collaborative gaze-enabled intention-aware artificial agent, and the demonstration of increased situation awareness through gaze awareness for human-agent collaboration. Overall, the thesis highlights the importance of incorporating nonverbal communication in human-AI interaction

Combining Low and Mid-Level Gaze Features for Desktop Activity Recognition

Human activity recognition (HAR) is an important research area due to its potential for building context-aware interactive systems. Though movement-based activity recognition is an established area of research, recognising sedentary activities remains an open research question. Previous works have explored eye-based activity recognition as a potential approach for this challenge, focusing on statistical measures derived from eye movement properties---low-level gaze features---or some knowledge of the Areas-of-Interest (AOI) of the stimulus---high-level gaze features. In this paper, we extend this body of work by employing the addition of mid-level gaze features; features that add a level of abstraction over low-level features with some knowledge of the activity, but not of the stimulus. We evaluated our approach on a dataset collected from 24 participants performing eight desktop computing activities. We trained a classifier extending 26 low-level features derived from existing literature with the addition of 24 novel candidate mid-level gaze features. Our results show an overall classification performance of 0.72 (F1-Score), with up to 4% increase in accuracy when adding our mid-level gaze features. Finally, we discuss the implications of combining low- and mid-level gaze features, as well as the future directions for eye-based activity recognition

Combining Implicit Gaze and AI for Real-Time Intention Projection:Combining implicit gaze and AI for real-time intention projection

Intention recognition is the process of using behavioural cues to infer an agent’s goals or future behaviour. In face-to-face communication, our gaze implicitly signals our point of interest within the environment and therefore, inadvertently leaks our unspoken intentions to others. In our published body of work, we leverage this implicit function of gaze together with the tendency of humans to plan before executing their actions, resulting in an artificial agent that can project humans intentions while human players engage in a competitive game. In this demo, we created a path-planning game to demonstrate the capability of our artificial agent in a playful manner. The agent projects future plans of players by combining the use of implicit gaze of human players with an AI planning-based model. The demo aims to illustrate that gaze is intentional and that socially interactive agents can harness gaze as a natural input implicitly to assist humans collaboratively with knowledge of their intentions

Towards a Gaze-Informed Movement Intention Model for Robot-Assisted Upper-Limb Rehabilitation

Gaze-based intention detection has been explored for robotic-assisted neuro-rehabilitation in recent years. As eye movements often precede hand movements, robotic devices can use gaze information to augment the detection of movement intention in upper-limb rehabilitation. However, due to the likely practical drawbacks of using head-mounted eye trackers and the limited generalisability of the algorithms, gaze-informed approaches have not yet been used in clinical practice.This paper introduces a preliminary model for a gazeinformed movement intention that separates the intention spatial component obtained from the gaze from the time component obtained from movement. We leverage the latter to isolate the relevant gaze information happening just before the movement initiation. We evaluated our approach with six healthy individuals using an experimental setup that employed a screen-mounted eye-tracker. The results showed a prediction accuracy of 60% and 73% for an arbitrary target choice and an imposed target choice, respectively.From these findings, we expect that the model could 1) generalise better to individuals with movement impairment (by not considering movement direction), 2) allow a generalisation to more complex, multi-stage actions including several submovements, and 3) facilitate a more natural human-robot interactions and empower patients with the agency to decide movement onset. Overall, the paper demonstrates the potential for using gaze-movement model and the use of screen-based eye trackers for robot-assisted upper-limb rehabilitation

Comparing Gaze, Head and Controller Selection of Dynamically Revealed Targets in Head-mounted Displays

This paper presents a head-mounted virtual reality study that compared gaze, head, and controller pointing for selection of dynamically revealed targets. Existing studies on head-mounted 3D interaction have focused on pointing and selection tasks where all targets are visible to the user. Our study compared the effects of screen width (field of view), target amplitude and width, and prior knowledge of target location on modality performance. Results show that gaze and controller pointing are significantly faster than head pointing and that increased screen width only positively impacts performance up to a certain point. We further investigated the applicability of existing pointing models. Our analysis confirmed the suitability of previously proposed two-component models for all modalities while uncovering differences for gaze at known and unknown target positions. Our findings provide new empirical evidence for understanding input with gaze, head, and controller and are significant for applications that extend around the user