Understanding 3D mid-air hand gestures with interactive surfaces and displays: a systematic literature review

3D gesture based systems are becoming ubiquitous and there are many mid-air hand gestures that exist for interacting with digital surfaces and displays. There is no well defined gesture set for 3D mid-air hand gestures which makes it difficult to develop applications that have consistent gestures. To understand what gestures exist we conducted the first comprehensive systematic literature review on mid-air hand gestures following existing research methods. The results of the review identified 65 papers where the mid-air hand gestures supported tasks for selection, navigation, and manipulation. We also classified the gestures according to a gesture classification scheme and identified how these gestures have been empirically evaluated. The results of the review provide a richer understanding of what mid-air hand gestures have been designed, implemented, and evaluated in the literature which can help developers design better user experiences for digital interactive surfaces and displays

Understanding 3D mid-air hand gestures with interactive surfaces and displays: a systematic literature review

3D gesture based systems are becoming ubiquitous and there are many mid-air hand gestures that exist for interacting with digital surfaces and displays. There is no well defined gesture set for 3D mid-air hand gestures which makes it difficult to develop applications that have consistent gestures. To understand what gestures exist we conducted the first comprehensive systematic literature review on mid-air hand gestures following existing research methods. The results of the review identified 65 papers where the mid-air hand gestures supported tasks for selection, navigation, and manipulation. We also classified the gestures according to a gesture classification scheme and identified how these gestures have been empirically evaluated. The results of the review provide a richer understanding of what mid-air hand gestures have been designed, implemented, and evaluated in the literature which can help developers design better user experiences for digital interactive surfaces and displays

Touching the invisible: Localizing ultrasonic haptic cues

While mid-air gestures offer new possibilities to interact with or around devices, some situations, such as interacting with applications, playing games or navigating, may require visual attention to be focused on a main task. Ultrasonic haptic feedback can provide 3D spatial haptic cues that do not demand visual attention for these contexts. In this paper, we present an initial study of active exploration of ultrasonic haptic virtual points that investigates the spatial localization with and without the use of the visual modality. Our results show that, when providing haptic feedback giving the location of a widget, users perform 50% more accurately compared to providing visual feedback alone. When provided with a haptic location of a widget alone, users are more than 30% more accurate than when given a visual location. When aware of the location of the haptic feedback, active exploration decreased the minimum recommended widget size from 2cm2 to 1cm2 when compared to passive exploration from previous studies. Our results will allow designers to create better mid-air interactions using this new form of haptic feedback

Evaluating the effects of feedback type on older adults’ performance in mid-air pointing and target selection

“Hands-free” pointing techniques used in mid-air gesture interaction require precise motor control and dexterity. Although being applied in a growing number of interaction contexts over the past few years, this input method can be challenging for older users (60+ years old) who experience natural decline in pointing abilities due to natural ageing process. We report the findings of a target acquisition experiment in which older adults had to perform “pointand- select” gestures in mid-air. The experiment investigated the effect of 6 feedback conditions on pointing and selection performance of older users. Our findings suggest that the bimodal combination of Visual and Audio feedback lead to faster target selection times for older adults, but did not lead to making less errors. Furthermore, target location on screen was found to play a more important role in both selection time and accuracy of pointand- select tasks than feedback type

Brave New GES World:A Systematic Literature Review of Gestures and Referents in Gesture Elicitation Studies

How to determine highly effective and intuitive gesture sets for interactive systems tailored to end users’ preferences? A substantial body of knowledge is available on this topic, among which gesture elicitation studies stand out distinctively. In these studies, end users are invited to propose gestures for specific referents, which are the functions to control for an interactive system. The vast majority of gesture elicitation studies conclude with a consensus gesture set identified following a process of consensus or agreement analysis. However, the information about specific gesture sets determined for specific applications is scattered across a wide landscape of disconnected scientific publications, which poses challenges to researchers and practitioners to effectively harness this body of knowledge. To address this challenge, we conducted a systematic literature review and examined a corpus of N=267 studies encompassing a total of 187, 265 gestures elicited from 6, 659 participants for 4, 106 referents. To understand similarities in users’ gesture preferences within this extensive dataset, we analyzed a sample of 2, 304 gestures extracted from the studies identified in our literature review. Our approach consisted of (i) identifying the context of use represented by end users, devices, platforms, and gesture sensing technology, (ii) categorizing the referents, (iii) classifying the gestures elicited for those referents, and (iv) cataloging the gestures based on their representation and implementation modalities. Drawing from the findings of this review, we propose guidelines for conducting future end-user gesture elicitation studies

Experience Prototyping for Automotive Applications

In recent years, we started to define our life through experiences we make instead of objectswe buy. To attend a concert of our favorite musician may be more important for us thanowning an expensive stereo system. Similarly, we define interactive systems not only by thequality of the display or its usability, but rather by the experiences we can make when usingthe device. A cell phone is primarily built for making calls and receiving text messages,but on an emotional level it might provide a way to be close to our loved ones, even thoughthey are far away sometimes. When designing interactive technology, we do not only haveto answer the question how people use our systems, but also why they use them. Thus,we need to concentrate on experiences, feelings and emotions arising during interaction.Experience Design is an approach focusing on the story that a product communicates beforeimplementing the system. In an interdisciplinary team of psychologists, industrial designers, product developers andspecialists in human-computer interaction, we applied an Experience Design process to theautomotive domain. A major challenge for car manufacturers is the preservation of theseexperiences throughout the development process. When implementing interactive systemsengineers rely on technical requirements and a set of constraints (e.g., safety) oftentimescontradicting aspects of the designed experience. To resolve this conflict, Experience Prototypingis an important tool translating experience stories to an actual interactive product. With this thesis I investigate the Experience Design process focusing on Experience Prototyping.Within the automotive context, I report on three case studies implementing threekinds of interactive systems, forming and following our approach. I implemented (1) anelectric vehicle information system called Heartbeat, communicating the state of the electricdrive and the batteries to the driver in an unobtrusive and ensuring way. I integrated Heartbeatinto the dashboard of a car mock-up with respect to safety and space requirements butat the same time holding on to the story in order to achieve a consistent experience. With (2)the Periscope I implemented a mobile navigation device enhancing the social and relatednessexperiences of the passengers in the car. I built and evaluated several experience prototypesin different stages of the design process and showed that they transported the designed experiencethroughout the implementation of the system. Focusing on (3) the experience offreehand gestures, GestShare explored this interaction style for in-car and car-to-car socialexperiences. We designed and implemented a gestural prototypes for small but effectivesocial interactions between drivers and evaluated the system in the lab and and in-situ study. The contributions of this thesis are (1) a definition of Experience Prototyping in the automotivedomain resulting from a literature review and my own work, showing the importanceand feasibility of Experience Prototyping for Experience Design. I (2) contribute three casestudies and describe the details of several prototypes as milestones on the way from a anexperience story to an interactive system. I (3) derive best practices for Experience Prototypingconcerning their characteristics such as fidelity, resolution and interactivity as well asthe evaluation in the lab an in situ in different stages of the process.Wir definieren unser Leben zunehmend durch Dinge, die wir erleben und weniger durchProdukte, die wir kaufen. Ein Konzert unseres Lieblingsmusikers zu besuchen kann dabeiwichtiger sein, als eine teure Stereoanlage zu besitzen. Auch interaktive Systeme bewertenwir nicht mehr nur nach der Qualität des Displays oder der Benutzerfreundlichkeit, sondernauch nach Erlebnissen, die durch die Benutzung möglich werden. Das Smartphone wurdehauptsächlich zum Telefonieren und Schreiben von Nachrichten entwickelt. Auf einer emotionalenEbene bietet es uns aber auch eine Möglichkeit, wichtigen Personen sehr nah zusein, auch wenn sie manchmal weit weg sind. Bei der Entwicklung interaktiver Systememüssen wir uns daher nicht nur fragen wie, sondern auch warum diese benutzt werden. Erlebnisse,Gefühle und Emotionen, die während der Interaktion entstehen, spielen dabei einewichtige Rolle. Experience Design ist eine Disziplin, die sich auf Geschichten konzentriert,die ein Produkt erzählt, bevor es tatsächlich implementiert wird. In einem interdisziplinären Team aus Psychologen, Industrie-Designern, Produktentwicklernund Spezialisten der Mensch-Maschine-Interaktion wurde ein Prozess zur Erlebnis-Gestaltung im automobilen Kontext angewandt. Die Beibehaltung von Erlebnissen über dengesamten Entwicklungsprozess hinweg ist eine große Herausforderung für Automobilhersteller.Ingenieure hängen bei der Implementierung interaktiver Systeme von technischen,sicherheitsrelevanten und ergonomischen Anforderungen ab, die oftmals dem gestaltetenErlebnis widersprechen. Die Bereitstellung von Erlebnis-Prototypen ermöglicht die Übersetzungvon Geschichten in interaktive Produkte und wirkt daher diesem Konflikt entgegen. Im Rahmen dieser Dissertation untersuche ich den Prozess zur Erlebnis-Gestaltung hinsichtlichder Bedeutung von Erlebnis-Prototypen. Ich berichte von drei Fallbeispielen im automobilenBereich, die die Gestaltung und Implementierung verschiedener interaktiver Systemenumfassen. (1) Ein Informationssystem für Elektrofahrzeuge, der Heartbeat, macht den Zustanddes elektrischen Antriebs und den Ladestand der Batterien für den Fahrer visuell undhaptisch erlebbar. Nach der Implementierung mehrerer Prototypen wurde Heartbeat unterBerücksichtigung verschiedener technischer und sicherheitsrelevanter Anforderungen in dieArmaturen eines Fahrzeugmodells integriert, ohne dass dabei das gestaltete Erlebnis verlorengegangen ist. (2) Das Periscope ist ein mobiles Navigationsgerät, das den Insassensoziale Erlebnisse ermöglicht und das Verbundenheitsgefühl stärkt. Durch die Implementierungmehrere Erlebnis-Prototypen und deren Evaluation in verschiedenen Phasen des Entwicklungsprozesseskonnten die gestalteten Erlebnisse konsistent erhalten werden. (3) ImProjekt GestShare wurde das Potential der Interaktion durch Freiraumgesten im Fahrzeuguntersucht. Dabei standen ein Verbundenheitserlebnis des Fahrers und soziale Interaktionenmit Fahrern anderer Fahrzeuge im Fokus. Es wurden mehrere Prototypen implementiert undauch in einer Verkehrssituation evaluiert. Die wichtigsten Beiträge dieser Dissertation sind (1) eine intensive Betrachtung und Anwendungvon Erlebnis-Prototypen im Auto und deren Relevanz bei der Erlebnis-Gestaltung,beruhend auf einer Literaturauswertung und der eigenen Erfahrung innerhalb des Projekts; (2) drei Fallstudien und eine detaillierte Beschreibung mehrere Prototypen in verschiedenenPhasen des Prozesses und (3) Empfehlungen zu Vorgehensweisen bei der Erstellung vonErlebnis-Prototypen hinsichtlich der Eigenschaften wie Nähe zum finalen Produkt, Anzahlder implementierten Details und Interaktivität sowie zur Evaluation im Labor und in tatsächlichenVerkehrssituationen in verschiedenen Phasen des Entwicklungsprozesses

Barehand Mode Switching in Touch and Mid-Air Interfaces

Raskin defines a mode as a distinct setting within an interface where the same user input will produce results different to those it would produce in other settings. Most interfaces have multiple modes in which input is mapped to different actions, and, mode-switching is simply the transition from one mode to another. In touch interfaces, the current mode can change how a single touch is interpreted: for example, it could draw a line, pan the canvas, select a shape, or enter a command. In Virtual Reality (VR), a hand gesture-based 3D modelling application may have different modes for object creation, selection, and transformation. Depending on the mode, the movement of the hand is interpreted differently. However, one of the crucial factors determining the effectiveness of an interface is user productivity. Mode-switching time of different input techniques, either in a touch interface or in a mid-air interface, affects user productivity. Moreover, when touch and mid-air interfaces like VR are combined, making informed decisions pertaining to the mode assignment gets even more complicated. This thesis provides an empirical investigation to characterize the mode switching phenomenon in barehand touch-based and mid-air interfaces. It explores the potential of using these input spaces together for a productivity application in VR. And, it concludes with a step towards defining and evaluating the multi-faceted mode concept, its characteristics and its utility, when designing user interfaces more generally

Exploring the use of hand-to-face input for interacting with head-worn displays

International audienceWe propose the use of Hand-to-Face input, a method to interact with head-worn displays (HWDs) that involves contact with the face. We explore Hand-to-Face interaction to find suitable techniques for common mobile tasks. We evaluate this form of interaction with document navigation tasks and examine its social acceptability. In a first study, users identify the cheek and forehead as predominant areas for interaction and agree on gestures for tasks involving continuous input, such as document navigation. These results guide the design of several Hand-to-Face navigation techniques and reveal that gestures performed on the cheek are more efficient and less tiring than interactions directly on the HWD. Initial results on the social acceptability of Hand-to-Face input allow us to further refine our design choices, and reveal unforeseen results: some gestures are considered culturally inappropriate and gender plays a role in selection of specific Hand-to-Face interactions. From our overall results, we provide a set of guidelines for developing effective Hand-to-Face interaction techniques