Systematic literature review of hand gestures used in human computer interaction interfaces

Gestures, widely accepted as a humans' natural mode of interaction with their surroundings, have been considered for use in human-computer based interfaces since the early 1980s. They have been explored and implemented, with a range of success and maturity levels, in a variety of fields, facilitated by a multitude of technologies. Underpinning gesture theory however focuses on gestures performed simultaneously with speech, and majority of gesture based interfaces are supported by other modes of interaction. This article reports the results of a systematic review undertaken to identify characteristics of touchless/in-air hand gestures used in interaction interfaces. 148 articles were reviewed reporting on gesture-based interaction interfaces, identified through searching engineering and science databases (Engineering Village, Pro Quest, Science Direct, Scopus and Web of Science). The goal of the review was to map the field of gesture-based interfaces, investigate the patterns in gesture use, and identify common combinations of gestures for different combinations of applications and technologies. From the review, the community seems disparate with little evidence of building upon prior work and a fundamental framework of gesture-based interaction is not evident. However, the findings can help inform future developments and provide valuable information about the benefits and drawbacks of different approaches. It was further found that the nature and appropriateness of gestures used was not a primary factor in gesture elicitation when designing gesture based systems, and that ease of technology implementation often took precedence

Intent Detection for Virtual Reality Architectural Design

In the context of optimization and reduction of cycles of product design in indus-try, digital collaborative tools have a major impact allowing an early-stage integra-tion of multidisciplinary challenges and oftentimes the search of global optimum rather than domain specific improvements. This paper presents a methodology for improving participants’ implication and performance during collaborative design sessions through virtual reality (VR) tools thanks to intention detection through body language interpretation and thus, reduction of cognitive workload. A proto-type of the methodology is being implemented based on an existing VR aided de-sign tool called DragonFly developed by Airbus. We will first discuss the choice of the different biological inputs to choose for our purpose, and how to merge these multimodal inputs in a meaningful way in order to ease further evolution and maintenance of our solution. Then, we will focus on the extraction of these inputs, their preprocessing, and the inference of intent and associated parameters. Finally, we will show the beginning of the application of this methodology to our specific use case of aircraft system installation

Introducción a las interfaces naturales gestuales con dispositivos de captura óptica

Las interfaces naturales gestuales son aquel tipo de interfaz de usuario en las que se interactúa con un sistema, aplicación, etcétera, sin utilizar métodos de entrada convencionales (como podrían ser un mouse, un teclado alfanumérico, un panel táctil, o joystick, etcétera) y en su lugar, se hace uso de movimientos gestuales del cuerpo o de alguna de sus partes, gestos faciales o sonidos. Pertenecen al grupo de las Interfaces Naturales, las cuales se basan en que la interacción con los dispositivos suceda de la misma manera que como lo hacemos con otros seres humanos, como por ejemplo a través del habla y los gestos. Este artículo hace una recopilación de dispositivos, métodos de captura y gestos comúnmente utilizados en el desarrollo de interfaces naturales gestuales, en función de familiarizar al lector con algunos conceptos básicos, enfocándonos en el funcionamiento, ventajas y desventajas de cada elemento.Facultad de Arte

Desarrollo de interfaces naturales gestuales con kinect

Este trabajo busca analizar tres programas de detección de manos, cada uno utilizando un distinto método de captación: cascadas de Haar, método de detección de objetos que recorre la imagen en sectores cada vez más pequeños y busca en cada uno si aparecen ciertas características visuales relacionadas con el objeto que se haya entrenado para detectar; blobs por diferencia, lo que implica identificar y delimitar las porciones más brillantes u oscuras de la imagen en regiones de interés llamadas blobs; y modelos de red neuronal, una serie de capas de nodos o neuronas interconectadas que procesan una información y devuelven otra. La definición de un modelo se realiza mediante el entrenamiento de la red, proceso en el que se le muestran ejemplos de lo que se quiere que el modelo realice (por ejemplo, imágenes en las que haya o no manos) para que este identifique la serie de relaciones correctas. Con este fin se desarrolló una interfaz gestual para la solicitud de turnos, como las que se pueden encontrar en hospitales, que puede ser controlada mediante cualquiera de los tres programas analizados. De cada uno se describe detalladamente el proceso de implementación teniendo en cuenta la facilidad de configuración en cada caso y se realizaron pruebas del desempeño en diversas condiciones espaciales. A partir de esto, se busca que este trabajo sirva como material pedagógico introductorio de las posibilidades y limitaciones en cuanto las diferentes formas de desarrollar una interfaz gestual, de tal forma que todo el código mencionado en este trabajo está disponible en un repositorio de GitHub.Facultad de Arte

ImagineD : a vision for cognitive driven creative design

CAD systems are well suited to later design phases, but do not effec-tively support the early ambiguous, iterative, and creative stages of de-sign. CAD is continually evolving, but only incrementally and by adapting to established design processes. We present a radically new vision for creative design – ImagineD – based on advances in HCI technology. In this vision, the designer is symbiotically connected to supporting computer systems via brain-computer and gesture recogni-tion interfaces, and the design process is directly driven by the design-er’s cognition (via neural signals) and natural behaviour (via intuitive gestures). Realising this vision requires advances in scientific models of cognition, neural activity, and gesture interaction in creative design. The paper presents the work and visions of the University of Strath-clyde, covering earlier CAD work before presenting ongoing empirical and theoretical research in the above areas by the ImagineD team. We conclude with key challenges

Effects of activity time limitation on gesture elicitation for form creation

Cognitive processing employed during design includes both time critical and time-consuming types of thinking. The ability to match the pace of design generation or modification with the designers thinking processes can be particularly important with gesture-based interfaces for form creation, especially where representation modes of input and response may influence the choice of activities performed. Particularly in gesture elicitation studies, time-consuming design activities can shift the focus on forming the analogies between problem at hand and prior knowledge and experiences, rather than intuitive gesture suggestions that would be the best fit for the given representation mode. However, design methodologies do not prescribe or discuss time limitations and their use in this context. In this paper, time limitation is explored during a gesture elicitation study for three-dimensional object creation, modification and manipulation, by comparing two study parts, one where time limitation was imposed and one where time was unlimited. Resulting gesture durations in both parts were comparable and elicited gestures were similar in nature and employing same elements of hand motion, supporting the hypothesis that time limitation can be a useful methodological approach when gestures are used for interaction with 3D objects and representation and interaction modalities are matched

Effects of activity time limitation on gesture elicitation for form creation

Cognitive processing employed during design includes both time critical and time-consuming types of thinking. The ability to match the pace of design generation or modification with the designers thinking processes can be particularly important with gesture-based interfaces for form creation, especially where representation modes of input and response may influence the choice of activities performed. Particularly in gesture elicitation studies, time-consuming design activities can shift the focus on forming the analogies between problem at hand and prior knowledge and experiences, rather than intuitive gesture suggestions that would be the best fit for the given representation mode. However, design methodologies do not prescribe or discuss time limitations and their use in this context. In this paper, time limitation is explored during a gesture elicitation study for three-dimensional object creation, modification and manipulation, by comparing two study parts, one where time limitation was imposed and one where time was unlimited. Resulting gesture durations in both parts were comparable and elicited gestures were similar in nature and employing same elements of hand motion, supporting the hypothesis that time limitation can be a useful methodological approach when gestures are used for interaction with 3D objects and representation and interaction modalities are matched

A Thematic and Reference Analysis of Touchless Technologies

The purpose of this research is to explore the utility and current state of touchless technologies. Five categories of technologies are identified as a result of collecting and reviewing literature: facial/biometric recognition, gesture recognition, touchless sensing, personal devices, and voice recognition. A thematic analysis was conducted to evaluate the advantages and disadvantages of the five categories. A reference analysis was also conducted to determine the similarities between articles in each category. Touchless sensing showed to have the most advantages and least similar references. Gesture recognition was the opposite. Comparing analyses shows more reliable technology types are more beneficial and diverse

A novel user-based gesture vocabulary for conceptual design

Research into hand gestures for human computer interaction has been prolific recently, but within it research on hand gestures for conceptual design has either focused on gestures that were defined by the researchers rather than the users, or those that were heavily influenced by what can be achieved using currently available technology. This paper reports on the study performed to identify a user elicited vocabulary of gestures for conceptual design, disassociated from the currently available technology, and its subsequent evaluation. The study included 44 product design engineering students (3rd, 4th year and recent graduates) and identified 1772 gestures that were analysed to build a novel gesture consensus set of vocabulary of hand gestures for conceptual design. This set is then evaluated by 10 other professionals, in order to generalise this set for a wider range of users and possibly reduce the need for training. The evaluation has shown that majority of gestures added to the vocabulary were easy to perform and appropriate for the activities, but that at the implementation stage the vocabulary will require another round of evaluation to account for the technology capabilities. The aim of this work is to create a starting point for a potential future system that could adapt to individual designers and allow them to use non-prescribed gestures that will support rather than inhibit their conceptual design thinking processes, akin to the developments that happened in hand writing recognition or predictive texting

User-based gesture vocabulary for form creation during a product design process

There are inconsistencies between the nature of the conceptual design and the functionalities of the computational systems supporting it, which disrupt the designers’ process, focusing on technology rather than designers’ needs. A need for elicitation of hand gestures appropriate for the requirements of the conceptual design, rather than those arbitrarily chosen or focusing on ease of implementation was identified.The aim of this thesis is to identify natural and intuitive hand gestures for conceptual design, performed by designers (3rd, 4th year product design engineering students and recent graduates) working on their own, without instruction and without limitations imposed by the facilitating technology. This was done via a user centred study including 44 participants. 1785 gestures were collected. Gestures were explored as a sole mean for shape creation and manipulation in virtual 3D space. Gestures were identified, described in writing, sketched, coded based on the taxonomy used, categorised based on hand form and the path travelled and variants identified. Then they were statistically analysed to ascertain agreement rates between the participants, significance of the agreement and the likelihood of number of repetitions for each category occurring by chance. The most frequently used and statistically significant gestures formed the consensus set of vocabulary for conceptual design. The effect of the shape of the manipulated object on the gesture performed, and if the sequence of the gestures participants proposed was different from the established CAD solid modelling practices were also observed.Vocabulary was evaluated by non-designer participants, and the outcomes have shown that the majority of gestures were appropriate and easy to perform. Evaluation was performed theoretically and in the VR environment. Participants selected their preferred gestures for each activity, and a variant of the vocabulary for conceptual design was created as an outcome, that aims to ensure that extensive training is not required, extending the ability to design beyond trained designers only.There are inconsistencies between the nature of the conceptual design and the functionalities of the computational systems supporting it, which disrupt the designers’ process, focusing on technology rather than designers’ needs. A need for elicitation of hand gestures appropriate for the requirements of the conceptual design, rather than those arbitrarily chosen or focusing on ease of implementation was identified.The aim of this thesis is to identify natural and intuitive hand gestures for conceptual design, performed by designers (3rd, 4th year product design engineering students and recent graduates) working on their own, without instruction and without limitations imposed by the facilitating technology. This was done via a user centred study including 44 participants. 1785 gestures were collected. Gestures were explored as a sole mean for shape creation and manipulation in virtual 3D space. Gestures were identified, described in writing, sketched, coded based on the taxonomy used, categorised based on hand form and the path travelled and variants identified. Then they were statistically analysed to ascertain agreement rates between the participants, significance of the agreement and the likelihood of number of repetitions for each category occurring by chance. The most frequently used and statistically significant gestures formed the consensus set of vocabulary for conceptual design. The effect of the shape of the manipulated object on the gesture performed, and if the sequence of the gestures participants proposed was different from the established CAD solid modelling practices were also observed.Vocabulary was evaluated by non-designer participants, and the outcomes have shown that the majority of gestures were appropriate and easy to perform. Evaluation was performed theoretically and in the VR environment. Participants selected their preferred gestures for each activity, and a variant of the vocabulary for conceptual design was created as an outcome, that aims to ensure that extensive training is not required, extending the ability to design beyond trained designers only