Understanding Visual Feedback in Large-Display Touchless Interactions: An Exploratory Study

Touchless interactions synthesize input and output from physically disconnected motor and display spaces without any haptic feedback. In the absence of any haptic feedback, touchless interactions primarily rely on visual cues, but properties of visual feedback remain unexplored. This paper systematically investigates how large-display touchless interactions are affected by (1) types of visual feedback—discrete, partial, and continuous; (2) alternative forms of touchless cursors; (3) approaches to visualize target-selection; and (4) persistent visual cues to support out-of-range and drag-and-drop gestures. Results suggest that continuous was more effective than partial visual feedback; users disliked opaque cursors, and efficiency did not increase when cursors were larger than display artifacts’ size. Semantic visual feedback located at the display border improved users’ efficiency to return within the display range; however, the path of movement echoed in drag-and-drop operations decreased efficiency. Our findings contribute key ingredients to design suitable visual feedback for large-display touchless environments.This work was partially supported by an IUPUI Research Support Funds Grant (RSFG)

Barehand Mode Switching in Touch and Mid-Air Interfaces

Raskin defines a mode as a distinct setting within an interface where the same user input will produce results different to those it would produce in other settings. Most interfaces have multiple modes in which input is mapped to different actions, and, mode-switching is simply the transition from one mode to another. In touch interfaces, the current mode can change how a single touch is interpreted: for example, it could draw a line, pan the canvas, select a shape, or enter a command. In Virtual Reality (VR), a hand gesture-based 3D modelling application may have different modes for object creation, selection, and transformation. Depending on the mode, the movement of the hand is interpreted differently. However, one of the crucial factors determining the effectiveness of an interface is user productivity. Mode-switching time of different input techniques, either in a touch interface or in a mid-air interface, affects user productivity. Moreover, when touch and mid-air interfaces like VR are combined, making informed decisions pertaining to the mode assignment gets even more complicated. This thesis provides an empirical investigation to characterize the mode switching phenomenon in barehand touch-based and mid-air interfaces. It explores the potential of using these input spaces together for a productivity application in VR. And, it concludes with a step towards defining and evaluating the multi-faceted mode concept, its characteristics and its utility, when designing user interfaces more generally

Measuring Tool Embodiment in Ready-to-Hand and Unready-to-Hand Situations Using Virtual and Physical Tools

Virtual environments can provide access to a variety of information that can be designed to mimic physical attributes or afford physical-like actions. Virtual reality and other ways of interactions such as multi-touch, tangible interaction, and mid-air gestures, often promise to be more natural, where the technology becomes invisible. However, there is limited investigation on how to measure the level of invisibility when interacting with technology with a quantitative measure. For example, virtual reality provides physical-like actions that mimics every aspect of the physical world interaction, but there is no direct methodology to measure these complex interactions. As a result, designers of novel interactive technologies do not have a clear understanding of how to measure these phenomena. The current research in human computer interaction focuses on using performance measures or self-reports questionnaires to evaluate interactive technologies. Research in psychology and philosophy, on the other hand, provides an understanding of the human condition in the physical environment. Consequently, the aim of this dissertation is to provide an effective methodology to measure the invisibility aspect of technology that applies both experimental psychology and HCI research. Study 1 presented in this dissertation used the after-effect phenomenon as a measure of object embodiment — when interacting with physical objects can affect haptic changes in perception. Study 2 investigated tool embodiment to measure the interaction with physical and virtual tools, where change in attention was used as a measure of tool embodiment. Finally, study 3 further examined tool embodiment with different tool states (broken or working tool) and different inputs alternatives. Over the past decade, multi-touch surfaces have become commonplace, with many researchers and practitioners describing the benefits of their natural, physical-like interactions. Study 1 presents an empirical investigation of the psychophysical effects of direct interaction with both physical and virtual objects. The phenomenon of Kinesthetic Figural After Effects — a change in understanding of the physical size of an object after a period of exposure to an object of different size, was used as a measure. While this effect is robustly reproducible when using physical artefacts, this same effect does not manifest when manipulating virtual objects on a direct, multi-touch tabletop display. Study 2 leveraged the phenomenon of tool embodiment as measure of interaction. Tool embodiment is when a tool becomes an extension of one’s body, where attention shifts to the task at hand, rather than the tool itself. This study tested tool embodiment framework to measure the aspect of being part of a tool by incorporating philosophical and psychological concepts. This framework was applied to design and conduct study 2 that uses attention to measure readiness-to-hand with both a physical tool and a virtual tool. A novel task where participants use a tool to rotate an object, while simultaneously responding to visual stimuli both near their hand and near the task was introduced in this study. The results demonstrated that participants paid more attention to the task than to both virtual and physical tools. Study 3 further investigated tool embodiment to measure ready-to-hand and unready-to-hand situations. Locus of attention index (LAI) was used to measure the level of tool embodiment in virtual environments. Three different input modalities were used to control the virtual tool to accomplish the task. The results of this study showed that the LAI is higher with the working tool indicating an increased level of tool embodiment, and lower with broken tool indicating a decreased level of tool embodiment. Overall, the research presented in this dissertation investigated embodied interactions with both physical and virtual environments. The contributions included the construction of an evolution measure of object interaction (using the measure of after effect with physical and virtual tools) and tool interaction (using the measure of attention and LIA with physical and virtual tools). The empirical results of study 1 revealed that the after-effect measure might not be a practical measure to evaluated embodied interactions in virtual environments. However, study 2 and 3 provided a reliable method to measures embodied interactions when using tools to interact with the virtual environments. This dissertation also provided tool embodiment framework that can be used as a guide for designers to evaluate the invisibility aspect of technology

Understanding interaction mechanics in touchless target selection

Indiana University-Purdue University Indianapolis (IUPUI)We use gestures frequently in daily life—to interact with people, pets, or objects. But interacting with computers using mid-air gestures continues to challenge the design of touchless systems. Traditional approaches to touchless interaction focus on exploring gesture inputs and evaluating user interfaces. I shift the focus from gesture elicitation and interface evaluation to touchless interaction mechanics. I argue for a novel approach to generate design guidelines for touchless systems: to use fundamental interaction principles, instead of a reactive adaptation to the sensing technology. In five sets of experiments, I explore visual and pseudo-haptic feedback, motor intuitiveness, handedness, and perceptual Gestalt effects. Particularly, I study the interaction mechanics in touchless target selection. To that end, I introduce two novel interaction techniques: touchless circular menus that allow command selection using directional strokes and interface topographies that use pseudo-haptic feedback to guide steering–targeting tasks. Results illuminate different facets of touchless interaction mechanics. For example, motor-intuitive touchless interactions explain how our sensorimotor abilities inform touchless interface affordances: we often make a holistic oblique gesture instead of several orthogonal hand gestures while reaching toward a distant display. Following the Gestalt theory of visual perception, we found similarity between user interface (UI) components decreased user accuracy while good continuity made users faster. Other findings include hemispheric asymmetry affecting transfer of training between dominant and nondominant hands and pseudo-haptic feedback improving touchless accuracy. The results of this dissertation contribute design guidelines for future touchless systems. Practical applications of this work include the use of touchless interaction techniques in various domains, such as entertainment, consumer appliances, surgery, patient-centric health settings, smart cities, interactive visualization, and collaboration

A experiência mediada por interfaces gestuais touchless em contexto turístico

Doutoramento em Informação e Comunicação em Plataformas DigitaisA evolução das Tecnologias da Informação e Comunicação impeliu novos modelos e estímulos para o sector do turismo. Estas mudanças, combinadas com uma nova postura do turista, repercutindo as dinâmicas da Web 2.0 e manifestando os contornos de uma cultura de participação, abriram espaço para o surgimento de novos serviços turísticos, de possível acesso ubíquo e personalizado ao longo de todo o ciclo da experiência turística. Simultaneamente, o surgimento de novos paradigmas de Interação Humano- Computador, de que são exemplo as interfaces gestuais touchless, acarretam oportunidades e desafios, quer ao nível da usabilidade e User Experience (UX), quer de um ponto de vista específico, quando concebida a sua potencial integração na experiência turística, como mais um veículo de consumo, partilha e manipulação de informação turística. A presente investigação, temporalmente, acompanhou o lançamento e sucesso do sensor Kinect, que aproximou e diversificou a aplicação e desenvolvimento de interfaces touchless em diferentes contextos. No âmbito turístico, foi identificado que a possível aplicação deste paradigma ainda não tinha sido explorado de forma detalhada. Verificava-se também a necessidade de contribuir para a definição de standards e estratégias para a exploração da UX em relação às interfaces gestuais touchless. Decorrendo da conjuntura apresentada, o presente estudo pretendeu focar a possível aplicação, potencialidades e experiência de utilização de soluções interativas com suporte de interação gestual touchless em contexto turístico. O estudo empírico desenhado e implementado envolveu dois momentos principais: a execução de entrevistas a experts e a realização de uma avaliação em contexto controlado de um protótipo de uma solução interativa touchless, destinada ao contexto turístico. A avaliação referida, na qual participaram 51 indivíduos, implicou o desenvolvimento de instrumentos e de um protocolo de teste adequado aos objetivos e características diferenciadoras do estudo. Como resultados gerais, o primeiro momento permitiu identificar um conjunto de vantagens e desvantagens, potencialidades e especificidades das interfaces gestuais touchless, quando concebida a sua aplicação ao turismo. O segundo momento, contando com o envolvimento dos participantes, destacou as questões relacionadas com a usabilidade e UX das interfaces touchless, permitindo estabelecer um conjunto de guias, metodologia e estratégias, que podem ser aplicadas no desenvolvimento e avaliação de outras soluções que suportem o paradigma referido. Recolheram-se ainda opiniões ao nível do potencial uso das mesmas em contexto turístico, identificadas no contributo dos utilizadores/participantes da avaliação em contexto controlado.The evolution of communication and information technologies drove new approaches in the tourism industry. This stimulus, combined with the new tourist behaviour, aware of Web 2.0 dynamics and participative in the social web culture, have provided new opportunities for new tourism services, with ubiquitous and personalized access during the entire cycle of the touristic experience. Also, the emergence of new human-computer interaction paradigms - such as touchless gestural interfaces - lead to challenges and opportunities in what concerns usability and user experience (UX). Furthermore, when integrated in the touristic experience, those interfaces may enhance information sharing and manipulation, adding a new dimension to how we experience tourism. This research aroused with the launch of the Kinect sensor, which allowed the application and development of touchless interfaces in different contexts. In tourism, the application of this paradigm has not yet been fully discussed. It was also relevant to contribute to the definition of standards and strategies for researching and evaluating the UX with touchless interfaces. Thus, this study intended to focus on the possible application, potentialities and UX resulting from using interactive solutions with touchless gestural interaction in tourism. The empirical study had two main stages: first, the performance of interviews with experts and second, the execution of an evaluation in a controlled setting, using a prototype of an interactive gestural touchless interface. This evaluation, which was attended by 51 participants, implied the development of suitable tools and evaluation protocol. As a result, the first stage enabled the identification of a set of advantages, disadvantages, possibilities and features of this type of interactive solutions. The second stage focused on the issues related to usability and user experience of touchless gestural interfaces, to establish a set of guidelines, methodologies and approaches. It also collected opinions from users about the application of touchless gestural interfaces in tourism