Understanding interaction mechanics in touchless target selection

Indiana University-Purdue University Indianapolis (IUPUI)We use gestures frequently in daily life—to interact with people, pets, or objects. But interacting with computers using mid-air gestures continues to challenge the design of touchless systems. Traditional approaches to touchless interaction focus on exploring gesture inputs and evaluating user interfaces. I shift the focus from gesture elicitation and interface evaluation to touchless interaction mechanics. I argue for a novel approach to generate design guidelines for touchless systems: to use fundamental interaction principles, instead of a reactive adaptation to the sensing technology. In five sets of experiments, I explore visual and pseudo-haptic feedback, motor intuitiveness, handedness, and perceptual Gestalt effects. Particularly, I study the interaction mechanics in touchless target selection. To that end, I introduce two novel interaction techniques: touchless circular menus that allow command selection using directional strokes and interface topographies that use pseudo-haptic feedback to guide steering–targeting tasks. Results illuminate different facets of touchless interaction mechanics. For example, motor-intuitive touchless interactions explain how our sensorimotor abilities inform touchless interface affordances: we often make a holistic oblique gesture instead of several orthogonal hand gestures while reaching toward a distant display. Following the Gestalt theory of visual perception, we found similarity between user interface (UI) components decreased user accuracy while good continuity made users faster. Other findings include hemispheric asymmetry affecting transfer of training between dominant and nondominant hands and pseudo-haptic feedback improving touchless accuracy. The results of this dissertation contribute design guidelines for future touchless systems. Practical applications of this work include the use of touchless interaction techniques in various domains, such as entertainment, consumer appliances, surgery, patient-centric health settings, smart cities, interactive visualization, and collaboration

Interactive advertising displays

Interactive public displays are the latest development in the field of out-of-home advertising. Throughout history characteristic shapes for billboards evolved such as flat rectangular displays, long displays or cylindrical advertising columns. This work presents novel interactive display designs that are based on these historical role models and allow passers-by to interact with them in a natural, touchless manner. It further pursues a vision where interactive public displays become more active themselves and actively influence passer-by behavior in order to increase their effectiveness, better attract attention and improve public interaction in front of them. First, to overcome the challenge that passers-by often do not expect public displays to be interactive and thus pay no attention to them, this work presents a solution called unaware initial interaction that surprises passers-by and communicates interactivity by giving visual feedback to their initial movements. To be effective, the visual feedback has to be designed considering the specific display shapes, their requirements to contents and the typical approaching trajectories. Second, to overcome the challenge that larger groups of passers-by often crowd together in front of wide public displays or do not take optimal positions for interaction, this work presents a solution to subtly and actively guide users by dynamic and interactive visual cues on the screen in order to better distribute them. To explore these concepts and following an initial analysis of the out-of-home domain and of typical display qualities, interactive counterparts to the classical display shapes are designed such as interactive advertising columns, long banner displays and life-size screens. Then interactive contents and visual feedbacks are designed which implement the presented interactivity concepts, and audience behavior around them is analyzed in several long-term field studies in public space. Finally the observed passer-by and user behavior and the effectiveness of the display and content designs are discussed and takeaways given that are useful for practitioners and researchers in the field of public interaction with out-of-home displays.Interaktive öffentliche Displays sind die neueste Entwicklung im Bereich der Außenwerbung. Im Laufe der Geschichte bildeten sich charakteristische Formen für Werbetafeln heraus wie flache rechteckige Displays, lange Displays oder zylindrische Werbesäulen. Die vorliegende Arbeit stellt neuartige Designs für Displays vor, die auf diesen historischen Vorbildern aufbauen und den Passanten erlauben, mit ihnen auf eine natürliche, berührungslose Art und Weise zu interagieren. Darüber hinaus verfolgt sie eine Vision, in der interaktive öffentliche Displays aktiver werden und entsprechend das Passantenverhalten beeinflussen, um ihre Wirksamkeit zu erhöhen, mehr Aufmerksamkeit auf sich zu ziehen und die öffentliche Interaktion mit ihnen zu verbessern. Zunächst stellt diese Arbeit eine als Unbewusste Initialinteraktion bezeichnete Lösung vor, welche die Passanten überrascht und mittels visuellem Feedback auf ihre anfänglichen Bewegungen Interaktivität übermittelt, um die Herausforderung zu bewältigen, dass Passanten oft nicht erwarten, dass öffentliche Displays interaktiv sind und sie ihnen somit keine Aufmerksamkeit schenken. Um effektiv zu sein, muss das visuelle Feedback dabei so gestaltet werden, dass es die spezifischen Displayformen, ihre Anforderungen an die dargestellten Inhalte und ihre typischen Annäherungswege berücksichtigt. Zweitens stellt sie eine Lösung vor, bei der die Nutzer auf subtile Weise und durch auf dem Bildschirm dargestellte dynamische und interaktive visuelle Reize aktiv geführt werden, um sie besser vor dem Display zu verteilen, um die Herausforderung zu bewältigen, dass größere Gruppen von Passanten sich oft vor breiten öffentlichen Displays zusammendrängen oder keine optimalen Positionen für die Interaktion einnehmen. Zur Erforschung dieser Konzepte werden im Anschluss an eine einführende Analyse von Außenwerbedisplays und ihrer typischen Eigenschaften interaktive Entsprechungen der klassischen Displayformen entwickelt wie interaktive Litfaßsäulen, lange Bannerdisplays und Life-size Screens. Weiter werden für diese Displays interaktive Inhalte und visuelle Feedbacks entwickelt, welche die vorgestellten Interaktivitätskonzepte umsetzen und das Verhalten des anwesenden Publikums in mehreren Langzeit-Feldstudien im öffentlichen Raum untersucht. Schließlich werden das beobachtete Passanten- und Nutzerverhalten und die Effektivität der entwickelten Display-Designs und Inhalte bewertet und nützliche Empfehlungen für Praktiker und Forscher auf dem Gebiet der öffentlichen Interaktion mit Außenwerbedisplays gegeben

On the Simultaneous Perception of Sound and Three-Dimensional Objects

Although examples of work investigating the perceptual relationship and possibilities of sound and image are common, relatively litle work has been carried out into multimedia works combining sound and three-dimensional objects. A practice-based investigation into this subject is presented with original artworks and contectual material from sound art, sculpture, moving image and psychology. The project sets out to more examine the perception of multimedia work, specifically through the creation and analysis of artworks combining sound and physical objects. It considers three main areas of study: sound’s ability to draw attention to, or modify, the existing properties of an object; techniques which encourage sound and object to appear cohesively as part of the same work; and a discussion of cognitive effects that may occur as a result of their simulataneous perception. Using the concept of the search space from evolutionary computing as an example, the case is made that multimedia artworksde can present a larger field of creative opportunity than single-media works, due to the enhanced interplay between the two media and the viewer's a priori knowledge. The roles of balance, dynamism and interactivity in multimedia work are also explored. Throughout the thesis examples of original artworks are given which exemplify the issues raised. The main outcome of the study is a proposed framework for categorising and analysing the perception of multimedia artworks, based on increasing semantic separation between the sensory elements. It is claimed that as the relationship between these elements becomes less obvious, more work is demanded of the viewer's imagination in trying to reconcile the gap, leading to active engagement and the possibility of extra imaginary forms which do not exist in the original material. It is proposed that the framework and ideas in this document will be applicable beyond the sound/object focus of this study, and it is hoped they will inform research into multimedia work in other forms

The Perception/Action loop: A Study on the Bandwidth of Human Perception and on Natural Human Computer Interaction for Immersive Virtual Reality Applications

Virtual Reality (VR) is an innovating technology which, in the last decade, has had a widespread success, mainly thanks to the release of low cost devices, which have contributed to the diversification of its domains of application. In particular, the current work mainly focuses on the general mechanisms underling perception/action loop in VR, in order to improve the design and implementation of applications for training and simulation in immersive VR, especially in the context of Industry 4.0 and the medical field. On the one hand, we want to understand how humans gather and process all the information presented in a virtual environment, through the evaluation of the visual system bandwidth. On the other hand, since interface has to be a sort of transparent layer allowing trainees to accomplish a task without directing any cognitive effort on the interaction itself, we compare two state of the art solutions for selection and manipulation tasks, a touchful one, the HTC Vive controllers, and a touchless vision-based one, the Leap Motion. To this aim we have developed ad hoc frameworks and methodologies. The software frameworks consist in the creation of VR scenarios, where the experimenter can choose the modality of interaction and the headset to be used and set experimental parameters, guaranteeing experiments repeatability and controlled conditions. The methodology includes the evaluation of performance, user experience and preferences, considering both quantitative and qualitative metrics derived from the collection and the analysis of heterogeneous data, as physiological and inertial sensors measurements, timing and self-assessment questionnaires. In general, VR has been found to be a powerful tool able to simulate specific situations in a realistic and involving way, eliciting user\u2019s sense of presence, without causing severe cybersickness, at least when interaction is limited to the peripersonal and near-action space. Moreover, when designing a VR application, it is possible to manipulate its features in order to trigger or avoid triggering specific emotions and voluntarily create potentially stressful or relaxing situations. Considering the ability of trainees to perceive and process information presented in an immersive virtual environment, results show that, when people are given enough time to build a gist of the scene, they are able to recognize a change with 0.75 accuracy when up to 8 elements are in the scene. For interaction, instead, when selection and manipulation tasks do not require fine movements, controllers and Leap Motion ensure comparable performance; whereas, when tasks are complex, the first solution turns out to be more stable and efficient, also because visual and audio feedback, provided as a substitute of the haptic one, does not substantially contribute to improve performance in the touchless case

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 13th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2022, held in Hamburg, Germany, in May 2022. The 36 regular papers included in this book were carefully reviewed and selected from 129 submissions. They were organized in topical sections as follows: haptic science; haptic technology; and haptic applications

The Benefits of Haptic Feedback in Mobile Phone Camera

Communication is basically the act of transferring information from one place to another. Feedback is a system where the reaction or response of the receiver arrives at the sender after he/she has interpreted the message. Feedback is inevitably essential to make two way communications effective. In fact, without feedback communication remains incomplete. At times, feedback could be verbal such as written and oral. Then in some cases, it could be nonverbal. Feedback is mainly a response from your audiences; it allows you to evaluate the effectiveness of your message. In fact research shows that the majority of the messages that have been sent are nonverbal and the ability to understand and use nonverbal communication is powerful tools that will help people connect with each other. As well as communication where nonverbal shows much more impressive, a sense of touch as known as haptics plays an important role in our new phase of technology. It is the science of applying touch sensation and control to interaction with computer applications by using special input/output devices. It gives users a slight jolt of energy at the point of touch, providing instant sensory feedback, while reducing the audio, visual or audio-visual demand. Haptic technology is an evolutionary step into interacting with objects as an extension of our mind and allows for more socially appropriate and subtle interaction. In this thesis, the benefits of haptic feedback in a mobile phone camera are explored and compared to the existing feedback mechanisms. Discovering expectations from users and gathering ideas in order to improve user experience in haptic feedback of a mobile phone camera will be the main focus as well as to understand “What make end users to use or not to use mobile phone camera?” and “What qualities of haptics could be used in the design of the user interface for mobile phone camera?”. Depending on the settings and the quality of the mobile phones, the feedback from the camera can affect the user experience in many ways. I believe that to improve the existing feedback by applying haptic output such as a vibration or a vibrotactile signal may also considerably improve the user experience. Because haptic feedback is a new technology and proved to be efficient, to apply it to the mobile phone camera feedback should provide better support for users when compared to the existing feedback signals, which are audio and visual only. One of the main objectives was to analyze the users’ needs and expectations regarding the mobile phone camera haptic feedback and applications in various types of difficult situations and challenges users have encountered. Therefore, a user study was done at the beginning of the thesis work. Its aim was to get general results, which can be applied to haptic interaction on the mobile phone camera in order to improve existing applications and help easing users in their photo taking activities with their mobile phone camera. In addition, the results are considered to provide input for further studies as well as to offer concrete input to the development of a prototype

Improving command selection in smart environments by exploiting spatial constancy

With the a steadily increasing number of digital devices, our environments are becoming increasingly smarter: we can now use our tablets to control our TV, access our recipe database while cooking, and remotely turn lights on and off. Currently, this Human-Environment Interaction (HEI) is limited to in-place interfaces, where people have to walk up to a mounted set of switches and buttons, and navigation-based interaction, where people have to navigate on-screen menus, for example on a smart-phone, tablet, or TV screen. Unfortunately, there are numerous scenarios in which neither of these two interaction paradigms provide fast and convenient access to digital artifacts and system commands. People, for example, might not want to touch an interaction device because their hands are dirty from cooking: they want device-free interaction. Or people might not want to have to look at a screen because it would interrupt their current task: they want system-feedback-free interaction. Currently, there is no interaction paradigm for smart environments that allows people for these kinds of interactions. In my dissertation, I introduce Room-based Interaction to solve this problem of HEI. With room-based interaction, people associate digital artifacts and system commands with real-world objects in the environment and point toward these real-world proxy objects for selecting the associated digital artifact. The design of room-based interaction is informed by a theoretical analysis of navigation- and pointing-based selection techniques, where I investigated the cognitive systems involved in executing a selection. An evaluation of room-based interaction in three user studies and a comparison with existing HEI techniques revealed that room-based interaction solves many shortcomings of existing HEI techniques: the use of real-world proxy objects makes it easy for people to learn the interaction technique and to perform accurate pointing gestures, and it allows for system-feedback-free interaction; the use of the environment as flat input space makes selections fast; the use of mid-air full-arm pointing gestures allows for device-free interaction and increases awareness of other’s interactions with the environment. Overall, I present an alternative selection paradigm for smart environments that is superior to existing techniques in many common HEI-scenarios. This new paradigm can make HEI more user-friendly, broaden the use cases of smart environments, and increase their acceptance for the average user

A Phenomenological approach to media art environments: The Immersive art experience and the Finnish art scene

This research focuses on immersive art, defined as a multimedia experience where visitors interact with artwork whilst immersed in a range of sensory experiences. In this dissertation, I investigate the immersive art experience from the perspective of art history, social theory, and media studies situated within a phenomenological theoretical framework. I present a comparative analysis of forms of immersive spatiality, including projected moving-image art, spatial environments, participatory installations, video art installations and interactive environments in the international art scene. One of my objectives is to emphasise the role of video art in the development of interactive and immersive art environments. The growing importance of spectators for giving meaning to the artwork allows immersivity to be analysed in relation to the notions of spectacle and spectatorship. I connect disciplines, practices and concepts by adopting principles from Maurice Merleau-Ponty’s phenomenological writings. Spatiality and motility are pivotal points in immersive experiences. Immersive art, as an embodied mutual experience, materialises the phenomenological concepts of spectatorship, corporeality, motility, porosity, chiasm, and encounter. I have selected a group of relevant Finnish artists from different generations to characterise the development of media art and, particularly, immersive media art in an international context. The group includes Eija-Liisa Ahtila, Lauri Astala, Laura Beloff, Hanna Haaslahti, Tuomas A. Laitinen, Erkka Nissinen, and Marjatta Oja. I examine the historical dissemination of phenomenology in Finland and a renewed interest in the 1990s which coincided with the spatialisation of video art and the emergence of immersivity. I also investigate the opening of Kiasma Museum of Contemporary Art and its impact on Finnish culture, and the recent Amos Rex Museum, specifically built for immersive exhibitions. Regarding the unstable nature of media art, I analyse the changes in displaying art collections and exhibitions, the new commitments of art museums and the innovative directions taken by media conservators. My examination of immersive art, with its performativity and transience, reveals environmentally friendly and sustainable aspects.Fenomenologinen tulokulma mediataideympäristöihin. Immersiivinen taidekokemus ja Suomen taidekenttä Tämä tutkimus käsittelee immersiivistä taidetta multimediaalisena kokemuksena. Immersiossa kävijät ovat erilaisten aistimellisten kokemusten ympäröiminä vuorovaikutuksessa taiteen kanssa. Tutkin väitöskirjassani immersiivistä taidekokemusta fenomenologisessa teoriakehyksessä taidehistorian, yhteiskuntateorian ja mediatutkimuksen näkökulmasta. Esitän vertailevan analyysin immersiivisistä tilallisuuden muodoista, joihin sisällytän liikkuvan kuvan projisoinnit, tilateokset, osallistavat installaatiot, videoinstallaatiot ja interaktiiviset ympäristöt kansainvälisen taidekentän ilmiöinä. Yhtenä pyrkimyksenäni on painottaa videotaiteen merkitystä interaktiivisen ja immersiivisen taiteen kehityksessä. Katsojien kasvava rooli taideteoksen merkityksen muodostuksessa tarjoaa perustan immersion analyysille nimenomaan spektaakkelin ja katsojuuden viitekehyksessä. Yhdistän eri tieteenaloja, käytäntöjä ja käsitteitä toisiinsa Maurice Merleau-Pontyn fenomenologisten kirjoitusten avulla. Tilallisuus ja liike ovat immersiivisten kokemusten ytimessä. Jaettuna ruumiillisena kokemuksena immersiivinen taide ilmentää materiaalisesti fenomenologisia katsojuuden, ruumiillisuuden, liikkeessä olemisen, huokoisuuden, kiasman ja kohtaamisen käsitteitä. Olen valinnut joukon eri sukupolvia edustavia suomalaistaiteilijoita hahmot-taakseni mediataiteen ja erityisesti immersiivisen mediataiteen kansainvälisiä kehityskulkuja. Heihin lukeutuvat Eija-Liisa Ahtila, Lauri Astala, Laura Beloff, Hanna Haaslahti, Tuomas A. Laitinen, Erkka Nissinen ja Marjatta Oja. Käsittelen fenomenologian saapumista Suomeen sekä siihen 1990-luvulla videotaiteen tilallistumisen ja immersion esiin nousun yhteydessä uudelleen virinnyttä mielenkiintoa. Tarkastelen myös Nykytaiteen museo Kiasman perustamista ja sen vaikutusta suomalaiseen kulttuuriin, samoin kuin vastikään avattua Amos Rex -taidemuseota, joka on rakennettu erityisesti immersiivisiä näyttelyitä silmällä pitäen. Analysoin muutoksia taidekokoelmien ja näyttelyiden esillepanossa, taidemuseoiden uudenlaisia sitoumuksia ja mediataiteen kuratoinnin uutta luovia suuntia suhteessa mediataiteen nopeasti muuttuvaan luonteeseen. Painottamalla performatiivisuutta ja hetkellisyyttä nostan immersiivisen taiteen analyysissani näkyville sen ympäristöystävällisiä ja kestäviä ulottuvuuksia

Touching is believing: creating illusions and feeling of embodiment with mid-air haptic technology

Over the last two decades, the sense of touch has received new attention from the scientific community.Several haptic devices have been developed to address the complexity of the sense of touch, the latest addition being mid-air (contactless) haptic technology. An interesting series of previous research has suggested an easier way to tackle the complexity of designing convincing tactile sensations by exploiting tactile illusions. Tactile illusions rely on perceptual shortcuts based on the psychophysics of the tactile receptors. Currently, studies exploring the perceptual space of mid-air haptics and its applicability in the tactile illusions field are still limited in number. This thesis aims to contribute to the field of Human-Computer Interaction (HCI) by investigating the perceptual design space of ultrasonic mid-air haptics technology. Specifically, in a first set of three studies, we investigate the absolute thresholds (minimal amount of a property of astimulus that a user can detect) for control points (CP) at different frequencies on the hand and arm (Study 1). Then we investigate the optimal sampling rate needed to drive the device in an optimal fashion and its relationship with shape size (Study 2). Next, we apply a new technique to increase users’ performance in a shape discrimination task (Study 3). In Study 4, we start the exploration of a tactile illusion of movement using contact touch and later, we apply a similar procedure to investigate the feasibility of creating a tactile illusion of movement between the two non-interconnected hands by using mid-air touch (Study 5). Finally, in Study 6, we explore our sense of touch in VR, while providing an illusion of rain drops through mid-air haptics, to recreate a virtual hand illusion (VHI) to explore the boundaries of our sense of embodiment. Therefore, the contribution of this work is threefold: a) we contribute by adding new knowledge on the psychophysical space for mid-air haptics, b) we test the potential to create realistic tactile sensations by exploiting tactile illusions with mid-air haptic technology, and c) we demonstrate how tactile illusions mediated by mid-air haptics can convey a sense of embodiment in VR environments