Group reaching over digital tabletops with digital arm embodiments

In almost all collaborative tabletop tasks, groups require coordinated access to the shared objects on the table’s surface. The physical social norms of close-proximity interactions built up over years of interacting around other physical bodies cause people to avoid interfering with other people (e.g., avoiding grabbing the same object simultaneously). However, some digital tabletop situations require the use of indirect input (e.g., when using mice, and when supporting remote users). With indirect input, people are no longer physically embodied during their reaching gestures, so most systems provide digital embodiments – visual representations of each person – to provide feedback to both the person who is reaching and to the other group members. Tabletop arm embodiments have been shown to better support group interactions than simple visual designs, providing awareness of actions to the group. However, researchers and digital tabletop designers know little of how the design of digital arm embodiments affects the fundamental group tabletop interaction of reaching for objects. Therefore, in this thesis, we evaluate how people coordinate their interactions over digital tabletops when using different types of embodiments. Specifically, in a series of studies, we investigate how the visual design (what they look like) and interaction design (how they work) of digital arm embodiments affects a group’s coordinative behaviours in an open- ended parallel tabletop task. We evaluated visual factors of size, transparency, and realism (through pictures and videos of physical arms), as well as interaction factors of input and augmentations (feedback of interactions), in both a co-located and distributed environment. We found that the visual design had little effect on a group’s ability to coordinate access to shared tabletop items, that embodiment augmentations are useful to support group coordinative actions, and that there are large differences when the person is not physically co-present. Our results demonstrate an initial exploration into the design of digital arm embodiments, providing design guidelines for future researchers and designers to use when designing the next generation of shared digital spaces

Design and semantics of form and movement (DeSForM 2006)

Design and Semantics of Form and Movement (DeSForM) grew from applied research exploring emerging design methods and practices to support new generation product and interface design. The products and interfaces are concerned with: the context of ubiquitous computing and ambient technologies and the need for greater empathy in the pre-programmed behaviour of the ‘machines’ that populate our lives. Such explorative research in the CfDR has been led by Young, supported by Kyffin, Visiting Professor from Philips Design and sponsored by Philips Design over a period of four years (research funding £87k). DeSForM1 was the first of a series of three conferences that enable the presentation and debate of international work within this field: • 1st European conference on Design and Semantics of Form and Movement (DeSForM1), Baltic, Gateshead, 2005, Feijs L., Kyffin S. & Young R.A. eds. • 2nd European conference on Design and Semantics of Form and Movement (DeSForM2), Evoluon, Eindhoven, 2006, Feijs L., Kyffin S. & Young R.A. eds. • 3rd European conference on Design and Semantics of Form and Movement (DeSForM3), New Design School Building, Newcastle, 2007, Feijs L., Kyffin S. & Young R.A. eds. Philips sponsorship of practice-based enquiry led to research by three teams of research students over three years and on-going sponsorship of research through the Northumbria University Design and Innovation Laboratory (nuDIL). Young has been invited on the steering panel of the UK Thinking Digital Conference concerning the latest developments in digital and media technologies. Informed by this research is the work of PhD student Yukie Nakano who examines new technologies in relation to eco-design textiles

Embodied geosensification-models, taxonomies and applications for engaging the body in immersive analytics of geospatial data

This thesis examines how we can use immersive multisensory displays and body-focused interaction technologies to analyze geospatial data. It merges relevant aspects from an array of interdisciplinary research areas, from cartography to the cognitive sciences, to form three taxonomies that describe the senses, data representations, and interactions made possible by these technologies. These taxonomies are then integrated into an overarching design model for such "Embodied Geosensifications". This model provides guidance for system specification and is validated with practical examples

Creating and evaluating embodied interactive experiences: case studies of full-body, sonic and tactile enaction.

This thesis contributes to the field of embodied and multimodal interaction by presenting the development of different original interactive systems. Using a constructive approach, a variety of real-time user interaction situations were designed and tested, two cases of human-virtual character bodily interaction, two interactive sonifications of trampoline jumping, collaborative interaction in mobile music performance and tangible and tactile interaction with virtual sounds. While diverse in terms of application, all the explored interaction techniques belong to the context of augmentation and are grounded in the theory of embodiment and strategies for natural human-computer interaction (HCI). The cases have been contextualized within the umbrella of enaction, a paradigm of cognitive science that addresses the user as an embodied agent situated in an environment and coupled to it through sensorimotor activity. This activity of sensing and action is studied through different modalities: auditory, tactile and visual and combinations of these. The designed applications aim at a natural interaction with the system, being full-body, tangible and spatially aware. Particularly sonic interaction has been explored in the context of music creation, sports and auditory display. These technology-mediated scenarios are evaluated in order to understand what the adopted interaction techniques can bring to the user experience, how they modify impressions and enjoyment. The publications also discuss the enabling technologies used for the development, including motion tracking and programmed hardware for the tactile-sonic interaction and sonic and tangible interaction. Results show that combining full-body interaction with auditory augmentation and sonic interaction can modify the perception, observed behavior and emotion during the experience. Using spatial interaction together with tangible interaction or tactile feedback provides for a multimodal experience of exploring a mixed reality environment where audio can be accessed and manipulated with natural interaction. Embodied and spatial interaction brings playfulness to a mobile music improvisation, shifting the focus of the experience from music-making towards movement-based gaming. Finally, two novel implementations of full-body interaction based on the enactive paradigm are presented. In these designed scenarios of enaction the participant is motion tracked and a virtual character rendered as a stick figure is displayed in front of her on a screen. Results from the user studies show how the involvement of the body is crucial in understanding the behavior of a virtual character or a digital representation of the self in a gaming scenario

Embodied gestures

This is a book about musical gestures: multiple ways to design instruments, compose musical performances, analyze sound objects and represent sonic ideas through the central notion of ‘gesture’. The writers share knowledge on major research projects, musical compositions and methodological tools developed among different disciplines, such as sound art, embodied music cognition, human-computer interaction, performative studies and artificial intelligence. They visualize how similar and compatible are the notions of embodied music cognition and the artistic discourses proposed by musicians working with ‘gesture’ as their compositional material. The authors and editors hope to contribute to the ongoing discussion around creative technologies and music, expressive musical interface design, the debate around the use of AI technology in music practice, as well as presenting a new way of thinking about musical instruments, composing and performing with them

Touch-sensitive : cybernetic images and replicant bodies in the post-industrial age

This thesis uses Deleuzian cybernetics to advance upon post-modern accounts of the contemporary image economy. It begins with the hypothesis that the schizophrenic behaviours of late capitalism have induced an irreparable crisis in the inherited `specular economy' (Irigaray). This is manifested as the breakdown of the laws of generalised equivalence between truth, value and meaning and the end of a stable signifier-signified relationship - theorised as the escape of reality into 'hyperreality', or the world become simulation according to Baudrillard. It will expose the insufficiency of post-modern accounts which theorise this crisis in representation via methods which fail to escape their own always already representational terms and it will then rigorously follow through the implications of an image economy which is constituted by simulations which are `genuinely' sourceless, which do not imitate a prior reality but which rather synthesise forces and relations. To escape the closed loop of representationalism, it will divert attention away from the signifier and will concentrate on the sub-representational power of images to re-engineer reality and to re-invent the limits of the body. Using the theory and practice of Deleuze, Spinoza, Bergson, Benjamin and Virilio, it will treat images as planes of corporeal becoming - as material entities, virtual avatars, possessional states and conductors of pre-personal affect. Post-modem accounts which cite the overwhelming predominance of images sit uncomfortably with the theories of French anti-ocularcentrism - accessed here via Irigaray and Lyotard - which mark the demise of vision and its attached representational order. This paradox requires that a new perceptual relation be mapped - figured here as entirely corporeal, as tactile and synesthetic (Mcluhan) and therefore immersive. Both 'affect' and 'intensity', as modes of pre-personal perception, will be treated as tactile interactions for these responses to images demand that a body be always 'in touch' with its environment, always anorganically altering its perceptual capacities by rules of feedback. It will be argued that in this reality studio, the body no longer perceives via a specular light source, solid form and assumed phallocentric meaning. The proposed synthesis between cybernetic imaging technologies, immanent perceptual criteria and the ever-changing state of the body requires an engagement with the female since she bears a privileged relation to this scenario. In the specular economy, women have been assumed, like faithful images, to secondarily reproduce an underlying, phallocentric truth. However, it will be shown that just as images can work nonrepresentationally, so too can female bodies; on the one hand appearing representational but on the other conducting radically subversive effects. Where bodies and images are such simulatory becomings it will be shown how the female is neither representationally ordered (social constructivism) nor essentially defined (biological reductivism) but is rather cybernetically engineered. Throughout, her privileged access to the virtual realm beyond language will be used to substantiate the major claim of this thesis that cybernetic simulation is more concerned with the material alteration of an environment rather than with the implementation of linguistic obligation

The embodied user : corporeal awareness & media technology

Human beings are proficient users of tools and technology. At times, our interactions with a technological artifact appear so effortless, that the distinction between the artifact and the body starts to fade. When operating anthropomorphically designed teleoperation systems, for example, some people develop the vivid experience that they are physically there at the remote site (i.e., telepresence). Others might even come to sense the slave robot’s arms and hands as their own. The process in which the central nervous system categorizes an object as a part of the body, and in which a discrimination is made between what is contained within and outside the bodily boundaries, is called self-attribution. The aim of this thesis is twofold: (a) To determine the personal factors (e.g., the characteristics of an individual’s psychological makeup) and situational factors (e.g., the appearance of objects) that constrain or facilitate self-attribution, and (b) to determine the degree to which these factors affect people’s experiences with media technology. In Chapter 2, we describe the theoretical framework of our research which is centered on a conception of the user of technology as an embodied agent. In this chapter we distinguish two important, but often confused aspects of embodiment: the body schema, and the body image. The body schema is defined as a dynamic distributed network of procedures aimed at guiding behavior. In contrast, we defined the body image as a part of the process of consciousness and, thus, as consisting of those higher-order discriminations (or qualia) that pertain to the body, and one’s self-perception thereof. To investigate the individual and situational factors that constrain or facilitate selfattribution (i.e., incorporation into the body image), we employ the experimental paradigm of the rubber-hand illusion (Botvinick & Cohen, 1998). In this illusion, which is induced by stroking a person’s concealed hand together with a visible fake one, some people start to sense the fake hand as an actual part of their body. In Chapter 3, we investigate the rubberhand illusion under two mediated conditions: (1) a virtual reality condition, where both the fake hand and its stimulation were projected on the table in front of the participant, and (2) a mixed reality condition, where the fake hand was projected, but its stimulation was unmediated. Our experiment reveals that people can develop the rubber-hand illusion under mediated conditions, but the resulting illusion may, depending on the technology used, be less vivid than in the traditional unmediated setup. In Chapter 4, we investigate the extent to which visual discrepancies between the foreign object and a human hand affect people in developing a vivid rubber-hand illusion. We found that people experience a more vivid illusion when the foreign object resembles the human hand in terms of both shape and texture. Taken together, the experiments in Chapters 3 and 4 support the view that the rubber-hand illusion is not merely governed by a bottom-up process (i.e., based on visuotactile integration), but is affected, top-down, by a cognitive representation of what the human body is like (e.g., Tsakiris and Haggard, 2005). In the rubber-hand illusion, people commonly misperceive the location of their concealed hand toward the direction of the fake hand (Tsakiris & Haggard, 2005). As such, this so-called proprioceptive drift is often used as an alternative to self-reports in assessing the vividness of the illusion (e.g., Tsakiris & Haggard, 2005). In Chapter 5, we investigate the extent to which the observed shift in felt position of the concealed hand can be attributed to experiencing the illusion. For this purpose, we test how various features of the experimental setup of the rubber-hand illusion, which in themselves are not sufficient to elicit the illusion, affect proprioceptive drift. We corroborate existing research which demonstrates that looking at a fake hand or a tabletop for five minutes, in absence of visuotactile stimulation, is sufficient to induce a change in the felt position of an unseen hand (e.g., Gross et al., 1974). Moreover, our experiments indicate that the use of proprioceptive drift as a measure for the strength of the rubber-hand illusion yields different conclusions than an assessment by means of self-reports. Based on these results, we question the validity of proprioceptive drift as an alternative measure of the vividness of the rubber-hand illusion. In Chapter 6, we propose and test a model of the vividness of the rubber-hand illusion. In two experiments, we successfully modeled people’s self-reported experiences related to the illusion (e.g., "the fake hand felt as my own") based on three estimates: (a) a person’s susceptibility for the rubber-hand illusion, (b) the processing demand that is required for a particular experience, and (c) the suppression/constraints imposed by the situation. We demonstrate that the impressions related to the rubber-hand illusion, and by inference the processes behind them, are comparable for different persons. This is a non-trivial finding as such invariance is required for an objective scaling of individual susceptibility and situational impediment on the basis of self-reported experiences. Regarding the validity of our vividness model, we confirm that asynchrony (e.g., Botvinick & Cohen, 1998) and information-poor stimulation (e.g., Armel & Ramachandran, 2003) constrain the development of a vivid rubber-hand illusion. Moreover, we demonstrate that the correlation between a person’s susceptibility for the rubber-hand illusion and the extent of his of her proprioceptive drift is fairly moderate, thereby confirming our conclusions from Chapter 5 regarding the limited validity of proprioceptive drift as a measure of the vividness of the rubber-hand illusion. In Chapter 7, we investigate the extent to which the large individual differences in people’s susceptibility for the illusion can be explained by body image instability, and the ability to engage in motor imagery of the hand (i.e., in mental own hand transformations). In addition, we investigate whether the vividness of the illusion is dependent on the anatomical implausibility of the fake hand’s orientation. With respect to body image instability, we corroborate a small, but significant, correlation between susceptibility and body image aberration scores: As expected, people with a more unstable body image are also more susceptible to the rubber-hand illusion (cf. Burrack & Brugger, 2005). With respect to the position and orientation of the fake hand on the table, we demonstrate that people experience a less vivid rubber-hand illusion when the fake hand is orientated in an anatomically impossible, as compared to an anatomically possible manner. This finding suggests that the attribution of foreign objects to the self is constrained by the morphological capabilities of the human body. With respect to motor imagery, our results indicate a small, but significant, correlation between susceptibility and response times to a speeded left and right hands identification task. In other words, people who are more attuned to engage in mental own hand transformations are also better equipped to develop vivid rubber-hand illusions. In Chapter 8, we examine the role of self-attribution in the experience of telepresence. For this purpose, we introduce the technological domain of mediated social touch (i.e., interpersonal touching over a distance). We anticipated that, compared to a morphologically incongruent input medium, a morphologically congruent medium would be more easily attributed to the self. As a result, we expected our participants to develop a stronger sense of telepresence when they could see their interaction partner performing the touches on a sensor-equipped mannequin as opposed to a touch screen. Our participants, as expected, reported higher levels of telepresence, and demonstrated more physiological arousal with the mannequin input medium. At the same time, our experiment revealed that these effects might not have resulted from self-attribution, and thus that other psychological mechanisms of identification might play a role in telepresence experiences. In Chapter 9, the epilogue, we discuss the main contributions and limitations of this thesis, while taking a broader perspective on the field of research on media technologies and corporeal awareness