Sentient Matter: Towards Affective Human-Architecture Interaction

Interactive design has been embedded into every aspect of our lives. Ranging from handy devices to architecturally scaled environments, these designs have not only shifted the way we facilitate interaction with other people, but they also actively reconfigure themselves in response to human stimuli. Following in the wake of interactive experimentation, sentient matter, the idea that matter embodies the capacity to perceive and respond to stimuli, attempts to engage in a challenging arena that few architects and architectural researchers have ventured into. In particular, the creation and simulation of emotive types of interaction between the architectural environment and its inhabitants. This ambition is made possible by the collaboration of multiple disciplines. Cybernetics, specifically the legacy of Pask’s conversation theory, inspires this thesis with the question of why emotion is needed in facilitating human–architecture communication; why emotion appraisal theory (P. Desmet) within psychology supports the feasibility of an architectural environment to elicit emotional changes on its participant as well as the possibility of generating a next-step response by having the participant’s emotive behaviors observed; and why movement notation systems, especially Laban Movement Analysis (a movement rating scale system), helps us to understand how emotions can be identified by motion elements that signify emotive behavior. Through the process of decomposing movement into several qualitative and quantitative factors such as velocity, openness, and smoothness, emotions embodied in motion can be detected and even manipulated by altering those movement factors. Moreover, with the employment of a Kinect sensor, live performance can be analyzed in real time. Based on the above research and inspired by the Kinetic sculptures of Margolin, the final product of this thesis is the development of a prototype that translates human movements that are expressive of emotion into continuous surface transformations, thus making evident how such emotive states might be transcoded into an architectural form. In this process, four typical emotive architectural expressions—joy, anger, excited, and sadness—are researched. This thesis also documents three virtual scenarios in order to examine the effect of this interactive system. Different contexts, kinetic types, and behavioral strategies are presented so that we may explore their potential applications. Sentient matter outlines a framework of syntheses, which is built upon the convergence of embedded computation (intelligence) and physical counterpart (kinetics). In the entire process, it considers people’s participation as materials that fuel the generation of legible emotional behaviors within an architectural environment. Consequently, there is potential for an architectural learning capacity coupled with an evolving data library of human behavioral knowledge. This opens doors for futuristic designs where the paradigm shifts from “What is that building?” to “What is that building doing?

Choreographic and Somatic Approaches for the Development of Expressive Robotic Systems

As robotic systems are moved out of factory work cells into human-facing environments questions of choreography become central to their design, placement, and application. With a human viewer or counterpart present, a system will automatically be interpreted within context, style of movement, and form factor by human beings as animate elements of their environment. The interpretation by this human counterpart is critical to the success of the system's integration: knobs on the system need to make sense to a human counterpart; an artificial agent should have a way of notifying a human counterpart of a change in system state, possibly through motion profiles; and the motion of a human counterpart may have important contextual clues for task completion. Thus, professional choreographers, dance practitioners, and movement analysts are critical to research in robotics. They have design methods for movement that align with human audience perception, can identify simplified features of movement for human-robot interaction goals, and have detailed knowledge of the capacity of human movement. This article provides approaches employed by one research lab, specific impacts on technical and artistic projects within, and principles that may guide future such work. The background section reports on choreography, somatic perspectives, improvisation, the Laban/Bartenieff Movement System, and robotics. From this context methods including embodied exercises, writing prompts, and community building activities have been developed to facilitate interdisciplinary research. The results of this work is presented as an overview of a smattering of projects in areas like high-level motion planning, software development for rapid prototyping of movement, artistic output, and user studies that help understand how people interpret movement. Finally, guiding principles for other groups to adopt are posited.Comment: Under review at MDPI Arts Special Issue "The Machine as Artist (for the 21st Century)" http://www.mdpi.com/journal/arts/special_issues/Machine_Artis

From head to toe:body movement for human-computer interaction

Our bodies are the medium through which we experience the world around us, so human-computer interaction can highly benefit from the richness of body movements and postures as an input modality. In recent years, the widespread availability of inertial measurement units and depth sensors led to the development of a plethora of applications for the body in human-computer interaction. However, the main focus of these works has been on using the upper body for explicit input. This thesis investigates the research space of full-body human-computer interaction through three propositions. The first proposition is that there is more to be inferred by natural users’ movements and postures, such as the quality of activities and psychological states. We develop this proposition in two domains. First, we explore how to support users in performing weight lifting activities. We propose a system that classifies different ways of performing the same activity; an object-oriented model-based framework for formally specifying activities; and a system that automatically extracts an activity model by demonstration. Second, we explore how to automatically capture nonverbal cues for affective computing. We developed a system that annotates motion and gaze data according to the Body Action and Posture coding system. We show that quality analysis can add another layer of information to activity recognition, and that systems that support the communication of quality information should strive to support how we implicitly communicate movement through nonverbal communication. Further, we argue that working at a higher level of abstraction, affect recognition systems can more directly translate findings from other areas into their algorithms, but also contribute new knowledge to these fields. The second proposition is that the lower limbs can provide an effective means of interacting with computers beyond assistive technology To address the problem of the dispersed literature on the topic, we conducted a comprehensive survey on the lower body in HCI, under the lenses of users, systems and interactions. To address the lack of a fundamental understanding of foot-based interactions, we conducted a series of studies that quantitatively characterises several aspects of foot-based interaction, including Fitts’s Law performance models, the effects of movement direction, foot dominance and visual feedback, and the overhead incurred by using the feet together with the hand. To enable all these studies, we developed a foot tracker based on a Kinect mounted under the desk. We show that the lower body can be used as a valuable complementary modality for computing input. Our third proposition is that by treating body movements as multiple modalities, rather than a single one, we can enable novel user experiences. We develop this proposition in the domain of 3D user interfaces, as it requires input with multiple degrees of freedom and offers a rich set of complex tasks. We propose an approach for tracking the whole body up close, by splitting the sensing of different body parts across multiple sensors. Our setup allows tracking gaze, head, mid-air gestures, multi-touch gestures, and foot movements. We investigate specific applications for multimodal combinations in the domain of 3DUI, specifically how gaze and mid-air gestures can be combined to improve selection and manipulation tasks; how the feet can support the canonical 3DUI tasks; and how a multimodal sensing platform can inspire new 3D game mechanics. We show that the combination of multiple modalities can lead to enhanced task performance, that offloading certain tasks to alternative modalities not only frees the hands, but also allows simultaneous control of multiple degrees of freedom, and that by sensing different modalities separately, we achieve a more detailed and precise full body tracking

Expanded Choreographies - Choreographic Histories

From objects to sounds, choreography is expanding beyond dance and human bodies in motion. This book offers one of the rare systematic investigations of expanded choreography as it develops in contemporaneity, and is the first to consider expanded choreography from a trans-historical perspective. Through case studies on different periods of European dance history - ranging from Renaissance dance to William Forsythe's choreographic objects and from Baroque court ballets to digital choreographies - it traces a journey of choreography as a practice transcending its sole association with dancing, moving, human bodies

Expanded Choreographies – Choreographic Histories

From objects to sounds, choreography is expanding beyond dance and human bodies in motion. This book offers one of the rare systematic investigations of expanded choreography as it develops in contemporaneity, and is the first to consider expanded choreography from a trans-historical perspective. Through case studies on different periods of European dance history – ranging from Renaissance dance to William Forsythe's choreographic objects and from Baroque court ballets to digital choreographies – it traces a journey of choreography as a practice transcending its sole association with dancing, moving, human bodies

Expanded Choreographies - Choreographic Histories

From objects to sounds, choreography is expanding beyond dance and human bodies in motion. This book offers one of the rare systematic investigations of expanded choreography as it develops in contemporaneity, and is the first to consider expanded choreography from a trans-historical perspective. Through case studies on different periods of European dance history - ranging from Renaissance dance to William Forsythe's choreographic objects and from Baroque court ballets to digital choreographies - it traces a journey of choreography as a practice transcending its sole association with dancing, moving, human bodies