VR/Urban: spread.gun - design process and challenges in developing a shared encounter for media façades

Designing novel interaction concepts for urban environments is not only a technical challenge in terms of scale, safety, portability and deployment, but also a challenge of designing for social configurations and spatial settings. To outline what it takes to create a consistent and interactive experience in urban space, we describe the concept and multidisciplinary design process of VR/Urban's media intervention tool called Spread.gun, which was created for the Media Façade Festival 2008 in Berlin. Main design aims were the anticipation of urban space, situational system configuration and embodied interaction. This case study also reflects on the specific technical, organizational and infrastructural challenges encountered when developing media façade installations

'The Ghosts of Roller Disco', a Choreographed, Interactive Performance for Robotic Roller Skates

The project investigates how interactions with complex (biologically inspired swarming) behaviors of multiple robots are understood by human participants within a performative and dramaturgical system. Nonanthropomorphic robots in the form of roller skates are used in innovative ways by creating social formations from their movements, for example a leader and followers in a conga line. Synchronized audio signals and speech-like sonic structures are used in innovative ways by influencing and engaging the participant's interactions with the robots. Localization data of the robots in space is mapped to control the surround sound and lighting within the space. This is used to enhance audience immersion and engagement within the interactive performance work

Rafigh: A Living Media System for Motivating Target Application Use for Children

Digital living media systems combine living media such as plants, animals and fungi with computational components. In this dissertation, I respond to the question of how can digital living media systems better motivate children to use target applications (i.e., learning and/or therapeutic applications)? To address this question, I employed a participatory design approach where I incorporated input from children, parents, speech language pathologists and teachers into the design of a new system. Rafigh is a digital embedded system that uses the growth of a living mushrooms colony to provide positive reinforcements to children when they conduct target activities. The growth of the mushrooms is affected by the amount of water administered to them, which in turn corresponds to the time children spend on target applications. I used an iterative design process to develop and evaluate three Rafigh prototypes. The evaluations showed that the system must be robust, customizable, and should include compelling engagement mechanisms to keep the children interested. I evaluated Rafigh using two case studies conducted in participants homes. In each case study, two siblings and their parent interacted with Rafigh over two weeks and the parents identified a series of target applications that Rafigh should motivate the children to use. The study showed that Rafigh motivated the children to spend significantly more time on target applications during the intervention phase and that it successfully engaged one out of two child participants in each case study who showed signs of responsibility, empathy and curiosity towards the living media. The study showed that the majority of participants described the relationship between using target applications and mushrooms growth correctly. Further, Rafigh encouraged more communication and collaboration between the participants. Rafighs slow responsivity did not impact the engagement of one out of two child participants in each case study and might even have contributed to their investment in the project. Finally, Rafighs presence as an ambient physical object allowed users to interact with it freely and as part of their home environment

Large Language Models for in Situ Knowledge Documentation and Access With Augmented Reality

Augmented reality (AR) has become a powerful tool for assisting operators in complex environments, such as shop floors, laboratories, and industrial settings. By displaying synthetic visual elements anchored in real environments and providing information for specific tasks, AR helps to improve efficiency and accuracy. However, a common bottleneck in these environments is introducing all necessary information, which often requires predefined structured formats and needs more ability for multimodal and Natural Language (NL) interaction. This work proposes a new method for dynamically documenting complex environments using AR in a multimodal, non-structured, and interactive manner. Our method employs Large Language Models (LLMs) to allow experts to describe elements from the real environment in NL and select corresponding AR elements in a dynamic and iterative process. This enables a more natural and flexible way of introducing information, allowing experts to describe the environment in their own words rather than being constrained by a predetermined structure. Any operator can then ask about any aspect of the environment in NL to receive a response and visual guidance from the AR system, thus allowing for a more natural and flexible way of introducing and retrieving information. These capabilities ultimately improve the effectiveness and efficiency of tasks in complex environments

Acoustic heritage and audio creativity: the creative application of sound in the representation, understanding and experience of past environments

Acoustic Heritage is one aspect of archaeoacoustics, and refers more specifically to the quantifiable acoustic properties of buildings, sites and landscapes from our architectural and archaeological past, forming an important aspect of our intangible cultural heritage. Auralisation, the audio equivalent of 3D visualization, enables these acoustic properties, captured via the process of measurement and survey, or computer based modelling, to form the basis of an audio reconstruction and presentation of the studied space. This paper examines the application of auralisation and audio creativity as a means to explore our acoustic heritage, thereby diversifying and enhancing the toolset available to the digital heritage or humanities researcher. The Open Acoustic Impulse Response (OpenAIR) library is an online repository for acoustic impulse response and auralisation data, with a significant part having been gathered from a broad range of heritage sites. The methodology used to gather this acoustic data is discussed, together with the processes used in generating and calibrating a comparable computer model, and how the data generated might be analysed and presented. The creative use of this acoustic data is also considered, in the context of music production, mixed media artwork and audio for gaming. More specifically to digital heritage is how these data can be used to create new experiences of past environments, as information, interpretation, guide or artwork and ultimately help to articulate new research questions and explorations of our acoustic heritage

Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

Paralinguistic vocal control of interactive media: how untapped elements of voice might enhance the role of non-speech voice input in the user's experience of multimedia.

Much interactive media development, especially commercial development, implies the dominance of the visual modality, with sound as a limited supporting channel. The development of multimedia technologies such as augmented reality and virtual reality has further revealed a distinct partiality to visual media. Sound, however, and particularly voice, have many aspects which have yet to be adequately investigated. Exploration of these aspects may show that sound can, in some respects, be superior to graphics in creating immersive and expressive interactive experiences. With this in mind, this thesis investigates the use of non-speech voice characteristics as a complementary input mechanism in controlling multimedia applications. It presents a number of projects that employ the paralinguistic elements of voice as input to interactive media including both screen-based and physical systems. These projects are used as a means of exploring the factors that seem likely to affect users’ preferences and interaction patterns during non-speech voice control. This exploration forms the basis for an examination of potential roles for paralinguistic voice input. The research includes the conceptual and practical development of the projects and a set of evaluative studies. The work submitted for Ph.D. comprises practical projects (50 percent) and a written dissertation (50 percent). The thesis aims to advance understanding of how voice can be used both on its own and in combination with other input mechanisms in controlling multimedia applications. It offers a step forward in the attempts to integrate the paralinguistic components of voice as a complementary input mode to speech input applications in order to create a synergistic combination that might let the strengths of each mode overcome the weaknesses of the other