Using AR and VR characters for enhancing user experience in a museum

Museums and cultural heritage institutions have used technology to create interactive exhibits and pedagogical tools that help spark visitors’ interests. The rise of Augmented, Virtual and Mixed Reality Systems has further enabled the creation of a new generation of immersive experiences that can engage and educate visitors. These technologies can be used to develop digital characters that can serve as virtual tour guides and improve user engagement by answering questions and forming social bonds with the users. While such tour guides have been deployed as exhibits at many museums, the implementation is usually limited to a single exhibit or a section of the museum space. We believe that visitors will be better served if the virtual guide not only enriches the onsite experience but also provides a take-home experience for users to encourage future visits. This thesis explores the enhancement in user experience that such a system can bring by offering onsite and offsite AR and WebVR technologies to create a virtual tour guide that assists visitors at the Genesee Country Village & Museum through interactive dialog as they explore the historic village on the museum campus

Using Synthetic Worlds for Work and Learning

Synthetic worlds [Castronova 2005] are graphically-rich, three-dimensional (3D), electronic environments where members assume an embodied persona (i.e., avatars) and engage in socializing, competitive quests, and economic transactions with globally distributed others. Frequently categorized as technologies of play, synthetic worlds range from massively multiplayer online games (MMOGs) such as World of Warcraft, to virtual reality environments such as Second Life. Increasingly, educators, researchers and corporations are recognizing these 3D online spaces as legitimate communication media, thereby blurring the lines between work and play, and between reality and virtuality. In this panel, presented at the 2007 International Conference on Information Systems, we explore how the fluid work-play and reality-virtuality boundaries are negotiated and managed in practice. The panelists will rely on their research, conducted in educational, corporate and game environments, to address questions about learning, working and playing in these new media spaces

Intermediated reality

Real-time solutions to reducing the gap between virtual and physical worlds for photorealistic interactive Augmented Reality (AR) are presented. First, a method of texture deformation with image inpainting, provides a proof of concept to convincingly re-animate fixed physical objects through digital displays with seamless visual appearance. This, in combination with novel methods for image-based retargeting of real shadows to deformed virtual poses and environment illumination estimation using in conspicuous flat Fresnel lenses, brings real-world props to life in compelling, practical ways. Live AR animation capability provides the key basis for interactive facial performance capture driven deformation of real-world physical facial props. Therefore, Intermediated Reality (IR) is enabled; a tele-present AR framework that drives mediated communication and collaboration for multiple users through the remote possession of toys brought to life.This IR framework provides the foundation of prototype applications in physical avatar chat communication, stop-motion animation movie production, and immersive video games. Specifically, a new approach to reduce the number of physical configurations needed for a stop-motion animation movie by generating the in-between frames digitally in AR is demonstrated. AR-generated frames preserve its natural appearance and achieve smooth transitions between real-world keyframes and digitally generated in-betweens. Finally, the methods integrate across the entire Reality-Virtuality Continuum to target new game experiences called Multi-Reality games. This gaming experience makes an evolutionary step toward the convergence of real and virtual game characters for visceral digital experiences

Because I am Not Here, Selected Second Life-Based Art Case Studies. Subjectivity, Autoempathy and Virtual World Aesthetics

Second Life is a virtual world accessible through the Internet in which users create objects and spaces, and interact socially through 3D avatars. Certain artists use the platform as a medium for art creation, using the aesthetic, spatial, temporal and technological features of SL as raw material. Code and scripts applied to animate and manipulate objects, avatars and spaces are important in this sense. These artists, their avatars and artwork in SL are at the centre of my research questions: what does virtual existence mean and what is its purpose when stemming from aesthetic exchange in SL? Through a qualitative research method mixing distribute aesthetics, digital art and media theories, the goal is to examine aesthetic exchange in the virtual: subjectivity and identity and their possible shifting patterns as reflected in avatar-artists. A theoretical and methodological emphasis from a media studies perspective is applied to digital media and networks, contributing to the reshaping of our epistemologies of these media, in contrast to the traditional emphasis on communicational aspects. Four case studies, discourse and text analysis, as well as interviews in-world and via email, plus observation while immersed in SL, are used in the collection of data, experiences, objects and narratives from avatars Eva and Franco Mattes, Gazira Babeli, Bryn Oh and China Tracy. The findings confirm the role that aesthetic exchange in virtual worlds has in the rearrangement of ideas and epistemologies on the virtual and networked self. This is reflected by the fact that the artists examined—whether in SL or AL—create and embody avatars from a liminal (ambiguous) modality of identity, subjectivity and interaction. Mythopoeia (narrative creation) and experiencing oneself as ‘another’ through multiplied identity and subjectivity are the outcomes of code performance and machinima (films created in-world). They constitute a modus operandi (syntax) in which episteme, techne and embodiment work in symbiosis with those of the machine, affected by the synthetic nature of code and liminality in SL. The combined perspective from media studies and distribute aesthetics proves to be an effective method for studying these subjects, contributing to the discussion of contemporary virtual worlds and art theories

Reconfiguring the reader : convergence and participation in modern young adult fantasy fiction

PhD ThesisThis thesis explores digital-age literary and reading practices as they were influenced by participatory culture at the turn of the century. Participatory culture is analysed here through the work of Henry Jenkins, Hans Heino Ewers, Margaret Mackey and Katy Varnelis and is recognised as one in which individuals are socially connected to each other in an environment that offers support for creating and sharing interpretations and original works. It has relatively low barriers to artistic expression and civic participation, and fosters the sense of community growing around people’s common interests and ideologies, as expressed through performative manifestations such as gaming and fandom. Because juvenile fantasy fiction generally, and J. K. Rowling’s Harry Potter series (1997-2007) specifically, were at the centre of significant developments in response to participatory culture, Rowling’s books are used as a case study on the basis of which changing practices of reading, writing and interpretation of story, principally by children and young people, are mapped and appraised. One aim of this thesis is to evaluate how far participatory culture has affected what it means to be a reader of a text that exists in multiple formats: how each version of the text constructs and addresses its readers/viewers/players/co-creators, and the dynamics and interdependence between the different versions. A second but related aim is to test the claims of new media theorists, including Janet Murray, Pierre Lévy and Marie-Laure Ryan, among others, to establish how far texts, readers and the processes of reading have in fact changed. Specifically, it looks at how far the promises of reader participation and co-creation have been fulfilled, especially within the genre of children’s literature

Voicing Kinship with Machines: Diffractive Empathetic Listening to Synthetic Voices in Performance.

This thesis contributes to the field of voice studies by analyzing the design and production of synthetic voices in performance. The work explores six case studies, consisting of different performative experiences of the last decade (2010- 2020) that featured synthetic voice design. It focusses on the political and social impact of synthetic voices, starting from yet challenging the concepts of voice in the machine and voice of the machine. The synthetic voices explored are often playing the role of simulated artificial intelligences, therefore this thesis expands its questions towards technology at large. The analysis of the case studies follows new materialist and posthumanist premises, yet it tries to confute the patriarchal and neoliberal approach towards technological development through feminist and de-colonial approaches, developing a taxonomy for synthetic voices in performance. Chapter 1 introduces terms and explains the taxonomy. Chapter 2 looks at familiar representations of fictional AI. Chapter 3 introduces headphone theatre exploring immersive practices. Chapters 4 and 5 engage with chatbots. Chapter 6 goes in depth exploring Human and Artificial Intelligence interaction, whereas chapter 7 moves slightly towards music production and live art. The body of the thesis includes the work of Pipeline Theatre, Rimini Protokoll, Annie Dorsen, Begüm Erciyas, and Holly Herndon. The analysis is informed by posthumanism, feminism, and performance studies, starting from my own practice as sound designer and singer, looking at aesthetics of reproduction, audience engagement, and voice composition. This thesis has been designed to inspire and provoke practitioners and scholars to explore synthetic voices further, question predominant biases of binarism and acknowledge their importance in redefining technology

From corporeality to virtual reality: theorizing literacy, bodies, and technology in the emerging media of virtual, augmented, and mixed realities

This dissertation explores the relationships between literacy, technology, and bodies in the emerging media of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). In response to the recent, rapid emergence of new media forms, questions arise as to how and why we should prepare to compose in new digital media. To interrogate the newness accorded to new media composing, I historicize the literacy practices demanded by new media by examining digital texts, such as video games and software applications, alongside analogous “antiquated” media, such as dioramas and museum exhibits. Comparative textual analysis of analogous digital and non-digital VR, AR, and MR texts reveals new media and “antiquated” media utilize common characteristics of dimensionality, layering, and absence/presence, respectively. The establishment of shared traits demonstrates how media operate on a continuum of mutually held textual practices; despite their distinctive forms, new media texts do not represent either a hierarchical or linear progression of maturing development. Such an understanding aids composing in new VR, AR, and MR media by enabling composers to make fuller use of prior knowledge in a rapidly evolving new media environment, a finding significant both for educators and communicators. As these technologies mature, we will continue to compose both traditional and new forms of texts. As such, we need literacy theory that attends to both the traditional and the new and also is comprehensive enough to encompass future acts of composing in media yet to emerge

Sensemaking in virtual settings: a practice-based approach

Since the mainstream uptake of computers and the internet, our world has become increasingly virtualised. Modern organisations are deeply reliant on virtual technologies to carry out their business across time and distance. Indeed, virtual technologies are now implicated in almost all organisational activities, from (virtual) meetings to (online) collaboration. Many scholars have been drawn to investigate the new organisational phenomena that have resulted from the virtualisation of our world, such a virtual learning, virtual leadership and virtual decision making. My research, however, tackles a more fundamental question about how organising more generally is accomplished in the virtual age. Namely, the research question is, “How does sensemaking, as the basis of organising, take place in virtual settings?” To explain, sensemaking – a foundational concept in Organisation Studies – underpins all organisational activities. Therefore understanding how sensemaking takes place in virtual settings will necessarily illuminate how organising more generally is accomplished virtually. To date, how sensemaking takes place in virtual settings has hardly been studied. Further, the studies that do exist impose Weick’s (1969, 1979, 1995) theory of sensemaking (which was developed at a time pre-dating virtual technologies) on to the new context. As a result, existing studies do not illuminate what is new, unique and interesting about how we make sense in virtual settings. In this thesis I develop an alternative, practice-based conception of sensemaking (which serves as the theoretical framework for the study) that sensitises me to previously overlooked but critical concepts, namely materiality, embodiment and ongoing accomplishment. First, materiality describes how things, which in virtual settings are often digital, are implicated in sensemaking. Second, embodiment describes how physical bodies, and their digital representations in virtual settings, are involved in accomplishment of activities. Finally, ongoing accomplishment describes how sensemaking takes place in the flow of activities as they are carried out in the physical world, the virtual world, or combination of both. This framework also enables me to position activities as the unit of analysis for sensemaking. Taken together, this is a novel approach that reveals new facets of the phenomenon of sensemaking in virtual settings. This theoretical framework is applied in three different fieldsites (of varying levels of virtuality) which are selected using a virtuality continuum developed within the thesis. These fieldsites are Yammer (a social media platform), telepresence (a video-based collaboration platform), and Second Life (a three-dimensional virtual world). The methodology is a hybrid traditional-virtual ethnography in which data is collected through participant observation, complemented by interviews. Empirical data are presented in the form of accounts that exemplify the key activities of practitioners in each fieldsite. The analysis reveals how sensemaking is enabled, constrained and altered owing to activities being carried out virtually (rather than in traditional settings). Further, various unique features of sensemaking as it takes place in each fieldsite are articulated, which become the subject of a cross-fieldsite comparison. By overlaying the results from each fieldsite on to the virtuality continuum, the question of how sensemaking takes place in virtual settings is answered in two ways. First, features of sensemaking that are common across all fieldsites, and therefore levels of virtuality, are identified. Second, I identify features of sensemaking that are specific to particular fieldsites and make inferences about how sensemaking features change depending on the level of virtuality of the setting. Some anomalies arising from this analysis are resolved by suggesting an alternative matrix model of virtuality which has potential to be included in future research. The findings culminate in articulation of a practice-based theoretical account of “virtual sensemaking”. This virtual sensemaking is then compared to traditional sensemaking, further illuminating the uniqueness of how sensemaking takes place in virtual settings. I then articulate contributions to the fields of sensemaking and organising as follows. This is the first study to articulate an account of sensemaking as it takes place specifically in virtual settings. Moreover, the account of virtual sensemaking broadens our understanding of sensemaking generally by opening up previously under-theorised aspects of how we accomplish (virtual) organisational activities. Contributions to broader organising include reconsideration of how we define quintessential organising activities, such as meetings. Practical implications pertain to creators, administrators and users of virtual technologies who may use this knowledge of virtual sensemaking to inform more effective and efficient design, implementation, management and application of virtual technologies in organisations. Finally, exciting avenues for future research are suggested, including opportunities to reconceptualise the theoretical, empirical and analytical landscape for investigating organising in the modern virtual age. Namely, we may let go of notions of organising that are rooted in traditional settings and embrace new conceptions of virtual organising. Organising is no longer place-specific or linear, nor does it require our physical presence or real-time participation. Instead, modern virtual organising is a complex, multi-dimensional blending of the physical and virtual. As technologies evolve and our activities become ever more integrated with them, understanding how we achieve this blending will be paramount to progressing the field of Organisation Studies generally

Experiential Perspectives on Sound and Music for Virtual Reality Technologies

This thesis examines the intersection of sound, music, and virtuality within current and next-generation virtual reality technologies, with a specific focus on exploring the experiential perspectives of users and participants within virtual experiences. The first half of the thesis constructs a new theoretical model for examining intersections of sound and virtual experience. In Chapter 1, a new framework for virtual experience is constructed consisting of three key elements: virtual hardware (e.g., displays, speakers); virtual software (e.g., rules and systems of interaction); and virtual externalities (i.e., physical spaces used for engaging in virtual experiences). Through using and applying this new model, methodical examinations of complex virtual experiences are possible. Chapter 2 examines the second axis of the thesis through constructing an understanding of how sound is designed, implemented, and received within virtual reality. The concept of soundscapes is explored in the context of experiential perspectives, serving as a useful approach for describing received auditory phenomena. Auditory environments are proposed as a new model for exploring how auditory phenomena can be broadcast to audiences. Chapter 3 explores how inauthenticity within sound can impact users in virtual experience and uses authenticity to critically examine challenges surrounding sound in virtual reality. Constructions of authenticity in music performance are used to illustrate how authenticity is constructed within virtual experience. Chapter 4 integrates music into the understanding of auditory phenomena constructed throughout the thesis: music is rarely part of the created world in a virtual experience. Rather, it is typically something which only the audience – as external observers of the created world – can hear. Therefore, music within immersive virtual reality may be challenging as the audience is placed within the created world.The second half of this thesis uses this theoretical model to consider contemporary and future approaches to virtual experiences. Chapter 5 constructs a series of case studies to demonstrate the use of the framework as a trans-medial and intra/inter-contextual tool of analysis. Through use of the framework, varying approaches to implementation of sound and music in virtual reality technologies are considered, which reveals trans-medial commonalities of immersion and engagement with virtual experiences through sound. Chapter 6 examines near-future technologies, including brain-computer interfaces and other full-immersion technologies, to identify key issues in the design and implementation of future virtual experiences and suggest how interdisciplinary collaboration may help to develop solutions to these issues. Chapter 7 considers how the proposed model for virtuality might allow for methodical examination of similar issues within other fields, such as acoustics and architecture, and examines the ethical considerations that may become relevant as virtual technology develops within the 21st Century.This research explores and rationalises theoretical models of virtuality and sound. This permits designers and developers to improve the implementation of sound and music in virtual experiences for the purpose of improving user outcomes.<br/