Interaction in immersive virtual reality:breakdowns as trouble-sources in co-present VR interaction

Abstract. This thesis examines breakdowns as trouble-sources in co-present interactions within immersive virtual reality (VR). The data examined in this study consists of video recordings of users playing Rec Room, an online multiplayer game, with HTC Vive head-mounted displays (HMDs). The users participating in the recording sessions were predominantly non-native English speakers with no prior immersive VR experience. The participants played the game in six groups of two. The participants were given minimal guidance to facilitate naturally occurring interactions within the virtual environment. Sequences of interest identified in the recorded footage were transcribed for the purpose of analysis. Conversation analysis was applied as the research methodology, while drawing upon past studies regarding the phenomenon of breakdowns in human-computer interaction and repair in interaction. Breakdowns were categorized into game-related and device-related breakdowns during the analysis process. The examined game-related breakdowns were connected to unclear game mechanics and difficulties of avatar interactions, while the examined device-related breakdowns were connected to difficulties with the participants’ HMDs and controllers. Breakdowns related to the game were found to be mildly disruptive to an ongoing interaction, with the participants managing to overcome them either by themselves or with the help of other users within VR. Breakdowns related to the VR devices proved more difficult for the participants to resolve, often requiring assistance not from other VR users, but from the observers of the recording sessions. Device-related issues could also be far more disruptive for the participants, leading to removing a user from the game entirely or forcing a premature end to a recording session at their most severe. A lack of information available to the participants was a recurring factor across both game-related and device-related breakdowns. Possible solutions to providing users with more information regarding both game mechanics and the status of their HMD and controllers were discussed.Tiivistelmä. Tämä tutkimus tarkastelee käyttöhäiriöitä ongelmalähteinä yhteisläsnäolevissa vuorovaikutustilanteissa immersiivisessä virtuaalitodellisuudessa. Tutkimuksessa tarkasteltu aineisto koostuu videonauhoituksista käyttäjistä pelaamassa Rec Room -verkkomoninpeliä HTC Vive virtuaalitodellisuuslaseja käyttäen. Nauhoitussessioihin osallistuneet käyttäjät olivat pääsääntöisesti ei-natiiveja englannin puhujia ilman aiempaa kokemusta immersiivisestä virtuaalitodellisuudesta. Osallistujat pelasivat peliä kuudessa kahden hengen ryhmässä. Osallistujia ohjeistettiin mahdollisimman vähän luontaisten interaktioiden helpottamiseksi virtuaaliympäristössä. Aineistosta tunnistetut kiinnostavat sekvenssit litteroitiin analyysiä varten. Tutkimusmetodina sovellettiin konversaatioanalyysiä, hyödyntäen myös aiempia tutkimuksia liittyen käyttöhäiriöihin ihmisen ja tietokoneen vuorovaikutuksessa sekä korjausilmiöihin vuorovaikutuksessa. Käyttöhäiriöt luokiteltiin peliin liittyviin ja laitteisiin littyviin käyttöhäiriöihin analyysiprosessin aikana. Tarkastellut peliin liittyvät käyttähäiriöt liittyivät epäselviin pelimekaniikkoihin sekä hankaluuksiin roolihahmojen vuorovaikutuksessa, kun taas tarkastellut laitteisiin liittyvät käyttöhäiriöt liittyivät hankaluuksiin osallistujien virtuaalilasien ja peliohjaimien kanssa. Peliin liittyneet käyttöhäiriöt häiritsivät vuorovaikutustilanteita lievästi, ja osallistujat pystyivät selviytymään tilanteesta joko yksin tai muiden virtuaalitodellisuudessa olevien käyttäjien avulla. Käyttöhäiriöt jotka liittyivät virtuaalitodellisuuslaitteisiin osoittautuivat osallistujille haastavammiksi selvittää, ja he tarvitsivat usein apua nauhoitussessioiden tarkkailijoilta muiden virtuaalitodellisuudessa olevien käyttäjien sijaan. Laitteisiin liittyvät ongelmat osoittautuivat myös huomattavasti häiritsevämmiksi osallistujille, vakavimmissa tapauksissa johtaen käyttäjän poistumiseen pelistä kokonaan tai nauhoitussession ennenaikaiseen loppuun. Osallistujille tarjolla olleen tiedon puute oli toistuva tekijä sekä peliin että laitteisiin liittyvissä käyttöhäiriöissä. Mahdollisia ratkaisuja pelimekaniikkoihin ja käyttäjien virtuaalilasien sekä peliohjaimien tilaan liittyvän tiedon tarjoamiselle tarkasteltiin

Making sense of digitally remediated touch in virtual reality experiences

Touch, often called the ‘first sense’, is fundamental to how we experience and know ourselves, others and the world. Increasingly, touch is being brought into the digital landscape. This paper explores this shifting landscape to understand the ways in which touch is re-mediated in the context of virtual reality. With attention to the sensoriality and sociality of touch, it asks what ‘counts’ as touch in VR, how is touch experienced and how is it incorporated into meaning making. We present and discuss findings from a multimodal and multisensory study of 16 participants interacting in two VR experiences to describe: the participants’ material encounters with the virtual through a focus on touch practices, expectations and norms; the ways in which participants made meaning of (and with) virtual touch through their dynamic selection and orchestration of the range of semiotic and experiential resources available; and how these virtual touch experiences translated into discourses of touch in VR to emphasize continuities and change between the past, present and futures. The paper comments on the methodological challenges of researching touch in the emergent landscape of VR and asks how multimodality might engage newly with touch, perhaps the most under-rated and neglected of modes and senses, and its digital remediation

In and out of control: Learning games differently

In this paper we make use of the theoretical resources of actor network theory as a ‘frame’ within which to organize video data we have been collecting on playing, and more specifically, on girls learning to play, digital games. Through a microanalysis of interaction, we closely examine intersecting trajectories of control -- self, other, and technology -- within the context of game play. Using MAP, a software program that supports multimodal analysis, we offer an illustrated account of the microgenesis of competence in collaborative, technologically-supported gameplay, drawing attention to developmentally significant behavioural regularities which, because they are embodied and not necessarily cognitive-linguistic in character, have not typically been evidenced in research on collaborative learning. A particular contribution of this paper is its study of group play, a relatively under-studied topic in gameplay research, and a perspective that has allowed us to look specifically at the phenomenon of the distributed development of competence central to learning in and through collaborative play

Computer Aided Drafting Virtual Reality Interface

Computer Aided Drafting (CAD) is pervasive in engineering fields today. It has become indispensable for planning, creating, visualizing, troubleshooting, collaborating, and communicating designs before they exist in physical form. From the beginning, CAD was created to be used by means of a mouse, keyboard, and monitor. Along the way, other, more specialized interface devices were created specifically for CAD that allowed for easier and more intuitive navigation within a 3D space, but they were at best stopgap solutions. Virtual Reality (VR) allows users to navigate and interact with digital 3D objects and environments the same way they would in the real world. For this reason, VR is a natural CAD interface solution. Using VR as an interface for CAD software, creating will be more intuitive and visualizing will be second nature. For this project, a prototype VR CAD program was created using Unreal Engine for use with the HTC Vive to compare against traditional WIMP (windows, icons, menus, pointer) interface CAD programs for the time it takes to learn each program, create similar models, and impressions of using each program, specifically the intuitiveness of the user interface and model manipulation. FreeCAD, SolidWorks, and Blender were the three traditional interface modeling programs chosen to compare against VR because of their wide-spread use for modeling in 3D printing, industry, and gaming, respectively. During the course of the project, two VR modeling programs were released, Google Blocks and MakeVR Pro; because they were of a similar type as the prototype software created in Unreal Engine, they were included for comparison as part of this project. The comparison showed that the VR CAD programs were faster to learn and create models and more intuitive to use than the traditional interface CAD programs

Understanding user interactions in stereoscopic head-mounted displays

2022 Spring.Includes bibliographical references.Interacting in stereoscopic head mounted displays can be difficult. There are not yet clear standards for how interactions in these environments should be performed. In virtual reality there are a number of well designed interaction techniques; however, augmented reality interaction techniques still need to be improved before they can be easily used. This dissertation covers work done towards understanding how users navigate and interact with virtual environments that are displayed in stereoscopic head-mounted displays. With this understanding, existing techniques from virtual reality devices can be transferred to augmented reality where appropriate, and where that is not the case, new interaction techniques can be developed. This work begins by observing how participants interact with virtual content using gesture alone, speech alone, and the combination of gesture+speech during a basic object manipulation task in augmented reality. Later, a complex 3-dimensional data-exploration environment is developed and refined. That environment is capable of being used in both augmented reality (AR) and virtual reality (VR), either asynchronously or simultaneously. The process of iteratively designing that system and the design choices made during its implementation are provided for future researchers working on complex systems. This dissertation concludes with a comparison of user interactions and navigation in that complex environment when using either an augmented or virtual reality display. That comparison contributes new knowledge on how people perform object manipulations between the two devices. When viewing 3D visualizations, users will need to feel able to navigate the environment. Without careful attention to proper interaction technique design, people may struggle to use the developed system. These struggles may range from a system that is uncomfortable and not fit for long-term use, or they could be as major as causing new users to not being able to interact in these environments at all. Getting the interactions right for AR and VR environments is a step towards facilitating their widespread acceptance. This dissertation provides the groundwork needed to start designing interaction techniques around how people utilize their personal space, virtual space, body, tools, and feedback systems

Exploitation of multiplayer interaction and development of virtual puppetry storytelling using gesture control and stereoscopic devices

With the rapid development of human-computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head-mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head-mounted display outperformed the traditional monoscopic display in a comparison between the two