Knowledge-driven Biometric Authentication in Virtual Reality

With the increasing adoption of virtual reality (VR) in public spaces, protecting users from observation attacks is becoming essential to prevent attackers from accessing context-sensitive data or performing malicious payment transactions in VR. In this work, we propose RubikBiom, a knowledge-driven behavioural biometric authentication scheme for authentication in VR. We show that hand movement patterns performed during interactions with a knowledge-based authentication scheme (e.g., when entering a PIN) can be leveraged to establish an additional security layer. Based on a dataset gathered in a lab study with 23 participants, we show that knowledge-driven behavioural biometric authentication increases security in an unobtrusive way. We achieve an accuracy of up to 98.91% by applying a Fully Convolutional Network (FCN) on 32 authentications per subject. Our results pave the way for further investigations towards knowledge-driven behavioural biometric authentication in VR

Virtual reality interfaces for seamless interaction with the physical reality

In recent years head-mounted displays (HMDs) for virtual reality (VR) have made the transition from research to consumer product, and are increasingly used for productive purposes such as 3D modeling in the automotive industry and teleconferencing. VR allows users to create and experience real-world like models of products; and enables users to have an immersive social interaction with distant colleagues. These solutions are a promising alternative to physical prototypes and meetings, as they require less investment in time and material. VR uses our visual dominance to deliver these experiences, making users believe that they are in another reality. However, while their mind is present in VR their body is in the physical reality. From the user’s perspective, this brings considerable uncertainty to the interaction. Currently, they are forced to take off their HMD in order to, for example, see who is observing them and to understand whether their physical integrity is at risk. This disrupts their interaction in VR, leading to a loss of presence – a main quality measure for the success of VR experiences. In this thesis, I address this uncertainty by developing interfaces that enable users to stay in VR while supporting their awareness of the physical reality. They maintain this awareness without having to take off the headset – which I refer to as seamless interaction with the physical reality. The overarching research vision that guides this thesis is, therefore, to reduce this disconnect between the virtual and physical reality. My research is motivated by a preliminary exploration of user uncertainty towards using VR in co-located, public places. This exploration revealed three main foci: (a) security and privacy, (b) communication with physical collaborators, and (c) managing presence in both the physical and virtual reality. Each theme represents a section in my dissertation, in which I identify central challenges and give directions towards overcoming them as have emerged from the work presented here. First, I investigate security and privacy in co-located situations by revealing to what extent bystanders are able to observe general tasks. In this context, I explicitly investigate the security considerations of authentication mechanisms. I review how existing authentication mechanisms can be transferred to VR and present novel approaches that are more usable and secure than existing solutions from prior work. Second, to support communication between VR users and physical collaborators, I add to the field design implications for VR interactions that enable observers to choose opportune moments to interrupt HMD users. Moreover, I contribute methods for displaying interruptions in VR and discuss their effect on presence and performance. I also found that different virtual presentations of co-located collaborators have an effect on social presence, performance and trust. Third, I close my thesis by investigating methods to manage presence in both the physical and virtual realities. I propose systems and interfaces for transitioning between them that empower users to decide how much they want to be aware of the other reality. Finally, I discuss the opportunity to systematically allocate senses to these two realities: the visual one for VR and the auditory and haptic one for the physical reality. Moreover, I provide specific design guidelines on how to use these findings to alert VR users about physical borders and obstacles.In den letzten Jahren haben Head-Mounted-Displays (HMDs) für virtuelle Realität (VR) den Übergang von der Forschung zum Konsumprodukt vollzogen und werden zunehmend für produktive Zwecke, wie 3D-Modellierung in der Automobilindustrie oder Telekonferenzen, eingesetzt. VR ermöglicht es den Benutzern, schnell und kostengünstig, Prototypen zu erstellen und erlaubt eine immersive soziale Interaktion mit entfernten Kollegen. VR nutzt unsere visuelle Dominanz, um diese Erfahrungen zu vermitteln und gibt Benutzern das Gefühl sich in einer anderen Realität zu befinden. Während der Nutzer jedoch in der virtuellen Realität mental präsent ist, befindet sich der Körper weiterhin in der physischen Realität. Aus der Perspektive des Benutzers bringt dies erhebliche Unsicherheit in die Nutzung von HMDs. Aktuell sind Nutzer gezwungen, ihr HMD abzunehmen, um zu sehen, wer sie beobachtet und zu verstehen, ob ihr körperliches Wohlbefinden gefährdet ist. Dadurch wird ihre Interaktion in der VR gestört, was zu einem Verlust der Präsenz führt - ein Hauptqualitätsmaß für den Erfolg von VR-Erfahrungen. In dieser Arbeit befasse ich mich mit dieser Unsicherheit, indem ich Schnittstellen entwickle, die es den Nutzern ermöglichen, in VR zu bleiben und gleichzeitig unterstützen sie die Wahrnehmung für die physische Realität. Sie behalten diese Wahrnehmung für die physische Realität bei, ohne das Headset abnehmen zu müssen - was ich als nahtlose Interaktion mit der physischen Realität bezeichne. Daher ist eine übergeordenete Vision von meiner Forschung diese Trennung von virtueller und physicher Realität zu reduzieren. Meine Forschung basiert auf einer einleitenden Untersuchung, die sich mit der Unsicherheit der Nutzer gegenüber der Verwendung von VR an öffentlichen, geteilten Orten befasst. Im Kontext meiner Arbeit werden Räume oder Flächen, die mit anderen ortsgleichen Menschen geteilt werden, als geteilte Orte bezeichnet. Diese Untersuchung ergab drei Hauptschwerpunkte: (1) Sicherheit und Privatsphäre, (2) Kommunikation mit physischen Kollaborateuren, und (3) Umgang mit der Präsenz, sowohl in der physischen als auch in der virtuellen Realität. Jedes Thema stellt einen Fokus in meiner Dissertation dar, in dem ich zentrale Herausforderungen identifiziere und Lösungsansätze vorstelle. Erstens, untersuche ich Sicherheit und Privatsphäre an öffentlichen, geteilten Orten, indem ich aufdecke, inwieweit Umstehende in der Lage sind, allgemeine Aufgaben zu beobachten. In diesem Zusammenhang untersuche ich explizit die Gestaltung von Authentifizierungsmechanismen. Ich untersuche, wie bestehende Authentifizierungsmechanismen auf VR übertragen werden können, und stelle neue Ansätze vor, die nutzbar und sicher sind. Zweitens, um die Kommunikation zwischen HMD-Nutzern und Umstehenden zu unterstützen, erweitere ich das Forschungsfeld um VR-Interaktionen, die es Beobachtern ermöglichen, günstige Momente für die Unterbrechung von HMD-Nutzern zu wählen. Darüber hinaus steuere ich Methoden zur Darstellung von Unterbrechungen in VR bei und diskutiere ihre Auswirkungen auf Präsenz und Leistung von Nutzern. Meine Arbeit brachte auch hervor, dass verschiedene virtuelle Präsentationen von ortsgleichen Kollaborateuren einen Effekt auf die soziale Präsenz, Leistung und Vertrauen haben. Drittens, schließe ich meine Dissertation mit der Untersuchung von Methoden zur Verwaltung der Präsenz, sowohl in der physischen als auch in der virtuellen Realität ab. Ich schlage Systeme und Schnittstellen für den Übergang zwischen den Realitäten vor, die die Benutzer in die Lage versetzen zu entscheiden, inwieweit sie sich der anderen Realität bewusst sein wollen. Schließlich diskutiere ich die Möglichkeit, diesen beiden Realitäten systematisch Sinne zuzuordnen: die visuelle für VR und die auditive und haptische für die physische Realität. Darüber hinaus stelle ich spezifische Design-Richtlinien zur Verfügung, wie diese Erkenntnisse genutzt werden können, um VR-Anwender auf physische Grenzen und Hindernisse aufmerksam zu machen

Surveying Consumer Understanding & Sentiment Of VR

Since the resurgence of consumer-grade virtual reality (VR), VR has successfully established itself on the consumer market. As with any emerging technology, differences can exist between how industry / academia view the technology and how consumers perceive it. We present results from a survey (N=210) conducted into consumer perception and attitudes towards VR. We report sentiment towards VR is positive. We show the associations linked with VR by our respondents match the defining characteristics of VR identified by experts in the literature (a fully virtual view, immersion, and head-worn technology). We identify 3 additional concepts associated with VR by our respondents: video games, futurism, and price. However, our results also show consumer expectations for VR are fixated around "VR for gaming" and suggest VR has to an extent been pigeonholed as primarily being a gaming device

Bystander Interruption of VR Users

Research has begun to investigate the interruption of VR users but lacks an understanding of how social factors (setting / relationship to the VR user) might impact the interruption and why bystanders interrupt as they do. We conducted a survey (N=100) into bystander comfort when interrupting a VR user (known / unknown) in 4 settings (private spaces, public spaces, private transport, public transport) and their willingness to use a range of interruption strategies. Our results suggest relationship to the VR user is more influential than setting when considering comfort and acceptability of interruption strategy. A follow-up lab study (N=16) investigated bystander interruption of a known VR user in a private setting. Most used a combination of speech and touch to interrupt though a subset used unconventional, improvised strategies we term playful interruptions. With the recent inclusion of open ear audio in VR headsets and preference towards verbal interruptions in our results we conducted a survey (N=76) into the consumer response to open ear audio in VR headsets. We report open ear audio was used regularly by 52.6% of our respondents 60% of who rate their experience with it positively

GazeRoomLock: Using Gaze and Head-Pose to Improve the Usability and Observation Resistance of 3D Passwords in Virtual Reality

Authentication has become an important component of Immersive Virtual Reality (IVR) applications, such as virtual shopping stores, social networks, and games. Recent work showed that compared to traditional graphical and alphanumeric passwords, a more promising form of passwords for IVR is 3D passwords. This work evaluates four multimodal techniques for entering 3D passwords in IVR that consist of multiple virtual objects selected in succession. Namely, we compare eye gaze and head pose for pointing, and dwell time and tactile input for selection. A comparison of a) usability in terms of entry time, error rate, and memorability, and b) resistance to real world and offline observations, reveals that: multimodal authentication in IVR by pointing at targets using gaze, and selecting them using a handheld controller significantly improves usability and security compared to the other methods and to prior work. We discuss how the choice of pointing and selection methods impacts the usability and security of 3D passwords in IVR

Virtual reality interfaces for seamless interaction with the physical reality

In recent years head-mounted displays (HMDs) for virtual reality (VR) have made the transition from research to consumer product, and are increasingly used for productive purposes such as 3D modeling in the automotive industry and teleconferencing. VR allows users to create and experience real-world like models of products; and enables users to have an immersive social interaction with distant colleagues. These solutions are a promising alternative to physical prototypes and meetings, as they require less investment in time and material. VR uses our visual dominance to deliver these experiences, making users believe that they are in another reality. However, while their mind is present in VR their body is in the physical reality. From the user’s perspective, this brings considerable uncertainty to the interaction. Currently, they are forced to take off their HMD in order to, for example, see who is observing them and to understand whether their physical integrity is at risk. This disrupts their interaction in VR, leading to a loss of presence – a main quality measure for the success of VR experiences. In this thesis, I address this uncertainty by developing interfaces that enable users to stay in VR while supporting their awareness of the physical reality. They maintain this awareness without having to take off the headset – which I refer to as seamless interaction with the physical reality. The overarching research vision that guides this thesis is, therefore, to reduce this disconnect between the virtual and physical reality. My research is motivated by a preliminary exploration of user uncertainty towards using VR in co-located, public places. This exploration revealed three main foci: (a) security and privacy, (b) communication with physical collaborators, and (c) managing presence in both the physical and virtual reality. Each theme represents a section in my dissertation, in which I identify central challenges and give directions towards overcoming them as have emerged from the work presented here. First, I investigate security and privacy in co-located situations by revealing to what extent bystanders are able to observe general tasks. In this context, I explicitly investigate the security considerations of authentication mechanisms. I review how existing authentication mechanisms can be transferred to VR and present novel approaches that are more usable and secure than existing solutions from prior work. Second, to support communication between VR users and physical collaborators, I add to the field design implications for VR interactions that enable observers to choose opportune moments to interrupt HMD users. Moreover, I contribute methods for displaying interruptions in VR and discuss their effect on presence and performance. I also found that different virtual presentations of co-located collaborators have an effect on social presence, performance and trust. Third, I close my thesis by investigating methods to manage presence in both the physical and virtual realities. I propose systems and interfaces for transitioning between them that empower users to decide how much they want to be aware of the other reality. Finally, I discuss the opportunity to systematically allocate senses to these two realities: the visual one for VR and the auditory and haptic one for the physical reality. Moreover, I provide specific design guidelines on how to use these findings to alert VR users about physical borders and obstacles.In den letzten Jahren haben Head-Mounted-Displays (HMDs) für virtuelle Realität (VR) den Übergang von der Forschung zum Konsumprodukt vollzogen und werden zunehmend für produktive Zwecke, wie 3D-Modellierung in der Automobilindustrie oder Telekonferenzen, eingesetzt. VR ermöglicht es den Benutzern, schnell und kostengünstig, Prototypen zu erstellen und erlaubt eine immersive soziale Interaktion mit entfernten Kollegen. VR nutzt unsere visuelle Dominanz, um diese Erfahrungen zu vermitteln und gibt Benutzern das Gefühl sich in einer anderen Realität zu befinden. Während der Nutzer jedoch in der virtuellen Realität mental präsent ist, befindet sich der Körper weiterhin in der physischen Realität. Aus der Perspektive des Benutzers bringt dies erhebliche Unsicherheit in die Nutzung von HMDs. Aktuell sind Nutzer gezwungen, ihr HMD abzunehmen, um zu sehen, wer sie beobachtet und zu verstehen, ob ihr körperliches Wohlbefinden gefährdet ist. Dadurch wird ihre Interaktion in der VR gestört, was zu einem Verlust der Präsenz führt - ein Hauptqualitätsmaß für den Erfolg von VR-Erfahrungen. In dieser Arbeit befasse ich mich mit dieser Unsicherheit, indem ich Schnittstellen entwickle, die es den Nutzern ermöglichen, in VR zu bleiben und gleichzeitig unterstützen sie die Wahrnehmung für die physische Realität. Sie behalten diese Wahrnehmung für die physische Realität bei, ohne das Headset abnehmen zu müssen - was ich als nahtlose Interaktion mit der physischen Realität bezeichne. Daher ist eine übergeordenete Vision von meiner Forschung diese Trennung von virtueller und physicher Realität zu reduzieren. Meine Forschung basiert auf einer einleitenden Untersuchung, die sich mit der Unsicherheit der Nutzer gegenüber der Verwendung von VR an öffentlichen, geteilten Orten befasst. Im Kontext meiner Arbeit werden Räume oder Flächen, die mit anderen ortsgleichen Menschen geteilt werden, als geteilte Orte bezeichnet. Diese Untersuchung ergab drei Hauptschwerpunkte: (1) Sicherheit und Privatsphäre, (2) Kommunikation mit physischen Kollaborateuren, und (3) Umgang mit der Präsenz, sowohl in der physischen als auch in der virtuellen Realität. Jedes Thema stellt einen Fokus in meiner Dissertation dar, in dem ich zentrale Herausforderungen identifiziere und Lösungsansätze vorstelle. Erstens, untersuche ich Sicherheit und Privatsphäre an öffentlichen, geteilten Orten, indem ich aufdecke, inwieweit Umstehende in der Lage sind, allgemeine Aufgaben zu beobachten. In diesem Zusammenhang untersuche ich explizit die Gestaltung von Authentifizierungsmechanismen. Ich untersuche, wie bestehende Authentifizierungsmechanismen auf VR übertragen werden können, und stelle neue Ansätze vor, die nutzbar und sicher sind. Zweitens, um die Kommunikation zwischen HMD-Nutzern und Umstehenden zu unterstützen, erweitere ich das Forschungsfeld um VR-Interaktionen, die es Beobachtern ermöglichen, günstige Momente für die Unterbrechung von HMD-Nutzern zu wählen. Darüber hinaus steuere ich Methoden zur Darstellung von Unterbrechungen in VR bei und diskutiere ihre Auswirkungen auf Präsenz und Leistung von Nutzern. Meine Arbeit brachte auch hervor, dass verschiedene virtuelle Präsentationen von ortsgleichen Kollaborateuren einen Effekt auf die soziale Präsenz, Leistung und Vertrauen haben. Drittens, schließe ich meine Dissertation mit der Untersuchung von Methoden zur Verwaltung der Präsenz, sowohl in der physischen als auch in der virtuellen Realität ab. Ich schlage Systeme und Schnittstellen für den Übergang zwischen den Realitäten vor, die die Benutzer in die Lage versetzen zu entscheiden, inwieweit sie sich der anderen Realität bewusst sein wollen. Schließlich diskutiere ich die Möglichkeit, diesen beiden Realitäten systematisch Sinne zuzuordnen: die visuelle für VR und die auditive und haptische für die physische Realität. Darüber hinaus stelle ich spezifische Design-Richtlinien zur Verfügung, wie diese Erkenntnisse genutzt werden können, um VR-Anwender auf physische Grenzen und Hindernisse aufmerksam zu machen

Investigation of police decision making using combined EEG and virtual reality methods

Police officers in the UK are granted additional powers to allow them to protect life and property crime. Of these powers, the sanction to use stopping, potentially lethal, force given to Authorised Firearms Officers (AFO) is arguably the most salient. Each decision made by an AFO to discharge their firearm or not has great impact and so it is important we research the cognitive processes that lead to such a decision.One challenge of researching these cognitive processes is eliciting ecologically valid behaviour while maintaining internal validity. We approached this challenge by developing combined electroencephalography (EEG) and virtual reality research methods. Using these methods, we produced scenarios that reflected features of AFO training. First, we tested simple versions of the scenarios on a novice population. Following this, we increased the complexity of the scenarios and collected data from both AFOs and novices.We found that participants were fastest when responding to threatening scenarios. Further, AFOs had consistently faster response times than novices. In line with similar ‘Go/No-Go’ paradigms, we found greater increases in pre-response frontal-midline theta when participants did not shoot versus when they did. Comparisons of EEG between AFOs and novices revealed greater pre-response increases in frontal-midline theta and central delta when they equipped a firearm. Greater differences in delta activity were also observed between different levels of threat in the AFO group.Together, these findings suggest that differences in performance between experts and novices may be due to their greater attention towards threat. Further investigation of expert decision making should build on our use of naturalistic stimuli and expert participants to ensure that findings are ecologically valid.With increasing accessibility of modern game engines and virtual reality technology, this approach will be beneficial to researchers in many fields where ecological validity is required

Don’t interrupt me while I’m speaking: Interruption in Everyday and Institutional Settings in Chinese

Interruption is a common phenomenon in conversation. Previous research of interruption has focused on three main aspects: the identification of interruption in relation to overlaps or overlapping speech, the categorisation of cooperative and disruptive interruptions, and the relationship between interruption and certain social factors, for instance, power asymmetry and gender differences. However, little attention has been paid to the degree of intrusiveness. Likewise, not much has been done to explore interactional factors that may intersect with interruptions. With these important research gaps in mind, I aim to explore the relationship between intrusiveness and interactional dimensions of interruptions in the Chinese context in this study. Two sets of conversational data were collected: telephone conversations and TV talk show conversations. The conversation analytic method was used to examine the fine-grained details of speakers’ conversational interaction (Haugh, 2012). Statistical methods were used to test the relationship between factors related to interruptions. Results from a linear regression model indicate that, in both settings, speakers tend to heed and boost the current information flow (e.g., supplementing further details) when expressing affiliative stances. More specifically, in the institutional conversation, speakers orient their interruption utterances towards the their assigned institutional role and task (Goffman, 1981; Heritage & Greatbatch, 1991). In the telephone conversation, there are frequent early interruptions, affiliative interruptions, and unexpected cases where interrupters align their opinions with the other whilst disrupting the current information flow. Based on what emerged from these analyses, I argue that the Chinese speakers in the two corpora feature a high involvement (Tannen, 2005) conversational style, which means they prioritise relationship over the task in discussion. In other words, speakers tend to distinctively emphasise their enthusiasm and engagement with the other speaker, but pay less attention to the one-speaker-at-a-time turn-taking rule (Sacks, Schegloff, & Jefferson, 1974). The finding of relationship-focus of Chinese talk-in-interaction supports the argument that Chinese society largely adheres to the polychronic time orientation (Hall, 1984). This study contributes to CA methodology by combining rigorous quantification methods with close examination of sequential organisation of interruptions. It is innovative in measuring intrusiveness by incorporating two aspects of interruptions: the interrupter’s stance-taking and the interrupter’s sequential alignment with the information flow of the prior utterance. In so doing, this study contributes to the understanding of interruption by demonstrating that intrusiveness is a gradient concept on a measurable continuum rather than a binary concept that is either cooperative or intrusive. This study contributes to the investigation into Chinese talk-in-interaction, particularly speakers’ conversational style, by proposing a novel perspective: interruption. Keywords: Interruption, intrusiveness, affiliation, information flow, interruption marker, interruption timing, Chinese talk-in-interactio