Teaching Social Virtual Reality With Ubiq

We share our experiences of teaching virtual reality with Ubiq, an open-source system for building social virtual reality (VR). VR as a subject touches on many areas, including perception, human–computer interaction, and psychology. In our VE module, we consider all aspects of VR. In recent years, networked VR, and in particular social VR, has become increasingly relevant, at the same time as demand for online and hybrid teaching has increased. Commercial social virtual reality systems have proliferated, but for a number of reasons, this has not resulted in systems any more suitable for research and teaching. As a result we created Ubiq, a system for building social VR applications designed first for research and teaching. In this article, we describe how Ubiq came to be, and our experiences of using it in our virtual environments module over the last two years

Some Lessons Learned Running Virtual Reality Experiments Out of the Laboratory

In the past twelve months, our team has had to move rapidly from conducting most of our user experiments in a laboratory setting, to running experiments in the wild away from the laboratory and without direct synchronous oversight from an experimenter. This has challenged us to think about what types of experiment we can run, and to improve our tools and methods to allow us to reliably capture the necessary data. It has also offered us an opportunity to engage with a more diverse population than we would normally engage with in the laboratory. In this position paper we elaborate on the challenges and opportunities, and give some lessons learned from our own experience

Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments

Developing mixed-reality (MR) experiments is a challenge as there is a wide variety of functionality to support. This challenge is exacerbated if the MR experiment is multi-user or if the experiment needs to be run out of the lab. We present Ubiq-Exp - a set of tools that provide a variety of functionality to facilitate distributed and remote MR experiments. We motivate our design and tools from recent practice in the field and a desire to build experiments that are easier to reproduce. Key features are the ability to support supervised and unsupervised experiments, and a variety of tools for the experimenter to facilitate operation and documentation of the experimental sessions. We illustrate the potential of the tools through three small-scale pilot experiments. Our tools and pilot experiments are released under a permissive open-source license to enable developers to appropriate and develop them further for their own needs

Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made both neurobiologically more plausible and computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, for example, fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of recurrent neural networks may be useful both as a model of brain function and as a machine learning tool. We illustrate the use of the rRNN by an application to the online decoding (i.e. recognition) of human kinematics

Position-Based Control of Under-Constrained Haptics: A System for the Dexmo Glove

The Dexmo glove is a haptic exoskeleton that provides kinesthetic feedback in virtual reality. Unlike many other gloves based on string–pulleys, the Dexmo uses a free-hinged link-bar to transfer forces from a crank to the fingertips. It also uses an admittance-based controller parameterized by position, as opposed to an impedance-based controller parameterized by force. When setting the controller’s target position, developers must use its native angular coordinate system. The Dexmo has a number of uninstrumented degrees of freedom. Mature forward models can reliably predict the hand pose, even with these unknowns. When it comes to computing angular controller parameters from a target pose in Cartesian space however, things become more difficult. Complex models that provide attractive visuals from a small number of sensors can be non-trivial or even impossible to invert. In this letter, we suggest side-stepping this issue. We sample the forward model in order to build a lookup table. This is embedded in three-dimensional space as a curve, on which traditional queries against world geometry can be performed. Controller parameters are stored as attributes of the sample points. To compute the driver parameters for a target position, the application constrains the position to the geometry, and interpolates them. This technique is generalizable, stable, simple, and fast. We validate our approach by implementing it in Unity 2017.3 and integrating it with a Dexmo glove

Extending the Open Source Social Virtual Reality Ecosystem to the Browser in Ubiq

Social VR (SVR) systems are VR systems with a common subset of features facilitating unstructured social interaction. In the real world, social situations have many purposes, each with a different set of requirements, and roles its participants take - creator, moderator, performer, visitor, etc. Yet, common SVR systems typically offer only a single client to users. Even if there are versions for different platforms, there is a one-size-fits-all approach to the user experience. Consequently users need to employ workarounds or build their own functionality to support specific roles, where this is possible at all. We argue that platforms need to develop more open frameworks that support different processes and user interactions. One way to do this is through using appropriate web standards and an open messaging system in order to allow distributed clients that can leverage the strongest features of heterogeneous computing platforms. Supporting asymmetrical capabilities greatly increases the scope of supported virtual social interactions and potential use cases of SVR. We take a qualitative experimental approach to exploring cross platform support in this way, from a designers perspective. We use the open-source SDK Ubiq, and create a library that allows building Ubiq Peers using web standards and thus clients that can operate solely in a web browser or certain Javascript environments. We validate our approach by demonstrating six proof of concept demonstrators that would be difficult or impossible to achieve in most other SVR systems, and report on what we encountered for the benefit of other SVR designers

Quality of Service Impact on Edge Physics Simulations for VR

—Mobile HMDs must sacrifice compute performance to achieve ergonomic and power requirements for extended use. Consequently, applications must either reduce rendering and simulation complexity - along with the richness of the experience - or offload complexity to a server. Within the context of edge-computing, a popular way to do this is through render streaming. Render streaming has been demonstrated for desktops and consoles. It has also been explored for HMDs. However, the latency requirements of head tracking make this application much more challenging. While mobile GPUs are not yet as capable as their desktop counterparts, we note that they are becoming more powerful and efficient. With the hard requirements of VR, it is worth continuing to investigate what schemes could optimally balance load, latency and quality. We propose an alternative we call edge-physics: streaming at the scene-graph level from a simulation running on edge-resources, analogous to cluster rendering. Scene streaming is not only straightforward, but compute and bandwidth efficient. The most demanding loops run locally. Jobs that hit the power-wall of mobile CPUs are off-loaded, while improving GPUs are leveraged, maximising compute utilisation. In this paper we create a prototypical implementation and evaluate its potential in terms of fidelity, bandwidth and performance. We show that an effective system which maintains high consistencies on typical edge-links can be easily built, but that some traditional concepts are not applicable, and a better understanding of the perception of motion is required to evaluate such a system comprehensively

Consensus Based Networking of Distributed Virtual Environments.

Distributed Virtual Environments (DVEs) are challenging to create as the goals of consistency and responsiveness become contradictory under increasing latency. DVEs have been considered as both distributed transactional databases and force-reflection systems. Both are good approaches, but they do have drawbacks. Transactional systems do not support Level 3 (L3) collaboration: manipulating the same degree-of-freedom at the same time. Force-reflection requires a client-server architecture and stabilisation techniques. With Consensus Based Networking (CBN), we suggest DVEs be considered as a distributed data-fusion problem. Many simulations run in parallel and exchange their states, with remote states integrated with continous authority. Over time the exchanges average out local differences, performing a distribued-average of a consistent, shared state. CBN aims to build simulations that are highly responsive, but consistent enough for use cases such as the piano-movers problem. CBN's support for heterogeneous nodes can transparently couple different input methods, avoid the requirement of determinism, and provide more options for personal control over the shared experience. Our work is early, however we demonstrate many successes, including L3 collaboration in room-scale VR, 1000's of interacting objects, complex configurations such as stacking, and transparent coupling of haptic devices. These have been shown before, but each with a different technique; CBN supports them all within a single, unified system

Reality of auditory verbal hallucinations

Distortion of the sense of reality, actualized in delusions and hallucinations, is the key feature of psychosis but the underlying neuronal correlates remain largely unknown. We studied 11 highly functioning subjects with schizophrenia or schizoaffective disorder while they rated the reality of auditory verbal hallucinations (AVH) during functional magnetic resonance imaging (fMRI). The subjective reality of AVH correlated strongly and specifically with the hallucination-related activation strength of the inferior frontal gyri (IFG), including the Broca's language region. Furthermore, how real the hallucination that subjects experienced was depended on the hallucination-related coupling between the IFG, the ventral striatum, the auditory cortex, the right posterior temporal lobe, and the cingulate cortex. Our findings suggest that the subjective reality of AVH is related to motor mechanisms of speech comprehension, with contributions from sensory and salience-detection-related brain regions as well as circuitries related to self-monitoring and the experience of agency

The relationship between puberty and social emotion processing

The social brain undergoes developmental change during adolescence, and pubertal hormones are hypothesized to contribute to this development. We used fMRI to explore how pubertal indicators (salivary concentrations of testosterone, oestradiol and DHEA; pubertal stage; menarcheal status) relate to brain activity during a social emotion task. Forty-two females aged 11.1 to 13.7 years underwent fMRI scanning while reading scenarios pertaining either to social emotions, which require the representation of another person’s mental states, or to basic emotions, which do not. Pubertal stage and menarcheal status were used to assign girls to early or late puberty groups. Across the entire sample, the contrast between social versus basic emotion resulted in activity within the social brain network, including dorsomedial prefrontal cortex (DMPFC), the posterior superior temporal sulcus, and the anterior temporal cortex (ATC) in both hemispheres. Increased hormone levels (independent of age) were associated with higher left ATC activity during social emotion processing. More advanced age (independent of hormone levels) was associated with lower DMPFC activity during social emotion processing. Our results suggest functionally dissociable effects of pubertal hormones and age on the adolescent social brain