Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing Neural Rendering

While head-mounted displays (HMDs) for Virtual Reality (VR) have become widely available in the consumer market, they pose a considerable obstacle for realistic face-to-face conversation in VR since HMDs hide a significant portion of the participants faces. Even with image streams from cameras directly attached to an HMD, stitching together a convincing image of an entire face remains a challenging task because of extreme capture angles and strong lens distortions due to a wide field of view. Compared to the long line of research in VR, reconstruction of faces hidden beneath an HMD is a very recent topic of research. While the current state-of-the-art solutions demonstrate photo-realistic 3D reconstruction results, many of them require high-cost laboratory equipment and large computational costs. We present an approach that focuses on low-cost hardware and can be used on a commodity gaming computer with a single GPU. We leverage the benefits of an end-to-end pipeline by means of Generative Adversarial Networks (GAN). Our GAN produces a frontal-facing 2.5D point cloud based on a training dataset captured with an RGBD camera. In our approach, the training process is offline, while the reconstruction runs in real-time. Our results show adequate reconstruction quality within the “learned” expressions. Expressions not learned by the network produce artifacts and can trigger the Uncanny Valley effect

A Perspective on Interface Techniques in Tangible Augmented Reality for Mobile Devices

One option for interactions in Augmented Reality (AR) is using physical objects, called tangibles. When virtual information augments tangibles, users can interact with the tangibles to manipulate the virtual content. This work introduces six interface techniques with different shaped tangibles for handheld devices in Tangible AR (TAR).We explore challenges and opportunities for the TAR-based interfaces and give practical insight into implementing them. In a user study, we show that users prefer performing touch interactions with user interface (UI) elements on a touchscreen over UI elements that are virtually projected onto a surface. In the latter case, users are challenged with hand coordination. Which tangible type they prefer depends on the application: For applications that focus on 3D interactions, users in our study prefer realistically shaped tangibles. For applications where users focus on the AR device’s screen, they preferred a generic tangible

Evaluating the Effects of Visual Fidelity and Magnified View on User Experience in Virtual Reality Games

Virtual reality has been becoming more affordable in recent years. This led to more content specifically developed for this medium. Training with virtual reality is one of the promising areas in terms of the benefits. Virtual reality properties may affect user performance. This study aims at exploring effects of visual fidelity (high and low) and view zoom (normal and magnified) on task performance in virtual reality. Effects of visual fidelity have previously been explored but yielded different results based on the task design. Effects of view zoom on task performance haven’t been explored yet. An inspection task in virtual reality was developed and a user study was performed with 15 participants. Results indicated that low visual fidelity led to better task performance whereas view zoom did not have an effect on the performance

A Sketch-based Interface for Real-time Control of Crowd Simulations that incorporate Dynamic Knowledge

Controlling crowd simulations typically involves tweaking complex parameter sets to attempt to reach a desired outcome, which can be unintuitive for non- technical users. This paper presents an approach to control pedestrian simulations in real time via sketching. Users are able to create entrances/exits, barriers to block paths, flow lines to guide pedestrians, waypoint areas, and storyboards to specify the journeys of crowd subgroups. Additionally, a timeline interface can be used to control when simulation events occur. The sketching approach is supported by a tiled navigation mesh (navmesh), based on the open source tool RE- CAST, to support pedestrian navigation. The navmesh is updated in real time based on the user’s sketches and the simulation updates accordingly. A comparison between our navmesh approach and the more often used grid-based navigation approach is given, showing that the navmesh approach scales better for large environments. The paper also presents possible solutions to address the question of when pedestrians should react to real-time changes to the environment, whether or not these changes are in their field of vision. The effectiveness of the system is demonstrated with a set of scenarios and a practical application which make use of a 3D model of an area of a UK city centre created using data from OPENSTREETMAP

Designing Mobile Multimodal Interaction for Visually Impaired and Older Adults: Challenges and Possible Solutions

This paper presents two early studies aimed at investigating issues concerning the design of multimodal interaction based on voice commands and one-hand mid-air gestures - with mobile technology specifically designed for visually impaired and elderly users. These studies were carried out on a new device allowing enhanced speech recognition (including lip movement analysis) and mid-air gesture interaction on Android operating system (smartphone and tablet PC). We discuss the initial findings and challenges raised by these novel interaction modalities, and in particular the issues regarding the design of feedback and feedforward, the problem of false positives, and the correct orientation and distance of the hand and the device during the interaction. Finally, we present a set of feedback and feedforward solutions designed to overcome the main issues highlighted

Position Estimation and Calibration of Inertial Motion Capture Systems Using Single Camera

The paper proposes a hybrid system for position estimation of a motion capture suit and gloves as well as a method for an automatic skeleton calibration for motion capture gloves. The skeleton calibration works with a single image scan of the hand where the skeleton is fitted. The position estimation is based on a synchronization of an inertial motion capture system and a single camera optical setup. The proposed synchronization uses an iterative optimization of an energy potential in image space, minimizing the error between the camera image and a rendered virtual representation of the scene. For each frame, an input skeleton pose from the mocap suit is used to render a silhouette of a subject. Moreover, the local neighborhood around the last known position is searched by matching the silhouette to the distance transform representation of the camera image based on Chamfer matching. Using the combination of the camera tracking and the inertial motion capture suit, it is possible to retrieve the position of the joints that are hidden from the camera view. Using the proposed hybrid technique, it is possible to capture the position even if it cannot be captured by the suit sensors. Our system can be used for both realtime tracking and off-line post-processing of already captured mocap data

Games as Blends: Understanding Hybrid Games

The meaning of what hybrid games are is often fixed to the context in which the term is used. For example, hybrid games have often been defined in relation to recent developments in technology. This creates issues in the terms usage and limitations in thinking. This paper argues that hybrid games should be understood through conceptual metaphors. Hybridity is the blending of different cognitive domains that are not usually associated together. Hybrid games usually blend domains related to games, for example digital and board games, but can blend also other domains. Through viewing game experiences as blends from different domains, designers can understand the inherent hybridity in various types of games and use that understanding when building new designs

3D reconstruction with a markerless tracking method of flexible and modular molecular physical models: towards tangible interfaces

Physical models have always been used in the field of molecular science as an understandable representation of complex molecules, particularly in chemistry. Even if physical models were recently completed by numerical in silico molecular visualizations which offer a wide range of molecular representations and rendering features, they are still involved in research work and teaching, because they are more suitable than virtual objects for manipulating and building molecular structures. In this paper, we present a markerless tracking method to construct a molecular virtual representation from a flexible and modular physical model. Our approach is based on a single RGB camera to reconstruct the physical model in interactive time in order to use it as a tangible interface, and thus benefits from both physical and virtual representations. This method was designed to require only a light virtual and augmented reality hardware setup, such as a smartphone or HMD & mounted camera, providing a markerless molecular tangible interface suitable for a classroom context or a classical biochemistry researcher desktop. The approach proposes a fast image processing algorithm based on color blob detection to extract 2D atom positions of a user-defined conformation in each frame of a video. A tracking algorithm recovers a set of 2D projected atom positions as an input of the 3D reconstruction stage, based on a Structure From Motion method. We tuned this method to robustly process a few key feature points and combine them within a global point cloud. Biological knowledge drives the final reconstruction, filling missing atoms to obtain the desired molecular conformation

Efficient Error-bounded Curvature Optimization for Smooth Machining Paths

Automated machining with 3-axis robots requires the generation of tool paths in form of positions of the tool tip. For 5-axis robots, the orientations of the tool at each position needs to be provided, as well. Such a tool path can be described in form of two curves, one for the positional information (as for 3-axis machining) and one for the orientational information, where the orientation is given by the vector that points from a point on the orientation curve to the respective point on the position curve. As the robots need to slow down for sharp turns, i.e., high curvatures in the tool path lead to slow processing, our goal is to generate tool paths with minimized curvatures and a guaranteed error bound. Starting from an initial tool path, which is given in the form of polygonal representations of the position and orientation curves, we generate optimized versions of the curves in the form of B-spline curves that lie within some error bounds of the input path. Our approach first computes an optimized version of the position curve within a tolerance band of the input curve. The outcome of this first step can directly be applied to 3-axis machining. Based on this first step, for 5-axis machining the orientation curve needs to be updated to again fit the position curve. Then, the orientation curve is optimized using a similar approach as for the position curve, but the error bounds are given in the form of tolerance frustums that define the tolerance in lead and tilt. For an efficient optimization procedure, our approach analyzes the input path and splits it into small (partially overlapping) groups before optimizing the position curve. The groups are categorized according to their geometric complexity and handled accordingly using two different optimization procedures. The simpler, but faster algorithm uses a local spline approximation, while the slower, but better algorithm uses a local sleeve approach. These algorithms are adapted to both the position and orientation curve optimization. Subsequently, the groups are combined into a complete tool path in the form of G2- continuous B-spline curves, where we have one such curve for 3-axis machining and two such curves defined over the same knot vector for 5-axis machining

Interactive Hyper Spectral Image Rendering on GPU

In this paper, we describe a framework focused on spectral images rendering. The rendering of a such image leads us to three major issues: the computation time, the footprint of the spectral image, and the memory consumption of the algorithm. The computation time can be drastically reduced by the use of GPUs, however, their memory capacity and bandwidth are limited. When the spectral dimension of the image will raise, the straightforward approach of the Path Tracing will lead us to high memory consumption and latency problems. To overcome these problems, we propose the DPEPT (Deferred Path Evaluation Path Tracing) which consists in decoupling the path evaluation from the path generation. This technique reduces the memory latency and consumption of the Path Tracing. It allows us to use an efficient wavelength samples batches parallelization pattern to optimize the path evaluation step and outperforms the straightforward approach