Dynamic Illumination for Augmented Reality with Real-Time Interaction

Current augmented and mixed reality systems suffer a lack of correct illumination modeling where the virtual objects render the same lighting condition as the real environment. While we are experiencing astonishing results from the entertainment industry in multiple media forms, the procedure is mostly accomplished offline. The illumination information extracted from the physical scene is used to interactively render the virtual objects which results in a more realistic output in real-time. In this paper, we present a method that detects the physical illumination with dynamic scene, then uses the extracted illumination to render the virtual objects added to the scene. The method has three steps that are assumed to be working concurrently in real-time. The first is the estimation of the direct illumination (incident light) from the physical scene using computer vision techniques through a 360° live-feed camera connected to AR device. The second is the simulation of indirect illumination (reflected light) from the real-world surfaces to virtual objects rendering using region capture of 2D texture from the AR camera view. The third is defining the virtual objects with proper lighting and shadowing characteristics using shader language through multiple passes. Finally, we tested our work with multiple lighting conditions to evaluate the accuracy of results based on the shadow falling from the virtual objects which should be consistent with the shadow falling from the real objects with a reduced performance cost

Immersive and non immersive 3D virtual city: decision support tool for urban sustainability

Sustainable urban planning decisions must not only consider the physical structure of the urban development but the economic, social and environmental factors. Due to the prolonged times scales of major urban development projects the current and future impacts of any decision made must be fully understood. Many key project decisions are made early in the decision making process with decision makers later seeking agreement for proposals once the key decisions have already been made, leaving many stakeholders, especially the general public, feeling marginalised by the process. Many decision support tools have been developed to aid in the decision making process, however many of these are expert orientated, fail to fully address spatial and temporal issues and do not reflect the interconnectivity of the separate domains and their indicators. This paper outlines a platform that combines computer game techniques, modelling of economic, social and environmental indicators to provide an interface that presents a 3D interactive virtual city with sustainability information overlain. Creating a virtual 3D urban area using the latest video game techniques ensures: real-time rendering of the 3D graphics; exploitation of novel techniques of how complex multivariate data is presented to the user; immersion in the 3D urban development, via first person navigation, exploration and manipulation of the environment with consequences updated in real-time. The use of visualisation techniques begins to remove sustainability assessment’s reliance on the existing expert systems which are largely inaccessible to many of the stakeholder groups, especially the general public

Contributing to VRPN with a new server for haptic devices (ext. version)

This article is an extended version of the poster paper: Cuevas-Rodriguez, M., Gonzalez-Toledo D., Molina-Tanco, L., Reyes-Lecuona A., 2015, November. “Contributing to VRPN with a new server for haptic devices”. In Proceedings of the ACM symposium on Virtual reality software and technology. ACM.http://dx.doi.org/10.1145/2821592.2821639VRPN is a middleware to access Virtual Reality peripherals. VRPN standard distribution supports Geomagic® (formerly Phantom) haptic devices through the now superseded GHOST library. This paper presents VRPN OpenHaptics Server, a contribution to VRPN library that fully reimplements VRPN support of Geomagic Haptic Devices. The implementation is based on the OpenHaptics v3.0 HLAPI layer, which supports all Geomagic Haptic Devices. We present the architecture of the contributed server, a detailed description of the offered API and an analysis of its performance in a set of example scenarios.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

A multi-projector CAVE system with commodity hardware and gesture-based interaction

Spatially-immersive systems such as CAVEs provide users with surrounding worlds by projecting 3D models on multiple screens around the viewer. Compared to alternative immersive systems such as HMDs, CAVE systems are a powerful tool for collaborative inspection of virtual environments due to better use of peripheral vision, less sensitivity to tracking errors, and higher communication possibilities among users. Unfortunately, traditional CAVE setups require sophisticated equipment including stereo-ready projectors and tracking systems with high acquisition and maintenance costs. In this paper we present the design and construction of a passive-stereo, four-wall CAVE system based on commodity hardware. Our system works with any mix of a wide range of projector models that can be replaced independently at any time, and achieves high resolution and brightness at a minimum cost. The key ingredients of our CAVE are a self-calibration approach that guarantees continuity across the screen, as well as a gesture-based interaction approach based on a clever combination of skeletal data from multiple Kinect sensors.Preprin

HeadOn: Real-time Reenactment of Human Portrait Videos

We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. On top of the coarse geometric proxy, we propose a video-based rendering technique that composites the modified target portrait video via view- and pose-dependent texturing, and creates photo-realistic imagery of the target actor under novel torso and head poses, facial expressions, and gaze directions. To this end, we propose a robust tracking of the face and torso of the source actor. We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.Comment: Video: https://www.youtube.com/watch?v=7Dg49wv2c_g Presented at Siggraph'1

Cosmic cookery : making a stereoscopic 3D animated movie.

This paper describes our experience making a short stereoscopic movie visualizing the development of structure in the universe during the 13.7 billion years from the Big Bang to the present day. Aimed at a general audience for the Royal Society's 2005 Summer Science Exhibition, the movie illustrates how the latest cosmological theories based on dark matter and dark energy are capable of producing structures as complex as spiral galaxies and allows the viewer to directly compare observations from the real universe with theoretical results. 3D is an inherent feature of the cosmology data sets and stereoscopic visualization provides a natural way to present the images to the viewer, in addition to allowing researchers to visualize these vast, complex data sets. The presentation of the movie used passive, linearly polarized projection onto a 2m wide screen but it was also required to playback on a Sharp RD3D display and in anaglyph projection at venues without dedicated stereoscopic display equipment. Additionally lenticular prints were made from key images in the movie. We discuss the following technical challenges during the stereoscopic production process; 1) Controlling the depth presentation, 2) Editing the stereoscopic sequences, 3) Generating compressed movies in display speci¯c formats. We conclude that the generation of high quality stereoscopic movie content using desktop tools and equipment is feasible. This does require careful quality control and manual intervention but we believe these overheads are worthwhile when presenting inherently 3D data as the result is signi¯cantly increased impact and better understanding of complex 3D scenes