Fusing Multimedia Data Into Dynamic Virtual Environments

In spite of the dramatic growth of virtual and augmented reality (VR and AR) technology, content creation for immersive and dynamic virtual environments remains a significant challenge. In this dissertation, we present our research in fusing multimedia data, including text, photos, panoramas, and multi-view videos, to create rich and compelling virtual environments. First, we present Social Street View, which renders geo-tagged social media in its natural geo-spatial context provided by 360° panoramas. Our system takes into account visual saliency and uses maximal Poisson-disc placement with spatiotemporal filters to render social multimedia in an immersive setting. We also present a novel GPU-driven pipeline for saliency computation in 360° panoramas using spherical harmonics (SH). Our spherical residual model can be applied to virtual cinematography in 360° videos. We further present Geollery, a mixed-reality platform to render an interactive mirrored world in real time with three-dimensional (3D) buildings, user-generated content, and geo-tagged social media. Our user study has identified several use cases for these systems, including immersive social storytelling, experiencing the culture, and crowd-sourced tourism. We next present Video Fields, a web-based interactive system to create, calibrate, and render dynamic videos overlaid on 3D scenes. Our system renders dynamic entities from multiple videos, using early and deferred texture sampling. Video Fields can be used for immersive surveillance in virtual environments. Furthermore, we present VRSurus and ARCrypt projects to explore the applications of gestures recognition, haptic feedback, and visual cryptography for virtual and augmented reality. Finally, we present our work on Montage4D, a real-time system for seamlessly fusing multi-view video textures with dynamic meshes. We use geodesics on meshes with view-dependent rendering to mitigate spatial occlusion seams while maintaining temporal consistency. Our experiments show significant enhancement in rendering quality, especially for salient regions such as faces. We believe that Social Street View, Geollery, Video Fields, and Montage4D will greatly facilitate several applications such as virtual tourism, immersive telepresence, and remote education

Improving neural networks for geospatial applications with geographic context embeddings

Geospatial data sits at the core of many data-driven application domains, from urban analytics to spatial epidemiology and climate science. Over recent years, ever-growing streams of data have allowed us to quantify more and more aspects of our lives and to deploy machine learning techniques to improve public and private services. But while modern neural network methods offer a flexible and scalable toolkit for high-dimensional data analysis, they can struggle with the complexities and dependencies of real-world geographic data. The particular challenges of geographic data are the subject of the geographic information sciences (GIS). This discipline has compiled a myriad of metrics and measures to quantify spatial effects and to improve modeling in the presence of spatial dependencies. In this dissertation, we deploy metrics of spatial interactions as embeddings to enrich neural network methods for geographic data. We utilize both, functional embeddings (such as measures of spatial autocorrelation) and parametric neural-network embeddings (such as semantic vector embeddings). The embeddings are then integrated into neural network methods using four different approaches: (1) model selection, (2) auxiliary task learning, (3) feature learning, and (4) embedding loss functions. Throughout the dissertation, we use experiments with various real-world datasets to highlight performance improvements of our geographically-explicit neural network methods over naive baselines. We focus specifically on generative and predictive modeling tasks. The dissertation highlights how geographic domain-expertise together with powerful neural network backbones can provide tailored, scalable modeling solutions for the era of real-time Earth observation and urban analytics