Occlusion Handling using Semantic Segmentation and Visibility-Based Rendering for Mixed Reality

Real-time occlusion handling is a major problem in outdoor mixed reality system because it requires great computational cost mainly due to the complexity of the scene. Using only segmentation, it is difficult to accurately render a virtual object occluded by complex objects such as trees, bushes etc. In this paper, we propose a novel occlusion handling method for real-time, outdoor, and omni-directional mixed reality system using only the information from a monocular image sequence. We first present a semantic segmentation scheme for predicting the amount of visibility for different type of objects in the scene. We also simultaneously calculate a foreground probability map using depth estimation derived from optical flow. Finally, we combine the segmentation result and the probability map to render the computer generated object and the real scene using a visibility-based rendering method. Our results show great improvement in handling occlusions compared to existing blending based methods

Photography as a tool of Alienation: Aura

Regular photographical imaging record volumetric planes with smooth surfaces. The reason is the camera’s deficiency in perceiving and documenting the visual richness of “persuasive” details in life. HDR imaging methods used in creating this artwork series titled “Aura” helped making invisible organism-like textures emerge and point to the notions of decay and symbiosis. One of the main objectives in this series of artworks is to facilitate the emergence of the experiential visual complexity between the animate and inanimate, that is otherwise not possible to record. The latent aura of textural presences around us is not always noticeable easily since we tend to consume things too fast. With the rich textures achieved after high-dynamic-range-imaging (HDRI) procedures, a new symbiotic painterly visual relationship between biological (humans) and non-biological (space) was intended. In addition, the paper will focus on photography rather as a tool of personal world making, instead of photography as witnessing. During the process of unfolding this practice; notions of superimposition, palimpsest, painting vs. photography, truth, photography as an apparatus to provoke de-familiarization will be covered. The final aim is to confirm photography as a visual language that enriches and transforms human perception

Robots, drugs, reality and education: how the future will change how we think

Emerging technologies for learning report - Article exploring various future trends and their potential impact on educatio

Planar Refrains

My practice explores phenomenal poetic truths that exist in fissures between the sensual and physical qualities of material constructs. Magnifying this confounding interspace, my work activates specific instruments within mutable, relational systems of installation, movement, and documentation. The tools I fabricate function within variable orientations and are implemented as both physical barriers and thresholds into alternate, virtual domains. Intersecting fragments of sound and moving image build a nexus of superimposed spatialities, while material constructions are enveloped in ephemeral intensities. Within this compounded environment, both mind and body are charged as active sites through which durational, contemplative experiences can pass. Reverberation, the ghostly refrain of a sound calling back to our ears from a distant plane, can intensify our emotional experience of place. My project Planar Refrains utilizes four electro-mechanical reverb plates, analog audio filters designed to simulate expansive acoustic arenas. Historically these devices have provided emotive voicings to popular studio recordings, dislocating the performer from the commercial studio and into a simulated reverberant territory of mythic proportions. The material resonance of steel is used to filter a recorded signal, shaping the sound of a human performance into something more transformative, a sound embodying otherworldly dynamics. In subverting the designed utility of reverb plates, I am exploring their value as active surfaces extending across different spatial realities. The background of ephemeral sonic residue is collapsed into the foreground, a filter becomes sculpture, and this sculpture becomes an instrument in an evolving soundscape

SiTAR: Situated Trajectory Analysis for In-the-Wild Pose Error Estimation

Virtual content instability caused by device pose tracking error remains a prevalent issue in markerless augmented reality (AR), especially on smartphones and tablets. However, when examining environments which will host AR experiences, it is challenging to determine where those instability artifacts will occur; we rarely have access to ground truth pose to measure pose error, and even if pose error is available, traditional visualizations do not connect that data with the real environment, limiting their usefulness. To address these issues we present SiTAR (Situated Trajectory Analysis for Augmented Reality), the first situated trajectory analysis system for AR that incorporates estimates of pose tracking error. We start by developing the first uncertainty-based pose error estimation method for visual-inertial simultaneous localization and mapping (VI-SLAM), which allows us to obtain pose error estimates without ground truth; we achieve an average accuracy of up to 96.1% and an average F1 score of up to 0.77 in our evaluations on four VI-SLAM datasets. Next we present our SiTAR system, implemented for ARCore devices, combining a backend that supplies uncertainty-based pose error estimates with a frontend that generates situated trajectory visualizations. Finally, we evaluate the efficacy of SiTAR in realistic conditions by testing three visualization techniques in an in-the-wild study with 15 users and 13 diverse environments; this study reveals the impact both environment scale and the properties of surfaces present can have on user experience and task performance.Comment: To appear in Proceedings of IEEE ISMAR 202

A framework for realistic 3D tele-immersion

Meeting, socializing and conversing online with a group of people using teleconferencing systems is still quite differ- ent from the experience of meeting face to face. We are abruptly aware that we are online and that the people we are engaging with are not in close proximity. Analogous to how talking on the telephone does not replicate the experi- ence of talking in person. Several causes for these differences have been identified and we propose inspiring and innova- tive solutions to these hurdles in attempt to provide a more realistic, believable and engaging online conversational expe- rience. We present the distributed and scalable framework REVERIE that provides a balanced mix of these solutions. Applications build on top of the REVERIE framework will be able to provide interactive, immersive, photo-realistic ex- periences to a multitude of users that for them will feel much more similar to having face to face meetings than the expe- rience offered by conventional teleconferencing systems

Geoscience after IT: Part J. Human requirements that shape the evolving geoscience information system

The geoscience record is constrained by the limitations of human thought and of the technology for handling information. IT can lead us away from the tyranny of older technology, but to find the right path, we need to understand our own limitations. Language, images, data and mathematical models, are tools for expressing and recording our ideas. Backed by intuition, they enable us to think in various modes, to build knowledge from information and create models as artificial views of a real world. Markup languages may accommodate more flexible and better connected records, and the object-oriented approach may help to match IT more closely to our thought processes

Mathematical Nature of Reality, Plus Gravitation-Electromagnetism Unification, Derived from Revised Gravitational Tidal Forces and Mass-from-Gravity Concept

This article had its beginning with Einstein's 1919 paper "Do gravitational fields play an essential role in the structure of elementary particles?" Together with General Relativity's statement that gravity is not a pull but is a push caused by the curvature of space-time, a hypothesis for Earth's ocean tides was developed that does not solely depend on the Sun and Moon as Kepler and Newton believed. It also borrows from Galileo. The breakup of planets and asteroids by white dwarfs, neutron stars or black holes is popularly ascribed by today's science to tidal forces (gravitation emanating from the stellar body and having a greater effect on the near side of a planet/asteroid than the farthest side). Remembering Einstein's 1919 paper, it was apparent that my revised idea of tidal forces improves on current accounts because it views matter and mass as unified with space-time whose curvature is gravitation. Unification is a necessity for modern science's developing view of one united and entangled universe – expressed in the Unified Field Theory, the Theory of Everything, String theory and Loop Quantum Gravity. The writing of this article was also assisted by visualizing the gravitational fields forming space-time being themselves formed by a multitude of weak and presently undetectable gravitational waves. The final part of this article concludes that the section BITS AND TOPOLOGY will lead to the conclusions in ETERNAL LIFE, WORLD PEACE AND PHYSICS' UNIFICATION. The final part also compares cosmology to biological enzymes and biology's substrate of reacting "chemicals" - using virtual particles, hidden variables, gravitation, electromagnetism, electronics’ binary digits, plus topology’s Mobius strip and figure-8 Klein bottle. The product is mass - enzyme, substrate and product are all considered mathematical in nature. Also, gravitation and electromagnetism are united using logic and topology – showing there’s no need in this article for things like mathematical formalism, field equations or tensor calculus