SPATIO-TEMPORAL REGISTRATION IN AUGMENTED REALITY

The overarching goal of Augmented Reality (AR) is to provide users with the illusion that virtual and real objects coexist indistinguishably in the same space. An effective persistent illusion requires accurate registration between the real and the virtual objects, registration that is spatially and temporally coherent. However, visible misregistration can be caused by many inherent error sources, such as errors in calibration, tracking, and modeling, and system delay. This dissertation focuses on new methods that could be considered part of "the last mile" of spatio-temporal registration in AR: closed-loop spatial registration and low-latency temporal registration: 1. For spatial registration, the primary insight is that calibration, tracking and modeling are means to an end---the ultimate goal is registration. In this spirit I present a novel pixel-wise closed-loop registration approach that can automatically minimize registration errors using a reference model comprised of the real scene model and the desired virtual augmentations. Registration errors are minimized in both global world space via camera pose refinement, and local screen space via pixel-wise adjustments. This approach is presented in the context of Video See-Through AR (VST-AR) and projector-based Spatial AR (SAR), where registration results are measurable using a commodity color camera. 2. For temporal registration, the primary insight is that the real-virtual relationships are evolving throughout the tracking, rendering, scanout, and display steps, and registration can be improved by leveraging fine-grained processing and display mechanisms. In this spirit I introduce a general end-to-end system pipeline with low latency, and propose an algorithm for minimizing latency in displays (DLP DMD projectors in particular). This approach is presented in the context of Optical See-Through AR (OST-AR), where system delay is the most detrimental source of error. I also discuss future steps that may further improve spatio-temporal registration. Particularly, I discuss possibilities for using custom virtual or physical-virtual fiducials for closed-loop registration in SAR. The custom fiducials can be designed to elicit desirable optical signals that directly indicate any error in the relative pose between the physical and projected virtual objects.Doctor of Philosoph

Optical-inertia space sextant for an advanced space navigation system, phase B

Optical-inertia space sextant for advanced space navigation syste

Kinect Range Sensing: Structured-Light versus Time-of-Flight Kinect

Recently, the new Kinect One has been issued by Microsoft, providing the next generation of real-time range sensing devices based on the Time-of-Flight (ToF) principle. As the first Kinect version was using a structured light approach, one would expect various differences in the characteristics of the range data delivered by both devices. This paper presents a detailed and in-depth comparison between both devices. In order to conduct the comparison, we propose a framework of seven different experimental setups, which is a generic basis for evaluating range cameras such as Kinect. The experiments have been designed with the goal to capture individual effects of the Kinect devices as isolatedly as possible and in a way, that they can also be adopted, in order to apply them to any other range sensing device. The overall goal of this paper is to provide a solid insight into the pros and cons of either device. Thus, scientists that are interested in using Kinect range sensing cameras in their specific application scenario can directly assess the expected, specific benefits and potential problem of either device.Comment: 58 pages, 23 figures. Accepted for publication in Computer Vision and Image Understanding (CVIU

The Robo-AO-2 facility for rapid visible/near-infrared AO imaging and the demonstration of hybrid techniques

We are building a next-generation laser adaptive optics system, Robo-AO-2, for the UH 2.2-m telescope that will deliver robotic, diffraction-limited observations at visible and near-infrared wavelengths in unprecedented numbers. The superior Maunakea observing site, expanded spectral range and rapid response to high-priority events represent a significant advance over the prototype. Robo-AO-2 will include a new reconfigurable natural guide star sensor for exquisite wavefront correction on bright targets and the demonstration of potentially transformative hybrid AO techniques that promise to extend the faintness limit on current and future exoplanet adaptive optics systems.Comment: 15 page

Characteristics of flight simulator visual systems

The physical parameters of the flight simulator visual system that characterize the system and determine its fidelity are identified and defined. The characteristics of visual simulation systems are discussed in terms of the basic categories of spatial, energy, and temporal properties corresponding to the three fundamental quantities of length, mass, and time. Each of these parameters are further addressed in relation to its effect, its appropriate units or descriptors, methods of measurement, and its use or importance to image quality

Projector Compensation for Unconventional Projection Surface

Projecting onto irregular textured surfaces found on buildings, automobiles and theatre stages calls for the development of radiometric and geometric compensation algorithms that require no user intervention and compensate for the patterning and colourization of the background surface. This process needs a projector-camera setup where the feedback from the camera is used to learn the background's geometric and radiometric properties. In this thesis, radiometric compensation, which is used to correct for the background texture distortion, is discussed in detail. Existing compensation frameworks assume no inter--pixel coupling and develop an independent compensation model for each projector pixel. This assumption is valid on background with uniform texture variation but fails at sharp contrast differences leading to visible edge artifacts in the compensated image. To overcome the edge artifacts, a novel radiometric compensation approach is presented that directly learns the compensation model, rather than inverting a learned forward model. That is, the proposed method uses spatially uniform camera images to learn the projector images that successfully hide the background. The proposed approach can be used with any existing radiometric compensation algorithm to improve its performance. Comparisons with classical and state-of-the-art methods show the superiority of the proposed method in terms of the perceived image quality and computational complexity. The modified target image from the radiometric compensation algorithm can exceed the limited dynamic range of the projector resulting in saturation artifacts in the compensated image. Since the achievable range of luminance on the background surface with the given projector is limited, the projector compensation should also consider the contents of the target image along with the background properties while calculating the projector image. A novel spatially optimized luminance modification approach is proposed using human visual system properties to reduce the saturation artifacts. Here, the tolerance of the human visual system is exploited to make perceptually less sensitive modifications to the target image that in turn reduces the luminance demands from the projector. The proposed spatial modification approach can be combined with any radiometric compensation models to improve its performance. The simulated results of the proposed luminance modification are evaluated to show the improvement in perceptual performance. The inverse approach combined with the spatial luminance modification concludes the proposed projector compensation, which enables the optimum compensated projection on an arbitrary background surface

Synchronized Illumination Modulation for Digital Video Compositing

Informationsaustausch ist eines der Grundbedürfnisse der Menschen. Während früher dazu Wandmalereien,Handschrift, Buchdruck und Malerei eingesetzt wurden, begann man später, Bildfolgen zu erstellen, die als sogenanntes ”Daumenkino” den Eindruck einer Animation vermitteln. Diese wurden schnell durch den Einsatz rotierender Bildscheiben, auf denen mit Hilfe von Schlitzblenden, Spiegeln oder Optiken eine Animation sichtbar wurde, automatisiert – mit sogenannten Phenakistiskopen,Zoetropen oder Praxinoskopen. Mit der Erfindung der Fotografie begannen in der zweiten Hälfte des 19. Jahrhunderts die ersten Wissenschaftler wie Eadweard Muybridge, Etienne-Jules Marey und Ottomar Anschütz, Serienbildaufnahmen zu erstellen und diese dann in schneller Abfolge, als Film, abzuspielen. Mit dem Beginn der Filmproduktion wurden auch die ersten Versuche unternommen, mit Hilfe dieser neuen Technik spezielle visuelle Effekte zu generieren, um damit die Immersion der Bewegtbildproduktionen weiter zu erhöhen. Während diese Effekte in der analogen Phase der Filmproduktion bis in die achtziger Jahre des 20.Jahrhunderts recht beschränkt und sehr aufwendig mit einem enormen manuellen Arbeitsaufwand erzeugt werden mussten, gewannen sie mit der sich rapide beschleunigenden Entwicklung der Halbleitertechnologie und der daraus resultierenden vereinfachten digitalen Bearbeitung immer mehr an Bedeutung. Die enormen Möglichkeiten, die mit der verlustlosen Nachbearbeitung in Kombination mit fotorealistischen, dreidimensionalen Renderings entstanden, führten dazu, dass nahezu alle heute produzierten Filme eine Vielfalt an digitalen Videokompositionseffekten beinhalten. ...Besides home entertainment and business presentations, video projectors are powerful tools for modulating images spatially as well as temporally. The re-evolving need for stereoscopic displays increases the demand for low-latency projectors and recent advances in LED technology also offer high modulation frequencies. Combining such high-frequency illumination modules with synchronized, fast cameras, makes it possible to develop specialized high-speed illumination systems for visual effects production. In this thesis we present different systems for using spatially as well as temporally modulated illumination in combination with a synchronized camera to simplify the requirements of standard digital video composition techniques for film and television productions and to offer new possibilities for visual effects generation. After an overview of the basic terminology and a summary of related methods, we discuss and give examples of how modulated light can be applied to a scene recording context to enable a variety of effects which cannot be realized using standard methods, such as virtual studio technology or chroma keying. We propose using high-frequency, synchronized illumination which, in addition to providing illumination, is modulated in terms of intensity and wavelength to encode technical information for visual effects generation. This is carried out in such a way that the technical components do not influence the final composite and are also not visible to observers on the film set. Using this approach we present a real-time flash keying system for the generation of perspectively correct augmented composites by projecting imperceptible markers for optical camera tracking. Furthermore, we present a system which enables the generation of various digital video compositing effects outside of completely controlled studio environments, such as virtual studios. A third temporal keying system is presented that aims to overcome the constraints of traditional chroma keying in terms of color spill and color dependency. ..

Robo-AO: autonomous and replicable laser-adaptive-optics and science system

We have created a new autonomous laser-guide-star adaptive-optics (AO) instrument on the 60-inch (1.5-m) telescope at Palomar Observatory called Robo-AO. The instrument enables diffraction-limited resolution observing in the visible and near-infrared with the ability to observe well over one-hundred targets per night due to its fully robotic operation. Robo-AO is being used for AO surveys of targets numbering in the thousands, rapid AO imaging of transient events and long-term AO monitoring not feasible on large diameter telescope systems. We have taken advantage of cost-effective advances in deformable mirror and laser technology while engineering Robo-AO with the intention of cloning the system for other few-meter class telescopes around the world

Low Latency Displays for Augmented Reality

The primary goal for Augmented Reality (AR) is bringing the real and virtual together into a common space. Maintaining this illusion, however, requires preserving spatially and temporally consistent registration despite changes in user or object pose. The greatest source of registration error is latency—the delay between when something moves and the display changes in response—which breaks temporal consistency. Furthermore, the real world varies greatly in brightness; ranging from bright sunlight to deep shadows. Thus, a compelling AR system must also support High-Dynamic Range (HDR) to maintain its virtual objects’ appearance both spatially and temporally consistent with the real world. This dissertation presents new methods, implementations, results (both visual and performance), and future steps for low latency displays, primarily in the context of Optical See-through Augmented Reality (OST-AR) Head-Mounted Displays, focusing on temporal consistency in registration, HDR color support, and spatial and temporal consistency in brightness: 1. For registration temporal consistency, the primary insight is breaking the conventional display paradigm: computers render imagery, frame by frame, and then transmit it to the display for emission. Instead, the display must also contribute towards rendering by performing a post-rendering, post-transmission warp of the computer-supplied imagery in the display hardware. By compensating in the display for system latency by using the latest tracking information, much of the latency can be short-circuited. Furthermore, the low latency display must support ultra-high frequency (multiple kHz) refreshing to minimize pose displacement between updates. 2. For HDR color support, the primary insight is developing new display modulation techniques. DMDs, a type of ultra-high frequency display, emit binary output, which require modulation to produce multiple brightness levels. Conventional modulation breaks low latency guarantees, and modulation of bright LEDs illuminators at frequencies to support kHz-order HDR exceeds their capabilities. Thus one must directly synthesize the necessary variation in brightness. 3. For spatial and temporal brightness consistency, the primary insight is integrating HDR light sensors into the display hardware: the same processes which both compensate for latency and generate HDR output can also modify it in response to the spatially sensed brightness of the real world.Doctor of Philosoph