Multiple View Geometry For Video Analysis And Post-production

Multiple view geometry is the foundation of an important class of computer vision techniques for simultaneous recovery of camera motion and scene structure from a set of images. There are numerous important applications in this area. Examples include video post-production, scene reconstruction, registration, surveillance, tracking, and segmentation. In video post-production, which is the topic being addressed in this dissertation, computer analysis of the motion of the camera can replace the currently used manual methods for correctly aligning an artificially inserted object in a scene. However, existing single view methods typically require multiple vanishing points, and therefore would fail when only one vanishing point is available. In addition, current multiple view techniques, making use of either epipolar geometry or trifocal tensor, do not exploit fully the properties of constant or known camera motion. Finally, there does not exist a general solution to the problem of synchronization of N video sequences of distinct general scenes captured by cameras undergoing similar ego-motions, which is the necessary step for video post-production among different input videos. This dissertation proposes several advancements that overcome these limitations. These advancements are used to develop an efficient framework for video analysis and post-production in multiple cameras. In the first part of the dissertation, the novel inter-image constraints are introduced that are particularly useful for scenes where minimal information is available. This result extends the current state-of-the-art in single view geometry techniques to situations where only one vanishing point is available. The property of constant or known camera motion is also described in this dissertation for applications such as calibration of a network of cameras in video surveillance systems, and Euclidean reconstruction from turn-table image sequences in the presence of zoom and focus. We then propose a new framework for the estimation and alignment of camera motions, including both simple (panning, tracking and zooming) and complex (e.g. hand-held) camera motions. Accuracy of these results is demonstrated by applying our approach to video post-production applications such as video cut-and-paste and shadow synthesis. As realistic image-based rendering problems, these applications require extreme accuracy in the estimation of camera geometry, the position and the orientation of the light source, and the photometric properties of the resulting cast shadows. In each case, the theoretical results are fully supported and illustrated by both numerical simulations and thorough experimentation on real data

Overcoming shadows in 3-source photometric stereo

Light occlusions are one of the most significant difficulties of photometric stereo methods. When three or more images are available without occlusion, the local surface orientation is overdetermined so that shape can be computed and the shadowed pixels can be discarded. In this paper, we look at the challenging case when only two images are available without occlusion, leading to a one degree of freedom ambiguity per pixel in the local orientation. We show that, in the presence of noise, integrability alone cannot resolve this ambiguity and reconstruct the geometry in the shadowed regions. As the problem is ill-posed in the presence of noise, we describe two regularization schemes that improve the numerical performance of the algorithm while preserving the data. Finally, the paper describes how this theory applies in the framework of color photometric stereo where one is restricted to only three images and light occlusions are common. Experiments on synthetic and real image sequences are presented

The cognitive science of holes and cast shadows

A comparative review of evidence related to the perceptual and conceptual representations of shadows and holes, with a list of open research questions

Recovering refined surface normals for relighting clothing in dynamic scenes

In this paper we present a method to relight captured 3D video sequences of non-rigid, dynamic scenes, such as clothing of real actors, reconstructed from multiple view video. A view-dependent approach is introduced to refine an initial coarse surface reconstruction using shape-from-shading to estimate detailed surface normals. The prior surface approximation is used to constrain the simultaneous estimation of surface normals and scene illumination, under the assumption of Lambertian surface reflectance. This approach enables detailed surface normals of a moving non-rigid object to be estimated from a single image frame. Refined normal estimates from multiple views are integrated into a single surface normal map. This approach allows highly non-rigid surfaces, such as creases in clothing, to be relit whilst preserving the detailed dynamics observed in video

Conversions of Relief: on the perception of depth in drawings

Of the many ways in which depth can be intimated in drawings, perspective has undoubtedly been one of the most frequently examined. But there is also an equally rich history associated with other forms of pictorial representation. Alternatives to perspective became particularly significant in the early twentieth century as artists and architects, intent on throwing off the conventions of their predecessors, looked to new ways of depicting depth. In architecture, this tendency was exemplified by Modernism’s preference for parallel projection – most notably axonometric and oblique. The use of these techniques gave architects the opportunity to convey a new and uniquely modern form of spatial expression. At once shallow and yet expansive, a key feature of these drawings was their ability to support perceptual ambiguity. This paper will consider the philosophy and science of vision, out of which these preoccupations emerged. In this context, the nineteenth-century discovery of stereopsis and the invention of the stereoscope will be used to illustrate the way in which attempts to test the limits of spatial perception led to an opening up of visual experience; and provided a definition of visual experience that could encompass the representational ambiguities later exploited by the early twentieth-century avant-garde

Differentiable Shadow Mapping for Efficient Inverse Graphics

We show how shadows can be efficiently generated in differentiable rendering of triangle meshes. Our central observation is that pre-filtered shadow mapping, a technique for approximating shadows based on rendering from the perspective of a light, can be combined with existing differentiable rasterizers to yield differentiable visibility information. We demonstrate at several inverse graphics problems that differentiable shadow maps are orders of magnitude faster than differentiable light transport simulation with similar accuracy -- while differentiable rasterization without shadows often fails to converge.Comment: CVPR 2023, project page: https://mworchel.github.io/differentiable-shadow-mappin

Constructing the invisible - Computer graphics and the end of Optical Media

This media archaeology of early computer graphics tackles the relation between the imaginary and technical media. We focus on the algorithmic procedures and mathematical principles driving 3D computer graphics during the 1970’s and we frame them in a discussion about the end of optical media and a reflection on the current situation in which images, commanded by techno-codes, delineate and structure the dominant code of communication with which we imagine. Even though algorithmic simulations of optical worlds do not represent the end of optical media as Friedrich Kittler once argued, computer graphics can be seen retrospectively as an escalation in the production of invisibility. We introduce Frieder Nake’s concept of ‘subfaces’ to describe digital images as entities that are composed out of visible and invisible processes. The subface constitutes our methodological tool to analyze computer graphics historically, through three early problems of 3D computer graphics. We complement this media archeology of early computer graphics with discussing the dialogue between Kittler and Vilém Flusser on the imaginary or techo-imagination

A perceptual-statistics shading model

The process of surface perception is complex and based on several influencing factors, e.g., shading, silhouettes, occluding contours, and top down cognition. The accuracy of surface perception can be measured and the influencing factors can be modified in order to decrease the error in perception. This paper presents a novel concept of how a perceptual evaluation of a visualization technique can contribute to its redesign with the aim of improving the match between the distal and the proximal stimulus. During analysis of data from previous perceptual studies, we observed that the slant of 3D surfaces visualized on 2D screens is systematically underestimated. The visible trends in the error allowed us to create a statistical model of the perceived surface slant. Based on this statistical model we obtained from user experiments, we derived a new shading model that uses adjusted surface normals and aims to reduce the error in slant perception. The result is a shape-enhancement of visualization which is driven by an experimentally-founded statistical model. To assess the efficiency of the statistical shading model, we repeated the evaluation experiment and confirmed that the error in perception was decreased. Results of both user experiments are publicly-available datasets