56 research outputs found
Spatiotemporal oriented energies for spacetime stereo
This paper presents a novel approach to recovering tem-porally coherent estimates of 3D structure of a dynamic scene from a sequence of binocular stereo images. The approach is based on matching spatiotemporal orientation distributions between left and right temporal image streams, which encapsulates both local spatial and temporal struc-ture for disparity estimation. By capturing spatial and tem-poral structure in this unified fashion, both sources of in-formation combine to yield disparity estimates that are nat-urally temporal coherent, while helping to resolve matches that might be ambiguous when either source is considered alone. Further, by allowing subsets of the orientation mea-surements to support different disparity estimates, an ap-proach to recovering multilayer disparity from spacetime stereo is realized. The approach has been implemented with real-time performance on commodity GPUs. Empir-ical evaluation shows that the approach yields qualitatively and quantitatively superior disparity estimates in compari-son to various alternative approaches, including the ability to provide accurate multilayer estimates in the presence of (semi)transparent and specular surfaces. 1
Two-Stream Convolutional Networks for Dynamic Texture Synthesis
This thesis introduces a two-stream model for dynamic texture synthesis. The model is based on pre-trained convolutional networks (ConvNets) that target two independent tasks: (i) object recognition, and (ii) optical flow regression. Given an input dynamic texture, statistics of filter responses from the object recognition and optical flow ConvNets encapsulate the per-frame appearance and dynamics of the input texture, respectively. To synthesize a dynamic texture, a randomly initialized input sequence is optimized to match the feature statistics from each stream of an example texture. In addition, the synthesis approach is applied to combine the texture appearance from one texture with the dynamics of another to generate entirely novel dynamic textures. Overall, the proposed approach generates high quality samples that match both the framewise appearance and temporal evolution of input texture. Finally, a quantitative evaluation of the proposed dynamic texture synthesis approach is performed via a large-scale user study
Learning Irreducible Representations of Noncommutative Lie Groups
Recent work has constructed neural networks that are equivariant to
continuous symmetry groups such as 2D and 3D rotations. This is accomplished
using explicit group representations to derive the equivariant kernels and
nonlinearities. We present two contributions motivated by frontier applications
of equivariance beyond rotations and translations. First, we relax the
requirement for explicit Lie group representations, presenting a novel
algorithm that finds irreducible representations of noncommutative Lie groups
given only the structure constants of the associated Lie algebra. Second, we
demonstrate that Lorentz-equivariance is a useful prior for object-tracking
tasks and construct the first object-tracking model equivariant to the
Poincar\'e group.Comment: 15 pages, 5 figure
Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
Motivated by the vast success of deep convolutional networks, there is a
great interest in generalizing convolutions to non-Euclidean manifolds. A major
complication in comparison to flat spaces is that it is unclear in which
alignment a convolution kernel should be applied on a manifold. The underlying
reason for this ambiguity is that general manifolds do not come with a
canonical choice of reference frames (gauge). Kernels and features therefore
have to be expressed relative to arbitrary coordinates. We argue that the
particular choice of coordinatization should not affect a network's inference
-- it should be coordinate independent. A simultaneous demand for coordinate
independence and weight sharing is shown to result in a requirement on the
network to be equivariant under local gauge transformations (changes of local
reference frames). The ambiguity of reference frames depends thereby on the
G-structure of the manifold, such that the necessary level of gauge
equivariance is prescribed by the corresponding structure group G. Coordinate
independent convolutions are proven to be equivariant w.r.t. those isometries
that are symmetries of the G-structure. The resulting theory is formulated in a
coordinate free fashion in terms of fiber bundles. To exemplify the design of
coordinate independent convolutions, we implement a convolutional network on
the M\"obius strip. The generality of our differential geometric formulation of
convolutional networks is demonstrated by an extensive literature review which
explains a large number of Euclidean CNNs, spherical CNNs and CNNs on general
surfaces as specific instances of coordinate independent convolutions.Comment: The implementation of orientation independent M\"obius convolutions
is publicly available at https://github.com/mauriceweiler/MobiusCNN
- …