Variational Fluid Motion Estimation with Physical Priors

In this thesis, techniques for Particle Image Velocimetry (PIV) and Particle Tracking Velocimetry (PTV) are developed that are based on variational methods. The basic idea is not to estimate displacement vectors locally and individually, but to estimate vector fields as a whole by minimizing a suitable functional defined over the entire image domain (which may be 2D or 3D and may also include the temporal dimension). Such functionals typically comprise two terms: a data-term measuring how well two images of a sequence match as a function of the vector field to be estimated, and a regularization term that brings prior knowledge into the energy functional. Our starting point are methods that were originally developed in the field of computer vision and that we modify for the purpose of PIV. These methods are based on the so-called optical flow: Optical flow denotes the estimated velocity vector inferred by a relative motion of camera and image scene and is based on the assumption of gray value conservation (i.e. the total derivative of the image gray value over time is zero). A regularization term (that demands e.g. smoothness of the velocity field, or of its divergence and rotation) renders the system mathematically well-posed. Experimental evaluation shows that this type of variational approach is able to outperform standard cross-correlation methods. In order to develop a variational method for PTV, we replace the continuous data term of variational approaches to PIV with a discrete non-differentiable particle matching term. This raises the problem of minimizing such data terms together with continuous regularization terms. We accomplish this with an advanced mathematical method, which guarantees convergence to a local minimum of such a non-convex variational approach to PTV. With this novel variational approach (there has been no previous work on modeling PTV methods with global variational approaches), we achieve results for image pairs and sequences in two and three dimensions that outperform the relaxation methods that are traditionally used for particle tracking. The key advantage of our variational particle image velocimetry methods, is the chance to include prior knowledge in a natural way. In the fluid environments that we are considering in this thesis, it is especially attractive to use priors that can be motivated from a physical point of view. Firstly, we present a method that only allows flow fields that satisfy the Stokes equation. The latter equation includes control variables that allow to control the optical flow so as to fit the apparent velocities of particles in a given image pair. Secondly, we present a variational approach to motion estimation of instationary fluid flows. This approach extends the prior method along two directions: (i) The full incompressible Navier-Stokes equation is employed in order to obtain a physically consistent regularization which does not suppress turbulent flow variations. (ii) Regularization along the time-axis is employed as well, but formulated in a receding horizon manner contrary to previous approaches to spatio-temporal regularization. Ground-truth evaluations for simulated turbulent flows demonstrate that the accuracy of both types of physically plausible regularization compares favorably with advanced cross-correlation approaches. Furthermore, the direct estimation of, e.g., pressure or vorticity becomes possible

Recent Progress in Image Deblurring

This paper comprehensively reviews the recent development of image deblurring, including non-blind/blind, spatially invariant/variant deblurring techniques. Indeed, these techniques share the same objective of inferring a latent sharp image from one or several corresponding blurry images, while the blind deblurring techniques are also required to derive an accurate blur kernel. Considering the critical role of image restoration in modern imaging systems to provide high-quality images under complex environments such as motion, undesirable lighting conditions, and imperfect system components, image deblurring has attracted growing attention in recent years. From the viewpoint of how to handle the ill-posedness which is a crucial issue in deblurring tasks, existing methods can be grouped into five categories: Bayesian inference framework, variational methods, sparse representation-based methods, homography-based modeling, and region-based methods. In spite of achieving a certain level of development, image deblurring, especially the blind case, is limited in its success by complex application conditions which make the blur kernel hard to obtain and be spatially variant. We provide a holistic understanding and deep insight into image deblurring in this review. An analysis of the empirical evidence for representative methods, practical issues, as well as a discussion of promising future directions are also presented.Comment: 53 pages, 17 figure

Self-similar prior and wavelet bases for hidden incompressible turbulent motion

This work is concerned with the ill-posed inverse problem of estimating turbulent flows from the observation of an image sequence. From a Bayesian perspective, a divergence-free isotropic fractional Brownian motion (fBm) is chosen as a prior model for instantaneous turbulent velocity fields. This self-similar prior characterizes accurately second-order statistics of velocity fields in incompressible isotropic turbulence. Nevertheless, the associated maximum a posteriori involves a fractional Laplacian operator which is delicate to implement in practice. To deal with this issue, we propose to decompose the divergent-free fBm on well-chosen wavelet bases. As a first alternative, we propose to design wavelets as whitening filters. We show that these filters are fractional Laplacian wavelets composed with the Leray projector. As a second alternative, we use a divergence-free wavelet basis, which takes implicitly into account the incompressibility constraint arising from physics. Although the latter decomposition involves correlated wavelet coefficients, we are able to handle this dependence in practice. Based on these two wavelet decompositions, we finally provide effective and efficient algorithms to approach the maximum a posteriori. An intensive numerical evaluation proves the relevance of the proposed wavelet-based self-similar priors.Comment: SIAM Journal on Imaging Sciences, 201

Strain Analysis by a Total Generalized Variation Regularized Optical Flow Model

In this paper we deal with the important problem of estimating the local strain tensor from a sequence of micro-structural images realized during deformation tests of engineering materials. Since the strain tensor is defined via the Jacobian of the displacement field, we propose to compute the displacement field by a variational model which takes care of properties of the Jacobian of the displacement field. In particular we are interested in areas of high strain. The data term of our variational model relies on the brightness invariance property of the image sequence. As prior we choose the second order total generalized variation of the displacement field. This prior splits the Jacobian of the displacement field into a smooth and a non-smooth part. The latter reflects the material cracks. An additional constraint is incorporated to handle physical properties of the non-smooth part for tensile tests. We prove that the resulting convex model has a minimizer and show how a primal-dual method can be applied to find a minimizer. The corresponding algorithm has the advantage that the strain tensor is directly computed within the iteration process. Our algorithm is further equipped with a coarse-to-fine strategy to cope with larger displacements. Numerical examples with simulated and experimental data demonstrate the very good performance of our algorithm. In comparison to state-of-the-art engineering software for strain analysis our method can resolve local phenomena much better

Disparity and Optical Flow Partitioning Using Extended Potts Priors

This paper addresses the problems of disparity and optical flow partitioning based on the brightness invariance assumption. We investigate new variational approaches to these problems with Potts priors and possibly box constraints. For the optical flow partitioning, our model includes vector-valued data and an adapted Potts regularizer. Using the notation of asymptotically level stable functions we prove the existence of global minimizers of our functionals. We propose a modified alternating direction method of minimizers. This iterative algorithm requires the computation of global minimizers of classical univariate Potts problems which can be done efficiently by dynamic programming. We prove that the algorithm converges both for the constrained and unconstrained problems. Numerical examples demonstrate the very good performance of our partitioning method

Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods that have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, and on real-world video frames. We present analyses of the learned network representations, showing it is implicitly learning a compact encoding of object appearance and motion. We also demonstrate a few of its applications, including visual analogy-making and video extrapolation.Comment: Journal preprint of arXiv:1607.02586 (IEEE TPAMI, 2019). The first two authors contributed equally to this work. Project page: http://visualdynamics.csail.mit.ed