Variational fluid flow measurements from image sequences: synopsis and perspectives

[Departement_IRSTEA]Ecotechnologies [TR1_IRSTEA]SPEEVariational approaches to image motion segmentation has been an active field of study in image processing and computer vision for two decades. We present a short overview over basic estimation schemes and report in more detail recent modifications and applications to fluid flow estimation. Key properties of these approaches are illustrated by numerical examples. We outline promising research directions and point out the potential of variational techniques in combination with correlation-based PIV methods, for improving the consistency of fluid flow estimation and simulation

Double Normalizing Flows: Flexible Bayesian Gaussian Process ODEs Learning

Recently, Gaussian processes have been utilized to model the vector field of continuous dynamical systems. Bayesian inference for such models \cite{hegde2022variational} has been extensively studied and has been applied in tasks such as time series prediction, providing uncertain estimates. However, previous Gaussian Process Ordinary Differential Equation (ODE) models may underperform on datasets with non-Gaussian process priors, as their constrained priors and mean-field posteriors may lack flexibility. To address this limitation, we incorporate normalizing flows to reparameterize the vector field of ODEs, resulting in a more flexible and expressive prior distribution. Additionally, due to the analytically tractable probability density functions of normalizing flows, we apply them to the posterior inference of GP ODEs, generating a non-Gaussian posterior. Through these dual applications of normalizing flows, our model improves accuracy and uncertainty estimates for Bayesian Gaussian Process ODEs. The effectiveness of our approach is demonstrated on simulated dynamical systems and real-world human motion data, including tasks such as time series prediction and missing data recovery. Experimental results indicate that our proposed method effectively captures model uncertainty while improving accuracy

Time-consistent estimators of 2D/3D motion of atmospheric layers from pressure images

In this paper, we face the challenging problem of estimation of time-consistent layer motion fields at various atmospheric depths. Based on a vertical decomposition of the atmosphere, we propose three different dense motion estimator relying on multi-layer dynamical models. In the first method, we propose a mass conservation model which constitutes the physical background of a multi-layer dense estimator. In the perspective of adapting motion analysis to atmospheric motion, we propose in this method a two-stage decomposition estimation scheme. The second method proposed in this paper relying on a 3D physical model for a stack of interacting layers allows us to recover a vertical motion information. In the last method, we use the exact shallow-water formulation of the Navier-Stokes equations to control the motion evolution across the sequence. This is done through a variational approach derived from data assimilation principle which combines the dynamical model and the pressure difference observations obtained from satellite images. The three methods use sparse pressure difference image observations derived from top of cloud images and classification maps. The proposed approaches are validated on synthetic example and applied to real world meteorological satellite image sequences

Novel learning-based techniques for dense fluid motion measurements

In this thesis novel learning-based approaches are presented for the estimation of dense fluid flow velocity fields from particle image sequences. The developed methods apply prior knowledge in form of typical spatio-temporal motion models. These motion models are obtained with methods of unsupervised learning using proper orthogonal decomposition (POD). The POD modes reveal dominant flow structures and contain relevant information of complex relations between neighboring flow vectors. The first high-energy POD modes obtained from appropriate training vector fields are used as typical motion models. Meaningful local flow structures can be expressed in the orthogonal space spanned by the motion models. Additional information about dominant flow events is gained by the motion models and related parameters. The proposed approaches are embedded in well-established local parametric and variational optical flow frameworks but are contrasted with these common techniques by the inclusion of prior knowledge. Further extensions of the methods use available information, which is generally discarded in other methods, to obtain robust motion estimations. The methods can easily be tuned for different flow applications by choice of training data and, thus, are universally applicable. Beyond their simple implementation, the approaches are very efficient, accurate and easily adaptable to all types of flow situations. All methods were tested on synthetic and real particle image sequences and the influence of the relevant parameters was investigated. For typical use cases of optical flow, such as small image displacements, they were more accurate compared to all competing methods including particle image velocimetry (PIV) and common optical flow techniques

3D Fluid Flow Estimation with Integrated Particle Reconstruction

The standard approach to densely reconstruct the motion in a volume of fluid is to inject high-contrast tracer particles and record their motion with multiple high-speed cameras. Almost all existing work processes the acquired multi-view video in two separate steps, utilizing either a pure Eulerian or pure Lagrangian approach. Eulerian methods perform a voxel-based reconstruction of particles per time step, followed by 3D motion estimation, with some form of dense matching between the precomputed voxel grids from different time steps. In this sequential procedure, the first step cannot use temporal consistency considerations to support the reconstruction, while the second step has no access to the original, high-resolution image data. Alternatively, Lagrangian methods reconstruct an explicit, sparse set of particles and track the individual particles over time. Physical constraints can only be incorporated in a post-processing step when interpolating the particle tracks to a dense motion field. We show, for the first time, how to jointly reconstruct both the individual tracer particles and a dense 3D fluid motion field from the image data, using an integrated energy minimization. Our hybrid Lagrangian/Eulerian model reconstructs individual particles, and at the same time recovers a dense 3D motion field in the entire domain. Making particles explicit greatly reduces the memory consumption and allows one to use the high-res input images for matching. Whereas the dense motion field makes it possible to include physical a-priori constraints and account for the incompressibility and viscosity of the fluid. The method exhibits greatly (~70%) improved results over our recently published baseline with two separate steps for 3D reconstruction and motion estimation. Our results with only two time steps are comparable to those of sota tracking-based methods that require much longer sequences.Comment: To appear in International Journal of Computer Vision (IJCV

Variational Fluid Motion Estimation with Physical Priors

In this thesis, techniques for Particle Image Velocimetry (PIV) and Particle Tracking Velocimetry (PTV) are developed that are based on variational methods. The basic idea is not to estimate displacement vectors locally and individually, but to estimate vector fields as a whole by minimizing a suitable functional defined over the entire image domain (which may be 2D or 3D and may also include the temporal dimension). Such functionals typically comprise two terms: a data-term measuring how well two images of a sequence match as a function of the vector field to be estimated, and a regularization term that brings prior knowledge into the energy functional. Our starting point are methods that were originally developed in the field of computer vision and that we modify for the purpose of PIV. These methods are based on the so-called optical flow: Optical flow denotes the estimated velocity vector inferred by a relative motion of camera and image scene and is based on the assumption of gray value conservation (i.e. the total derivative of the image gray value over time is zero). A regularization term (that demands e.g. smoothness of the velocity field, or of its divergence and rotation) renders the system mathematically well-posed. Experimental evaluation shows that this type of variational approach is able to outperform standard cross-correlation methods. In order to develop a variational method for PTV, we replace the continuous data term of variational approaches to PIV with a discrete non-differentiable particle matching term. This raises the problem of minimizing such data terms together with continuous regularization terms. We accomplish this with an advanced mathematical method, which guarantees convergence to a local minimum of such a non-convex variational approach to PTV. With this novel variational approach (there has been no previous work on modeling PTV methods with global variational approaches), we achieve results for image pairs and sequences in two and three dimensions that outperform the relaxation methods that are traditionally used for particle tracking. The key advantage of our variational particle image velocimetry methods, is the chance to include prior knowledge in a natural way. In the fluid environments that we are considering in this thesis, it is especially attractive to use priors that can be motivated from a physical point of view. Firstly, we present a method that only allows flow fields that satisfy the Stokes equation. The latter equation includes control variables that allow to control the optical flow so as to fit the apparent velocities of particles in a given image pair. Secondly, we present a variational approach to motion estimation of instationary fluid flows. This approach extends the prior method along two directions: (i) The full incompressible Navier-Stokes equation is employed in order to obtain a physically consistent regularization which does not suppress turbulent flow variations. (ii) Regularization along the time-axis is employed as well, but formulated in a receding horizon manner contrary to previous approaches to spatio-temporal regularization. Ground-truth evaluations for simulated turbulent flows demonstrate that the accuracy of both types of physically plausible regularization compares favorably with advanced cross-correlation approaches. Furthermore, the direct estimation of, e.g., pressure or vorticity becomes possible

Power laws and inverse motion modeling: application to turbulence measurements from satellite images

International audienceIn the context of tackling the ill-posed inverse problem of motion estimation from image sequences, we propose to introduce prior knowledge on flow regularity given by turbulence statistical models. Prior regularity is formalized using turbulence power laws describing statistically self-similar structure of motion increments across scales. The motion estimation method minimizes the error of an image observation model while constraining second order structure function to behave as a power law within a prescribed range. Thanks to a Bayesian modeling framework, the motion estimation method is able to jointly infer the most likely power law directly from image data. The method is assessed on velocity fields of 2D or quasi-2D flows. Estimation accuracy is first evaluated on a synthetic image sequence of homogeneous and isotropic 2D turbulence. Results obtained with the approach based on physics of fluids outperforms state-of-the-art. Then, the method analyzes atmospheric turbulence using a real meteorological image sequence. Selecting the most likely power law model enables the recovery of physical quantities which are of major interest for turbulence atmospheric characterization. In particular, from meteorological images we are able to estimate energy and enstrophy fluxes of turbulent cascades, which are in agreement with previous in situ measurements

Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants

Recent progress in autoencoder-based sparse identification of nonlinear dynamics (SINDy) under $\ell_1$ constraints allows joint discoveries of governing equations and latent coordinate systems from spatio-temporal data, including simulated video frames. However, it is challenging for $\ell_1$-based sparse inference to perform correct identification for real data due to the noisy measurements and often limited sample sizes. To address the data-driven discovery of physics in the low-data and high-noise regimes, we propose Bayesian SINDy autoencoders, which incorporate a hierarchical Bayesian sparsifying prior: Spike-and-slab Gaussian Lasso. Bayesian SINDy autoencoder enables the joint discovery of governing equations and coordinate systems with a theoretically guaranteed uncertainty estimate. To resolve the challenging computational tractability of the Bayesian hierarchical setting, we adapt an adaptive empirical Bayesian method with Stochatic gradient Langevin dynamics (SGLD) which gives a computationally tractable way of Bayesian posterior sampling within our framework. Bayesian SINDy autoencoder achieves better physics discovery with lower data and fewer training epochs, along with valid uncertainty quantification suggested by the experimental studies. The Bayesian SINDy autoencoder can be applied to real video data, with accurate physics discovery which correctly identifies the governing equation and provides a close estimate for standard physics constants like gravity $g$, for example, in videos of a pendulum.Comment: 28 pages, 11 figure