Linear Asymptotic Convergence of Anderson Acceleration: Fixed-Point Analysis

We study the asymptotic convergence of AA($m$), i.e., Anderson acceleration with window size $m$ for accelerating fixed-point methods $x_{k+1}=q(x_{k})$, $x_k \in R^n$. Convergence acceleration by AA($m$) has been widely observed but is not well understood. We consider the case where the fixed-point iteration function $q(x)$ is differentiable and the convergence of the fixed-point method itself is root-linear. We identify numerically several conspicuous properties of AA($m$) convergence: First, AA($m$) sequences $\{x_k\}$ converge root-linearly but the root-linear convergence factor depends strongly on the initial condition. Second, the AA($m$) acceleration coefficients $\beta^{(k)}$ do not converge but oscillate as $\{x_k\}$ converges to $x^*$. To shed light on these observations, we write the AA($m$) iteration as an augmented fixed-point iteration $z_{k+1} =\Psi(z_k)$, $z_k \in R^{n(m+1)}$ and analyze the continuity and differentiability properties of $\Psi(z)$ and $\beta(z)$. We find that the vector of acceleration coefficients $\beta(z)$ is not continuous at the fixed point $z^*$. However, we show that, despite the discontinuity of $\beta(z)$, the iteration function $\Psi(z)$ is Lipschitz continuous and directionally differentiable at $z^*$ for AA(1), and we generalize this to AA($m$) with $m>1$ for most cases. Furthermore, we find that $\Psi(z)$ is not differentiable at $z^*$. We then discuss how these theoretical findings relate to the observed convergence behaviour of AA($m$). The discontinuity of $\beta(z)$ at $z^*$ allows $\beta^{(k)}$ to oscillate as $\{x_k\}$ converges to $x^*$, and the non-differentiability of $\Psi(z)$ allows AA($m$) sequences to converge with root-linear convergence factors that strongly depend on the initial condition. Additional numerical results illustrate our findings

Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

We address the problem of video representation learning without human-annotated labels. While previous efforts address the problem by designing novel self-supervised tasks using video data, the learned features are merely on a frame-by-frame basis, which are not applicable to many video analytic tasks where spatio-temporal features are prevailing. In this paper we propose a novel self-supervised approach to learn spatio-temporal features for video representation. Inspired by the success of two-stream approaches in video classification, we propose to learn visual features by regressing both motion and appearance statistics along spatial and temporal dimensions, given only the input video data. Specifically, we extract statistical concepts (fast-motion region and the corresponding dominant direction, spatio-temporal color diversity, dominant color, etc.) from simple patterns in both spatial and temporal domains. Unlike prior puzzles that are even hard for humans to solve, the proposed approach is consistent with human inherent visual habits and therefore easy to answer. We conduct extensive experiments with C3D to validate the effectiveness of our proposed approach. The experiments show that our approach can significantly improve the performance of C3D when applied to video classification tasks. Code is available at https://github.com/laura-wang/video_repres_mas.Comment: CVPR 201

Local Fourier analysis for saddle-point problems

The numerical solution of saddle-point problems has attracted considerable interest in recent years, due to their indefiniteness and often poor spectral properties that make efficient solution difficult. While much research already exists, developing efficient algorithms remains challenging. Researchers have applied finite-difference, finite element, and finite-volume approaches successfully to discretize saddle-point problems, and block preconditioners and monolithic multigrid methods have been proposed for the resulting systems. However, there is still much to understand. Magnetohydrodynamics (MHD) models the flow of a charged fluid, or plasma, in the presence of electromagnetic fields. Often, the discretization and linearization of MHD leads to a saddle-point system. We present vector-potential formulations of MHD and a theoretical analysis of the existence and uniqueness of solutions of both the continuum two-dimensional resistive MHD model and its discretization. Local Fourier analysis (LFA) is a commonly used tool for the analysis of multigrid and other multilevel algorithms. We first adapt LFA to analyse the properties of multigrid methods for both finite-difference and finite-element discretizations of the Stokes equations, leading to saddle-point systems. Monolithic multigrid methods, based on distributive, Braess-Sarazin, and Uzawa relaxation are discussed. From this LFA, optimal parameters are proposed for these multigrid solvers. Numerical experiments are presented to validate our theoretical results. A modified two-level LFA is proposed for high-order finite-element methods for the Lapalce problem, curing the failure of classical LFA smoothing analysis in this setting and providing a reliable way to estimate actual multigrid performance. Finally, we extend LFA to analyze the balancing domain decomposition by constraints (BDDC) algorithm, using a new choice of basis for the space of Fourier harmonics that greatly simplifies the application of LFA. Improved performance is obtained for some two- and three-level variants