On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

The proximal point algorithm (PPA) has been well studied in the literature. In particular, its linear convergence rate has been studied by Rockafellar in 1976 under certain condition. We consider a generalized PPA in the generic setting of finding a zero point of a maximal monotone operator, and show that the condition proposed by Rockafellar can also sufficiently ensure the linear convergence rate for this generalized PPA. Indeed we show that these linear convergence rates are optimal. Both the exact and inexact versions of this generalized PPA are discussed. The motivation to consider this generalized PPA is that it includes as special cases the relaxed versions of some splitting methods that are originated from PPA. Thus, linear convergence results of this generalized PPA can be used to better understand the convergence of some widely used algorithms in the literature. We focus on the particular convex minimization context and specify Rockafellar's condition to see how to ensure the linear convergence rate for some efficient numerical schemes, including the classical augmented Lagrangian method proposed by Hensen and Powell in 1969 and its relaxed version, the original alternating direction method of multipliers (ADMM) by Glowinski and Marrocco in 1975 and its relaxed version (i.e., the generalized ADMM by Eckstein and Bertsekas in 1992). Some refined conditions weaker than existing ones are proposed in these particular contexts.Comment: 22 pages, 1 figur

Splitting Methods in Image Processing

It is often necessary to restore digital images which are affected by noise (denoising), blur (deblurring), or missing data (inpainting). We focus here on variational methods, i.e., the restored image is the minimizer of an energy functional. The first part of this thesis deals with the algorithmic framework of how to compute such a minimizer. It turns out that operator splitting methods are very useful in image processing to derive fast algorithms. The idea is that, in general, the functional we want to minimize has an additive structure and we treat its summands separately in each iteration of the algorithm which yields subproblems that are easier to solve. In our applications, these are typically projections onto simple sets, fast shrinkage operations, and linear systems of equations with a nice structure. The two operator splitting methods we focus on here are the forward-backward splitting algorithm and the Douglas-Rachford splitting algorithm. We show based on older results that the recently proposed alternating split Bregman algorithm is equivalent to the Douglas-Rachford splitting method applied to the dual problem, or, equivalently, to the alternating direction method of multipliers. Moreover, it is illustrated how this algorithm allows us to decouple functionals which are sums of more than two terms. In the second part, we apply the above techniques to existing and new image restoration models. For the Rudin-Osher-Fatemi model, which is well suited to remove Gaussian noise, the following topics are considered: we avoid the staircasing effect by using an additional gradient fitting term or by combining first- and second-order derivatives via an infimal-convolution functional. For a special setting based on Parseval frames, a strong connection between the forward-backward splitting algorithm, the alternating split Bregman method and iterated frame shrinkage is shown. Furthermore, the good performance of the alternating split Bregman algorithm compared to the popular multistep methods is illustrated. A special emphasis lies here on the choice of the step-length parameter. Turning to a corresponding model for removing Poisson noise, we show the advantages of the alternating split Bregman algorithm in the decoupling of more complicated functionals. For the inpainting problem, we improve an existing wavelet-based method by incorporating anisotropic regularization techniques to better restore boundaries in an image. The resulting algorithm is characterized as a forward-backward splitting method. Finally, we consider the denoising of a more general form of images, namely, tensor-valued images where a matrix is assigned to each pixel. This type of data arises in many important applications such as diffusion-tensor MRI

A Behavioral Approach to Robust Machine Learning

Machine learning is revolutionizing almost all fields of science and technology and has been proposed as a pathway to solving many previously intractable problems such as autonomous driving and other complex robotics tasks. While the field has demonstrated impressive results on certain problems, many of these results have not translated to applications in physical systems, partly due to the cost of system fail- ure and partly due to the difficulty of ensuring reliable and robust model behavior. Deep neural networks, for instance, have simultaneously demonstrated both incredible performance in game playing and image processing, and remarkable fragility. This combination of high average performance and a catastrophically bad worst case performance presents a serious danger as deep neural networks are currently being used in safety critical tasks such as assisted driving. In this thesis, we propose a new approach to training models that have built in robustness guarantees. Our approach to ensuring stability and robustness of the models trained is distinct from prior methods; where prior methods learn a model and then attempt to verify robustness/stability, we directly optimize over sets of models where the necessary properties are known to hold. Specifically, we apply methods from robust and nonlinear control to the analysis and synthesis of recurrent neural networks, equilibrium neural networks, and recurrent equilibrium neural networks. The techniques developed allow us to enforce properties such as incremental stability, incremental passivity, and incremental l2 gain bounds / Lipschitz bounds. A central consideration in the development of our model sets is the difficulty of fitting models. All models can be placed in the image of a convex set, or even R^N , allowing useful properties to be easily imposed during the training procedure via simple interior point methods, penalty methods, or unconstrained optimization. In the final chapter, we study the problem of learning networks of interacting models with guarantees that the resulting networked system is stable and/or monotone, i.e., the order relations between states are preserved. While our approach to learning in this chapter is similar to the previous chapters, the model set that we propose has a separable structure that allows for the scalable and distributed identification of large-scale systems via the alternating directions method of multipliers (ADMM)