Better and Simpler Error Analysis of the Sinkhorn-Knopp Algorithm for Matrix Scaling

Given a non-negative real matrix A, the matrix scaling problem is to determine if it is possible to scale the rows and columns so that each row and each column sums to a specified target value for it. The matrix scaling problem arises in many algorithmic applications, perhaps most notably as a preconditioning step in solving linear system of equations. One of the most natural and by now classical approach to matrix scaling is the Sinkhorn-Knopp algorithm (also known as the RAS method) where one alternately scales either all rows or all columns to meet the target values. In addition to being extremely simple and natural, another appeal of this procedure is that it easily lends itself to parallelization. A central question is to understand the rate of convergence of the Sinkhorn-Knopp algorithm. Specifically, given a suitable error metric to measure deviations from target values, and an error bound epsilon, how quickly does the Sinkhorn-Knopp algorithm converge to an error below epsilon? While there are several non-trivial convergence results known about the Sinkhorn-Knopp algorithm, perhaps somewhat surprisingly, even for natural error metrics such as ell_1-error or ell_2-error, this is not entirely understood. In this paper, we present an elementary convergence analysis for the Sinkhorn-Knopp algorithm that improves upon the previous best bound. In a nutshell, our approach is to show (i) a simple bound on the number of iterations needed so that the KL-divergence between the current row-sums and the target row-sums drops below a specified threshold delta, and (ii) then show that for a suitable choice of delta, whenever KL-divergence is below delta, then the ell_1-error or the ell_2-error is below epsilon. The well-known Pinsker\u27s inequality immediately allows us to translate a bound on the KL divergence to a bound on ell_1-error. To bound the ell_2-error in terms of the KL-divergence, we establish a new inequality, referred to as (KL vs ell_1/ell_2) inequality in the paper. This new inequality is a strengthening of the Pinsker\u27s inequality that we believe is of independent interest. Our analysis of ell_2-error significantly improves upon the best previous convergence bound for ell_2-error. The idea of studying Sinkhorn-Knopp convergence via KL-divergence is not new and has indeed been previously explored. Our contribution is an elementary, self-contained presentation of this approach and an interesting new inequality that yields a significantly stronger convergence guarantee for the extensively studied ell_2-error

Efficient and Accurate Optimal Transport with Mirror Descent and Conjugate Gradients

We design a novel algorithm for optimal transport by drawing from the entropic optimal transport, mirror descent and conjugate gradients literatures. Our scalable and GPU parallelizable algorithm is able to compute the Wasserstein distance with extreme precision, reaching relative error rates of $10^{-8}$ without numerical stability issues. Empirically, the algorithm converges to high precision solutions more quickly in terms of wall-clock time than a variety of algorithms including log-domain stabilized Sinkhorn's Algorithm. We provide careful ablations with respect to algorithm and problem parameters, and present benchmarking over upsampled MNIST images, comparing to various recent algorithms over high-dimensional problems. The results suggest that our algorithm can be a useful addition to the practitioner's optimal transport toolkit

Development and evaluation of methods for control and modelling of multiple-input multiple-output systems

In control, a common type of system is the multiple-input multiple-output (MIMO) system, where the same input may affect multiple outputs, or conversely, the same output is affected by multiple inputs. In this thesis two methods for controlling MIMO systems are examined, namely linear quadratic Gaussian (LQG) control and decentralized control, and some of the difficulties associated with them.One difficulty when implementing decentralized control is to decide which inputs should control which outputs, also called the input-output pairing problem. There are multiple ways to solve this problem, among them using gramian based measures, which include the Hankel interaction index array, the participation matrix and the Σ2 method.\ua0 These methods take into account system dynamics as opposed to many other methods which only consider the steady-state system. However, the gramian based methods have issues with input and output scaling. Generally, this is handled by scaling all inputs and outputs to have equal range. However, in this thesis it is demonstrated how this can cause an incorrect pairing. Furthermore, this thesis examines other methods of scaling the gramian based measures, using either row or column sums, or by utilizing the Sinkhorn-Knopp algorithm. It is shown that there are considerable benefits to be gained from the alternative scaling of the gramian based measures, especially when using the Sinkhorn-Knopp algorithm. The use of this method also has the advantage that the results are completely independent of the original scaling of the inputs and outputs.An expansion to the decentralized control structure is the sparse control, in which a decentralized controller is expanded to include feed-forward or MIMO blocks. In this thesis we explore how to best use the gramian based measures to find sparse control structures, and propose a method which demonstrates considerable improvement compared to existing methods of sparse control structure design.A prerequisite to implementing control configuration methods is an understanding of the processes in question. In this thesis we examine the pulp refining process and design both static and dynamic models for pulp and paper properties such as shives width, fiber length and tensile index, and various available inputs. We demonstrate that utilizing internal variables (primarily consistencies) estimated from temperature measurements yields improved results compared to using solely measured variables. The measurement data from the refiners is noisy, sometimes sparse and generally irregularly sampled. This thesis discusses the challenges posed by these constraints and how they can be resolved.\ua0\ua0 An alternative way to control a MIMO system is to implement an LQG controller, which yields a single control structure for the entire system using a state based controller. It has been proposed that LQG control can be an effective control scheme to be used on networked control systems with wireless channels. These channels have a tendency to be unreliable with packet delays and packet losses. This thesis examines how to implement an LQG controller over such unreliable communication channels, and derives the optimal controller minimizing the cost function expressed in actuated controls.When new methods of control system design and analysis are introduced in the control engineering field, it is important to compare the new results with existing methods. Often this requires application of the methods on examples, and for this purpose benchmark processes are introduced. However, in many areas of control engineering research the number of examples are relatively few, in particular when MIMO systems are considered. For a thorough assessment of a method, however, as large number of relevant models as possible should be used. As a remedy, a framework has been developed for generating linear MIMO models based on predefined system properties, such as model type, size, stability, time constants, delays etc. This MIMO generator, which is presented in this thesis, is demonstrated by using it to evaluate the previously described scaling methods for the gramian based pairing methods

Approximating optimal transport with linear programs

In the regime of bounded transportation costs, additive approximations for the optimal transport problem are reduced (rather simply) to relative approximations for positive linear programs, resulting in faster additive approximation algorithms for optimal transport.Comment: To appear in SOSA 201

Approximating Optimal Transport With Linear Programs

In the regime of bounded transportation costs, additive approximations for the optimal transport problem are reduced (rather simply) to relative approximations for positive linear programs, resulting in faster additive approximation algorithms for optimal transport