Patterns of Scalable Bayesian Inference

Datasets are growing not just in size but in complexity, creating a demand for rich models and quantification of uncertainty. Bayesian methods are an excellent fit for this demand, but scaling Bayesian inference is a challenge. In response to this challenge, there has been considerable recent work based on varying assumptions about model structure, underlying computational resources, and the importance of asymptotic correctness. As a result, there is a zoo of ideas with few clear overarching principles. In this paper, we seek to identify unifying principles, patterns, and intuitions for scaling Bayesian inference. We review existing work on utilizing modern computing resources with both MCMC and variational approximation techniques. From this taxonomy of ideas, we characterize the general principles that have proven successful for designing scalable inference procedures and comment on the path forward

A Finite-Time Cutting Plane Algorithm for Distributed Mixed Integer Linear Programming

Many problems of interest for cyber-physical network systems can be formulated as Mixed Integer Linear Programs in which the constraints are distributed among the agents. In this paper we propose a distributed algorithm to solve this class of optimization problems in a peer-to-peer network with no coordinator and with limited computation and communication capabilities. In the proposed algorithm, at each communication round, agents solve locally a small LP, generate suitable cutting planes, namely intersection cuts and cost-based cuts, and communicate a fixed number of active constraints, i.e., a candidate optimal basis. We prove that, if the cost is integer, the algorithm converges to the lexicographically minimal optimal solution in a finite number of communication rounds. Finally, through numerical computations, we analyze the algorithm convergence as a function of the network size.Comment: 6 pages, 3 figure

ALADIN-α—An open-source MATLAB toolbox for distributed non-convex optimization

This article introduces an open-source software for distributed and decentralized non-convex optimization named ALADIN-α. ALADIN-α is a MATLAB implementation of tailored variants of the Augmented Lagrangian Alternating Direction Inexact Newton (ALADIN) algorithm. It is user interface is convenient for rapid prototyping of non-convex distributed optimization algorithms. An improved version of the recently proposed bi-level variant of ALADIN is included enabling decentralized non-convex optimization with reduced information exchange. A collection of examples from different applications fields including chemical engineering, robotics, and power systems underpins the potential of ALADIN-α

Privacy-Preserving Distributed Optimization via Subspace Perturbation: A General Framework

As the modern world becomes increasingly digitized and interconnected, distributed signal processing has proven to be effective in processing its large volume of data. However, a main challenge limiting the broad use of distributed signal processing techniques is the issue of privacy in handling sensitive data. To address this privacy issue, we propose a novel yet general subspace perturbation method for privacy-preserving distributed optimization, which allows each node to obtain the desired solution while protecting its private data. In particular, we show that the dual variables introduced in each distributed optimizer will not converge in a certain subspace determined by the graph topology. Additionally, the optimization variable is ensured to converge to the desired solution, because it is orthogonal to this non-convergent subspace. We therefore propose to insert noise in the non-convergent subspace through the dual variable such that the private data are protected, and the accuracy of the desired solution is completely unaffected. Moreover, the proposed method is shown to be secure under two widely-used adversary models: passive and eavesdropping. Furthermore, we consider several distributed optimizers such as ADMM and PDMM to demonstrate the general applicability of the proposed method. Finally, we test the performance through a set of applications. Numerical tests indicate that the proposed method is superior to existing methods in terms of several parameters like estimated accuracy, privacy level, communication cost and convergence rate

ALADIN-α\alpha -- An open-source MATLAB toolbox for distributed non-convex optimization

This paper introduces an open-source software for distributed and decentralized non-convex optimization named ALADIN-$\alpha$. ALADIN-$\alpha$ is a MATLAB implementation of the Augmented Lagrangian Alternating Direction Inexact Newton (ALADIN) algorithm, which is tailored towards rapid prototyping for non-convex distributed optimization. An improved version of the recently proposed bi-level variant of ALADIN is included enabling decentralized non-convex optimization. A collection of application examples from different applications fields including chemical engineering, robotics, and power systems underpins the application potential of ALADIN-$\alpha$