Multi-Criteria Dimensionality Reduction with Applications to Fairness

Dimensionality reduction is a classical technique widely used for data analysis. One foundational instantiation is Principal Component Analysis (PCA), which minimizes the average reconstruction error. In this paper, we introduce the "multi-criteria dimensionality reduction" problem where we are given multiple objectives that need to be optimized simultaneously. As an application, our model captures several fairness criteria for dimensionality reduction such as our novel Fair-PCA problem and the Nash Social Welfare (NSW) problem. In Fair-PCA, the input data is divided into $k$ groups, and the goal is to find a single $d$-dimensional representation for all groups for which the minimum variance of any one group is maximized. In NSW, the goal is to maximize the product of the individual variances of the groups achieved by the common low-dimensional space. Our main result is an exact polynomial-time algorithm for the two-criterion dimensionality reduction problem when the two criteria are increasing concave functions. As an application of this result, we obtain a polynomial time algorithm for Fair-PCA for $k=2$ groups and a polynomial time algorithm for NSW objective for $k=2$ groups. We also give approximation algorithms for $k>2$. Our technical contribution in the above results is to prove new low-rank properties of extreme point solutions to semi-definite programs. We conclude with experiments indicating the effectiveness of algorithms based on extreme point solutions of semi-definite programs on several real-world data sets.Comment: The preliminary version appeared in NeurIPS2019. This version combines the motivation from "The Price of Fair PCA: One Extra Dimension" (NeurIPS 2018) by the same set of author, adds new a motivation, and introduces new heuristics and more experimental result

Algorithms and dimensionality reductions for continuous multifacility ordered median location problems

In this paper we propose a general methodology for solving a broad class of continuous, multifacility location problems, in any dimension and with $\ell_\tau$-norms proposing two different methodologies: 1) by a new second order cone mixed integer programming formulation and 2) by formulating a sequence of semidefinite programs that converges to the solution of the problem; each of these relaxed problems solvable with SDP solvers in polynomial time. We apply dimensionality reductions of the problems by sparsity and symmetry in order to be able to solve larger problems.Comment: 26 pages, 2 table

Iterative Universal Rigidity

A bar framework determined by a finite graph $G$ and configuration $\bf p$ in $d$ space is universally rigid if it is rigid in any ${\mathbb R}^D \supset {\mathbb R}^d$. We provide a characterization of universally rigidity for any graph $G$ and any configuration ${\bf p}$ in terms of a sequence of affine subsets of the space of configurations. This corresponds to a facial reduction process for closed finite dimensional convex cones.Comment: 41 pages, 12 figure

Semidefnite Relaxation Bounds for Indefinite Homogeneous Quadratic Optimization

In this paper we study the relationship between the optimal value of a homogeneous quadratic optimization problem and that of its Semidefinite Programming (SDP) relaxation. We consider two quadratic optimization models: (1) $\min \{x^* C x \mid x^* A_k x \ge 1, x\in\mathbb{F}^n, k=0,1,...,m\}$; and (2) $\max \{x^* C x \mid x^* A_k x \le 1, x\in\mathbb{F}^n, k=0,1,...,m\}$. If \emph{one} of $A_k$'s is indefinite while others and $C$ are positive semidefinite, we prove that the ratio between the optimal value of (1) and its SDP relaxation is upper bounded by $O(m^2)$ when $\mathbb{F}$ is the real line $\mathbb{R}$, and by $O(m)$ when $\mathbb{F}$ is the complex plane $\mathbb{C}$. This result is an extension of the recent work of Luo {\em et al.} \cite{LSTZ}. For (2), we show that the same ratio is bounded from below by $O(1/\log m)$ for both the real and complex case, whenever all but one of $A_k$'s are positive semidefinite while $C$ can be indefinite. This result improves the so-called approximate S-Lemma of Ben-Tal {\em et al.} \cite{BNR02}. We also consider (2) with multiple indefinite quadratic constraints and derive a general bound in terms of the problem data and the SDP solution. Throughout the paper, we present examples showing that all of our results are essentially tight

Rounding Lasserre SDPs using column selection and spectrum-based approximation schemes for graph partitioning and Quadratic IPs

We present an approximation scheme for minimizing certain Quadratic Integer Programming problems with positive semidefinite objective functions and global linear constraints. This framework includes well known graph problems such as Minimum graph bisection, Edge expansion, Sparsest Cut, and Small Set expansion, as well as the Unique Games problem. These problems are notorious for the existence of huge gaps between the known algorithmic results and NP-hardness results. Our algorithm is based on rounding semidefinite programs from the Lasserre hierarchy, and the analysis uses bounds for low-rank approximations of a matrix in Frobenius norm using columns of the matrix. For all the above graph problems, we give an algorithm running in time $n^{O(r/\epsilon^2)}$ with approximation ratio $\frac{1+\epsilon}{\min\{1,\lambda_r\}}$, where $\lambda_r$ is the $r$'th smallest eigenvalue of the normalized graph Laplacian $\mathcal{L}$. In the case of graph bisection and small set expansion, the number of vertices in the cut is within lower-order terms of the stipulated bound. Our results imply $(1+O(\epsilon))$ factor approximation in time $n^{O(r^\ast/\epsilon^2)}$ where is the number of eigenvalues of $\mathcal{L}$ smaller than $1-\epsilon$ (for variants of sparsest cut, $\lambda_{r^\ast} \ge \mathrm{OPT}/\epsilon$ also suffices, and as $\mathrm{OPT}$ is usually $o(1)$ on interesting instances of these problems, this requirement on $r^\ast$ is typically weaker). For Unique Games, we give a factor $(1+\frac{2+\epsilon}{\lambda_r})$ approximation for minimizing the number of unsatisfied constraints in $n^{O(r/\epsilon)}$ time, improving upon an earlier bound for solving Unique Games on expanders. We also give an algorithm for independent sets in graphs that performs well when the Laplacian does not have too many eigenvalues bigger than $1+o(1)$.Comment: This manuscript is a merged and definitive version of (Guruswami, Sinop: FOCS 2011) and (Guruswami, Sinop: SODA 2013), with a significantly revised presentation. arXiv admin note: substantial text overlap with arXiv:1104.474

Border Basis relaxation for polynomial optimization

A relaxation method based on border basis reduction which improves the efficiency of Lasserre's approach is proposed to compute the optimum of a polynomial function on a basic closed semi algebraic set. A new stopping criterion is given to detect when the relaxation sequence reaches the minimum, using a sparse flat extension criterion. We also provide a new algorithm to reconstruct a finite sum of weighted Dirac measures from a truncated sequence of moments, which can be applied to other sparse reconstruction problems. As an application, we obtain a new algorithm to compute zero-dimensional minimizer ideals and the minimizer points or zero-dimensional G-radical ideals. Experimentations show the impact of this new method on significant benchmarks.Comment: Accepted for publication in Journal of Symbolic Computatio

Robust Beamforming in Cache-Enabled Cloud Radio Access Networks

Popular content caching is expected to play a major role in efficiently reducing backhaul congestion and achieving user satisfaction in next generation mobile radio systems. Consider the downlink of a cache-enabled cloud radio access network (CRAN), where each cache-enabled base station (BS) is equipped with limited-size local cache storage. The central computing unit (cloud) is connected to the BSs via a limited capacity backhaul link and serves a set of single-antenna mobile users (MUs). This paper assumes that only imperfect channel state information (CSI) is available at the cloud. It focuses on the problem of minimizing the total network power and backhaul cost so as to determine the beamforming vector of each user across the network, the quantization noise covariance matrix, and the BS clustering subject to imperfect channel state information and fixed cache placement assumptions. The paper suggests solving such a difficult, non-convex optimization problem using the semidefinite relaxation (SDR). The paper then uses the $\ell_0$-norm approximation to provide a feasible, sub-optimal solution using the majorization-minimization (MM) approach. Simulation results particularly show how the cache-enabled network significantly improves the backhaul cost especially at high signal-to-interference-plus-noise ratio (SINR) values as compared to conventional cache-less CRANs.Comment: 7 pages, 3 figure

Sparse Semidefinite Programs with Guaranteed Near-Linear Time Complexity via Dualized Clique Tree Conversion

Clique tree conversion solves large-scale semidefinite programs by splitting an $n\times n$ matrix variable into up to $n$ smaller matrix variables, each representing a principal submatrix of up to $\omega\times\omega$. Its fundamental weakness is the need to introduce overlap constraints that enforce agreement between different matrix variables, because these can result in dense coupling. In this paper, we show that by dualizing the clique tree conversion, the coupling due to the overlap constraints is guaranteed to be sparse over dense blocks, with a block sparsity pattern that coincides with the adjacency matrix of a tree. We consider two classes of semidefinite programs with favorable sparsity patterns that encompass the MAXCUT and MAX $k$-CUT relaxations, the Lovasz Theta problem, and the AC optimal power flow relaxation. Assuming that $\omega\ll n$, we prove that the per-iteration cost of an interior-point method is linear $O(n)$ time and memory, so an $\epsilon$-accurate and $\epsilon$-feasible iterate is obtained after $O(\sqrt{n}\log(1/\epsilon))$ iterations in near-linear $O(n^{1.5}\log(1/\epsilon))$ time. We confirm our theoretical insights with numerical results on semidefinite programs as large as $n=13659$. (Supporting code at https://github.com/ryz-codes/dual_ctc )Comment: [v1] appeared in IEEE CDC 2018; [v2+] To appear in Mathematical Programmin

Phase Retrieval using Alternating Minimization

Phase retrieval problems involve solving linear equations, but with missing sign (or phase, for complex numbers) information. More than four decades after it was first proposed, the seminal error reduction algorithm of (Gerchberg and Saxton 1972) and (Fienup 1982) is still the popular choice for solving many variants of this problem. The algorithm is based on alternating minimization; i.e. it alternates between estimating the missing phase information, and the candidate solution. Despite its wide usage in practice, no global convergence guarantees for this algorithm are known. In this paper, we show that a (resampling) variant of this approach converges geometrically to the solution of one such problem -- finding a vector $\mathbf{x}$ from $\mathbf{y},\mathbf{A}$, where $\mathbf{y} = \left|\mathbf{A}^{\top}\mathbf{x}\right|$ and $|\mathbf{z}|$ denotes a vector of element-wise magnitudes of $\mathbf{z}$ -- under the assumption that $\mathbf{A}$ is Gaussian. Empirically, we demonstrate that alternating minimization performs similar to recently proposed convex techniques for this problem (which are based on "lifting" to a convex matrix problem) in sample complexity and robustness to noise. However, it is much more efficient and can scale to large problems. Analytically, for a resampling version of alternating minimization, we show geometric convergence to the solution, and sample complexity that is off by log factors from obvious lower bounds. We also establish close to optimal scaling for the case when the unknown vector is sparse. Our work represents the first theoretical guarantee for alternating minimization (albeit with resampling) for any variant of phase retrieval problems in the non-convex setting.Comment: Accepted for publication in IEEE Transactions on Signal Processin

Improved Canonical Dual Algorithms for the Maxcut Problem

By introducing a quadratic perturbation to the canonical dual of the maxcut problem, we transform the integer programming problem into a concave maximization problem over a convex positive domain under some circumstances, which can be solved easily by the well-developed optimization methods. Considering that there may exist no critical points in the dual feasible domain, a reduction technique is used gradually to guarantee the feasibility of the reduced solution, and a compensation technique is utilized to strengthen the robustness of the solution. The similar strategy is also applied to the maxcut problem with linear perturbation and its hybrid with quadratic perturbation. Experimental results demonstrate the effectiveness of the proposed algorithms when compared with other approaches