99 research outputs found
Topics in Matrix Sampling Algorithms
We study three fundamental problems of Linear Algebra, lying in the heart of
various Machine Learning applications, namely: 1)"Low-rank Column-based Matrix
Approximation". We are given a matrix A and a target rank k. The goal is to
select a subset of columns of A and, by using only these columns, compute a
rank k approximation to A that is as good as the rank k approximation that
would have been obtained by using all the columns; 2) "Coreset Construction in
Least-Squares Regression". We are given a matrix A and a vector b. Consider the
(over-constrained) least-squares problem of minimizing ||Ax-b||, over all
vectors x in D. The domain D represents the constraints on the solution and can
be arbitrary. The goal is to select a subset of the rows of A and b and, by
using only these rows, find a solution vector that is as good as the solution
vector that would have been obtained by using all the rows; 3) "Feature
Selection in K-means Clustering". We are given a set of points described with
respect to a large number of features. The goal is to select a subset of the
features and, by using only this subset, obtain a k-partition of the points
that is as good as the partition that would have been obtained by using all the
features. We present novel algorithms for all three problems mentioned above.
Our results can be viewed as follow-up research to a line of work known as
"Matrix Sampling Algorithms". [Frieze, Kanna, Vempala, 1998] presented the
first such algorithm for the Low-rank Matrix Approximation problem. Since then,
such algorithms have been developed for several other problems, e.g. Graph
Sparsification and Linear Equation Solving. Our contributions to this line of
research are: (i) improved algorithms for Low-rank Matrix Approximation and
Regression (ii) algorithms for a new problem domain (K-means Clustering).Comment: PhD Thesis, 150 page
Data Summarizations for Scalable, Robust and Privacy-Aware Learning in High Dimensions
The advent of large-scale datasets has offered unprecedented amounts of information for building statistically powerful machines, but, at the same time, also introduced a remarkable computational challenge: how can we efficiently process massive data? This thesis presents a suite of data reduction methods that make learning algorithms scale on large datasets, via extracting a succinct model-specific representation that summarizes the
full data collection—a coreset. Our frameworks support by design datasets of arbitrary dimensionality, and can be used for general purpose Bayesian inference under real-world constraints, including privacy preservation and robustness to outliers, encompassing diverse uncertainty-aware data analysis tasks, such as density estimation, classification
and regression.
We motivate the necessity for novel data reduction techniques in the first place by developing a reidentification attack on coarsened representations of private behavioural data. Analysing longitudinal records of human mobility, we detect privacy-revealing structural patterns, that remain preserved in reduced graph representations of individuals’ information with manageable size. These unique patterns enable mounting linkage attacks via structural similarity computations on longitudinal mobility traces, revealing an overlooked, yet existing, privacy threat.
We then propose a scalable variational inference scheme for approximating posteriors on large datasets via learnable weighted pseudodata, termed pseudocoresets. We show that the use of pseudodata enables overcoming the constraints on minimum summary size for given approximation quality, that are imposed on all existing Bayesian coreset constructions due to data dimensionality. Moreover, it allows us to develop a scheme for pseudocoresets-based summarization that satisfies the standard framework of differential privacy by construction; in this way, we can release reduced size privacy-preserving representations for sensitive datasets that are amenable to arbitrary post-processing.
Subsequently, we consider summarizations for large-scale Bayesian inference in scenarios when observed datapoints depart from the statistical assumptions of our model. Using robust divergences, we develop a method for constructing coresets resilient to model misspecification. Crucially, this method is able to automatically discard outliers from the generated data summaries. Thus we deliver robustified scalable representations
for inference, that are suitable for applications involving contaminated and unreliable data sources.
We demonstrate the performance of proposed summarization techniques on multiple parametric statistical models, and diverse simulated and real-world datasets, from music genre features to hospital readmission records, considering a wide range of data dimensionalities.Nokia Bell Labs,
Lundgren Fund,
Darwin College, University of Cambridge
Department of Computer Science & Technology, University of Cambridg
Coresets for Clustering with General Assignment Constraints
Designing small-sized \emph{coresets}, which approximately preserve the costs
of the solutions for large datasets, has been an important research direction
for the past decade. We consider coreset construction for a variety of general
constrained clustering problems. We significantly extend and generalize the
results of a very recent paper (Braverman et al., FOCS'22), by demonstrating
that the idea of hierarchical uniform sampling (Chen, SICOMP'09; Braverman et
al., FOCS'22) can be applied to efficiently construct coresets for a very
general class of constrained clustering problems with general assignment
constraints, including capacity constraints on cluster centers, and assignment
structure constraints for data points (modeled by a convex body .
Our main theorem shows that a small-sized -coreset exists as long
as a complexity measure of the structure
constraint, and the \emph{covering exponent}
for metric space are bounded. The complexity measure
for convex body is the Lipschitz
constant of a certain transportation problem constrained in ,
called \emph{optimal assignment transportation problem}. We prove nontrivial
upper bounds of for various polytopes, including
the general matroid basis polytopes, and laminar matroid polytopes (with better
bound). As an application of our general theorem, we construct the first
coreset for the fault-tolerant clustering problem (with or without capacity
upper/lower bound) for the above metric spaces, in which the fault-tolerance
requirement is captured by a uniform matroid basis polytope
- …