50 research outputs found
Large-scale Binary Quadratic Optimization Using Semidefinite Relaxation and Applications
In computer vision, many problems such as image segmentation, pixel
labelling, and scene parsing can be formulated as binary quadratic programs
(BQPs). For submodular problems, cuts based methods can be employed to
efficiently solve large-scale problems. However, general nonsubmodular problems
are significantly more challenging to solve. Finding a solution when the
problem is of large size to be of practical interest, however, typically
requires relaxation. Two standard relaxation methods are widely used for
solving general BQPs--spectral methods and semidefinite programming (SDP), each
with their own advantages and disadvantages. Spectral relaxation is simple and
easy to implement, but its bound is loose. Semidefinite relaxation has a
tighter bound, but its computational complexity is high, especially for large
scale problems. In this work, we present a new SDP formulation for BQPs, with
two desirable properties. First, it has a similar relaxation bound to
conventional SDP formulations. Second, compared with conventional SDP methods,
the new SDP formulation leads to a significantly more efficient and scalable
dual optimization approach, which has the same degree of complexity as spectral
methods. We then propose two solvers, namely, quasi-Newton and smoothing Newton
methods, for the dual problem. Both of them are significantly more efficiently
than standard interior-point methods. In practice, the smoothing Newton solver
is faster than the quasi-Newton solver for dense or medium-sized problems,
while the quasi-Newton solver is preferable for large sparse/structured
problems. Our experiments on a few computer vision applications including
clustering, image segmentation, co-segmentation and registration show the
potential of our SDP formulation for solving large-scale BQPs.Comment: Fixed some typos. 18 pages. Accepted to IEEE Transactions on Pattern
Analysis and Machine Intelligenc
Understanding the Generalization Performance of Spectral Clustering Algorithms
The theoretical analysis of spectral clustering mainly focuses on
consistency, while there is relatively little research on its generalization
performance. In this paper, we study the excess risk bounds of the popular
spectral clustering algorithms: \emph{relaxed} RatioCut and \emph{relaxed}
NCut. Firstly, we show that their excess risk bounds between the empirical
continuous optimal solution and the population-level continuous optimal
solution have a convergence rate, where is the
sample size. Secondly, we show the fundamental quantity in influencing the
excess risk between the empirical discrete optimal solution and the
population-level discrete optimal solution. At the empirical level, algorithms
can be designed to reduce this quantity. Based on our theoretical analysis, we
propose two novel algorithms that can not only penalize this quantity, but also
cluster the out-of-sample data without re-eigendecomposition on the overall
sample. Experiments verify the effectiveness of the proposed algorithms
Diffusion, methods and applications
Tesis doctoral inédita leída en la Universidad Autónoma de Madrid, Escuela Politécnica Superior, Departamento de Ingeniería Informática. Fecha de lectura: junio de 2014Big Data, an important problem nowadays, can be understood in terms of a very large number of
patterns, a very large pattern dimension or, often, both. In this thesis, we will concentrate on the
high dimensionality issue, applying manifold learning techniques for visualizing and analyzing
such patterns.
The core technique will be Di usion Maps (DM) and its Anisotropic Di usion (AD) version,
introduced by Ronald R. Coifman and his school at Yale University, and of which we will give
a complete, systematic, compact and self-contained treatment. This will be done after a brief
survey of previous manifold learning methods.
The algorithmic contributions of the thesis will be centered in two computational challenges of
di usion methods: the potential high cost of the similarity matrix eigenanalysis that is needed
to define the di usion embedding coordinates, and the di culty of computing this embedding
over new patterns not available for the initial eigenanalysis. With respect to the first issue, we
will show how the AD set up can be used to skip it when looking for local models. In this case,
local patterns will be selected through a k-Nearest Neighbors search using a properly defined
local Mahalanobis distance, that enables neighbors to be found over the latent variable space
underlying the AD model while we can work directly with the observable patterns and, thus,
avoiding the potentially costly similarity matrix eigenanalysis.
The second proposed algorithm, that we will call Auto-adaptative Laplacian Pyramids (ALP),
focuses in the out-of-sample embedding extension and consists in a modification of the classical
Laplacian Pyramids (LP) method. In this new algorithm the LP iterations will be combined with
an estimate of the Leave One Out CV error, something that makes possible to directly define
during training a criterion to estimate the optimal stopping point of this iterative algorithm.
This thesis will also present several application contributions to important problems in renewable
energy and medical imaging. More precisely, we will show how DM is a good method
for dimensionality reduction of meteorological weather predictions, providing tools to visualize
and describe these data, as well as to cluster them in order to define local models.
In turn, we will apply our AD-based localized search method first to find the location in the
human body of CT scan images and then to predict wind energy ramps on both individual farms
and over the whole of Spain. We will see that, in both cases, our results improve on the current
state of the art methods.
Finally, we will compare our ALP proposal with the well-known Nyström method as well as
with LP on two large dimensional problems, the time compression of meteorological data and
the analysis of meteorological variables relevant in daily radiation forecasts. In both cases we
will show that ALP compares favorably with the other approaches for out-of-sample extension
problemsBig Data es un problema importante hoy en día, que puede ser entendido en términos de un
amplio número de patrones, una alta dimensión o, como sucede normalmente, de ambos. Esta
tesis se va a centrar en problemas de alta dimensión, aplicando técnicas de aprendizaje de
subvariedades para visualizar y analizar dichos patrones.
La técnica central será Di usion Maps (DM) y su versión anisotrópica, Anisotropic Di usion
(AD), introducida por Ronald R. Coifman y su escuela en la Universidad de Yale, la cual va a
ser tratada de manera completa, sistemática, compacta y auto-contenida. Esto se llevará a cabo
tras un breve repaso de métodos previos de aprendizaje de subvariedades.
Las contribuciones algorítmicas de esta tesis estarán centradas en dos de los grandes retos en
métodos de difusión: el potencial alto coste que tiene el análisis de autovalores de la matriz de
similitud, necesaria para definir las coordenadas embebidas; y la dificultad para calcular este
mismo embedding sobre nuevos datos que no eran accesibles cuando se realizó el análisis de
autovalores inicial. Respecto al primer tema, se mostrará cómo la aproximación AD se puede
utilizar para evitar el cálculo del embedding cuando estamos interesados en definir modelos locales.
En este caso, se seleccionarán patrones cercanos por medio de una búsqueda de vecinos
próximos (k-NN), usando como distancia una medida de Mahalanobis local que permita encontrar
vecinos sobre las variables latentes existentes bajo el modelo de AD. Todo esto se llevará
a cabo trabajando directamente sobre los patrones observables y, por tanto, evitando el costoso
cálculo que supone el cálculo de autovalores de la matriz de similitud.
El segundo algoritmo propuesto, que llamaremos Auto-adaptative Laplacian Pyramids (ALP),
se centra en la extensión del embedding para datos fuera de la muestra, y se trata de una modificación
del método denominado Laplacian Pyramids (LP). En este nuevo algoritmo, las iteraciones
de LP se combinarán con una estimación del error de Leave One Out CV, permitiendo definir
directamente durante el periodo de entrenamiento, un criterio para estimar el criterio de parada
óptimo para este método iterativo.
En esta tesis se presentarán también una serie de contribuciones de aplicación de estas técnicas
a importantes problemas en energías renovables e imágenes médicas. Más concretamente, se
muestra como DM es un buen método para reducir la dimensión de predicciones del tiempo
meteorológico, sirviendo por tanto de herramienta de visualización y descripción, así como de
clasificación de los datos con vistas a definir modelos locales sobre cada grupo descrito.
Posteriormente, se aplicará nuestro método de búsqueda localizada basado en AD tanto a la
búsqueda de la correspondiente posición de tomografías en el cuerpo humano, como para la
detección de rampas de energía eólica en parques individuales o de manera global en España.
En ambos casos se verá como los resultados obtenidos mejoran los métodos del estado del arte
actual.
Finalmente se comparará el algoritmo de ALP propuesto frente al conocido método de Nyström
y al método de LP, en dos problemas de alta dimensión: el problema de compresión temporal
de datos meteorológicos y el análisis de variables meteorológicas relevantes para la predicción
de la radiación diaria. En ambos casos se mostrará cómo ALP es comparativamente mejor que
otras aproximaciones existentes para resolver el problema de extensión del embedding a puntos
fuera de la muestr
HIGH PERFORMANCE SPECTRAL METHODS FOR GRAPH-BASED MACHINE LEARNING
Graphs play a critical role in machine learning and data mining fields. The success of graph-based machine learning algorithms highly depends on the quality of the underlying graphs. Desired graphs should have two characteristics: 1) they should be able to well-capture the underlying structures of the data sets. 2) they should be sparse enough so that the downstream algorithms can be performed efficiently on them.
This dissertation first studies the application of a two-phase spectrum-preserving spectral sparsification method that enables to construct very sparse sparsifiers with guaranteed preservation of original graph spectra for spectral clustering. Experiments show that the computational challenge due to the eigen-decomposition procedure in spectral clustering can be fundamentally addressed.
We then propose a highly-scalable spectral graph learning approach GRASPEL. GRASPEL can learn high-quality graphs from high dimensional input data. Compared with prior state-of-the-art graph learning and construction methods , GRASPEL leads to substantially improved algorithm performance
A Comprehensive Review of Community Detection in Graphs
The study of complex networks has significantly advanced our understanding of
community structures which serves as a crucial feature of real-world graphs.
Detecting communities in graphs is a challenging problem with applications in
sociology, biology, and computer science. Despite the efforts of an
interdisciplinary community of scientists, a satisfactory solution to this
problem has not yet been achieved. This review article delves into the topic of
community detection in graphs, which serves as a crucial role in understanding
the organization and functioning of complex systems. We begin by introducing
the concept of community structure, which refers to the arrangement of vertices
into clusters, with strong internal connections and weaker connections between
clusters. Then, we provide a thorough exposition of various community detection
methods, including a new method designed by us. Additionally, we explore
real-world applications of community detection in diverse networks. In
conclusion, this comprehensive review provides a deep understanding of
community detection in graphs. It serves as a valuable resource for researchers
and practitioners in multiple disciplines, offering insights into the
challenges, methodologies, and applications of community detection in complex
networks
New Approaches in Multi-View Clustering
Many real-world datasets can be naturally described by multiple views. Due to this, multi-view learning has drawn much attention from both academia and industry. Compared to single-view learning, multi-view learning has demonstrated plenty of advantages. Clustering has long been serving as a critical technique in data mining and machine learning. Recently, multi-view clustering has achieved great success in various applications. To provide a comprehensive review of the typical multi-view clustering methods and their corresponding recent developments, this chapter summarizes five kinds of popular clustering methods and their multi-view learning versions, which include k-means, spectral clustering, matrix factorization, tensor decomposition, and deep learning. These clustering methods are the most widely employed algorithms for single-view data, and lots of efforts have been devoted to extending them for multi-view clustering. Besides, many other multi-view clustering methods can be unified into the frameworks of these five methods. To promote further research and development of multi-view clustering, some popular and open datasets are summarized in two categories. Furthermore, several open issues that deserve more exploration are pointed out in the end
Unsupervised Spectral Ranking For Anomaly Detection
Anomaly detection is the problem of finding deviations from expected normal patterns. A wide variety of applications, such as fraud detection for credit cards and insurance, medical image monitoring, network intrusion detection, and military surveillance, can be viewed as anomaly detection. For anomaly detection, obtaining accurate labels, especially labels for anomalous cases, is costly and time consuming, if not
practically infeasible. This makes supervised anomaly detection less desirable in the domain of anomaly detection.
In this thesis, we propose a novel unsupervised spectral ranking method for anomaly detection (SRA). Based on the 1st non-principal eigenvectors from Laplacian matrices, the proposed SRA can generate anomaly ranking either with respect to a single majority class or with respect to multiple majority classes. The ranking type is based on whether the percentage of the smaller class instances (positive or negative) is larger than the expected upper bound of the anomaly ratio. We justify the proposed spectral ranking by establishing a connection between the unsupervised support vector machine optimization and the spectral Laplacian optimization problem. Using both synthetic and real data sets, we show that our proposed SRA is a meaningful and effective alternative to the state-of-art unsupervised anomaly ranking methods. In addition, we show that, in certain scenarios, unsupervised SRA method surpasses the state-of-art unsupervised anomaly ranking methods in terms of performance and robustness of parameter tuning. Finally, we demonstrate that choosing appropriate similarity measures remains crucial in applying our proposed SRA algorithm
Different approaches to community detection
A precise definition of what constitutes a community in networks has remained
elusive. Consequently, network scientists have compared community detection
algorithms on benchmark networks with a particular form of community structure
and classified them based on the mathematical techniques they employ. However,
this comparison can be misleading because apparent similarities in their
mathematical machinery can disguise different reasons for why we would want to
employ community detection in the first place. Here we provide a focused review
of these different motivations that underpin community detection. This
problem-driven classification is useful in applied network science, where it is
important to select an appropriate algorithm for the given purpose. Moreover,
highlighting the different approaches to community detection also delineates
the many lines of research and points out open directions and avenues for
future research.Comment: 14 pages, 2 figures. Written as a chapter for forthcoming Advances in
network clustering and blockmodeling, and based on an extended version of The
many facets of community detection in complex networks, Appl. Netw. Sci. 2: 4
(2017) by the same author