1,205 research outputs found
Graph Kernels
We present a unified framework to study graph kernels, special cases of which include the random
walk (Gärtner et al., 2003; Borgwardt et al., 2005) and marginalized (Kashima et al., 2003, 2004;
Mahé et al., 2004) graph kernels. Through reduction to a Sylvester equation we improve the time
complexity of kernel computation between unlabeled graphs with n vertices from O(n^6) to O(n^3).
We find a spectral decomposition approach even more efficient when computing entire kernel matrices.
For labeled graphs we develop conjugate gradient and fixed-point methods that take O(dn^3)
time per iteration, where d is the size of the label set. By extending the necessary linear algebra to
Reproducing Kernel Hilbert Spaces (RKHS) we obtain the same result for d-dimensional edge kernels,
and O(n^4) in the infinite-dimensional case; on sparse graphs these algorithms only take O(n^2)
time per iteration in all cases. Experiments on graphs from bioinformatics and other application
domains show that these techniques can speed up computation of the kernel by an order of magnitude
or more. We also show that certain rational kernels (Cortes et al., 2002, 2003, 2004) when
specialized to graphs reduce to our random walk graph kernel. Finally, we relate our framework to
R-convolution kernels (Haussler, 1999) and provide a kernel that is close to the optimal assignment
kernel of Fröhlich et al. (2006) yet provably positive semi-definite
Euclidean Distance Matrices: Essential Theory, Algorithms and Applications
Euclidean distance matrices (EDM) are matrices of squared distances between
points. The definition is deceivingly simple: thanks to their many useful
properties they have found applications in psychometrics, crystallography,
machine learning, wireless sensor networks, acoustics, and more. Despite the
usefulness of EDMs, they seem to be insufficiently known in the signal
processing community. Our goal is to rectify this mishap in a concise tutorial.
We review the fundamental properties of EDMs, such as rank or
(non)definiteness. We show how various EDM properties can be used to design
algorithms for completing and denoising distance data. Along the way, we
demonstrate applications to microphone position calibration, ultrasound
tomography, room reconstruction from echoes and phase retrieval. By spelling
out the essential algorithms, we hope to fast-track the readers in applying
EDMs to their own problems. Matlab code for all the described algorithms, and
to generate the figures in the paper, is available online. Finally, we suggest
directions for further research.Comment: - 17 pages, 12 figures, to appear in IEEE Signal Processing Magazine
- change of title in the last revisio
Self-Supervised Learning with Lie Symmetries for Partial Differential Equations
Machine learning for differential equations paves the way for computationally
efficient alternatives to numerical solvers, with potentially broad impacts in
science and engineering. Though current algorithms typically require simulated
training data tailored to a given setting, one may instead wish to learn useful
information from heterogeneous sources, or from real dynamical systems
observations that are messy or incomplete. In this work, we learn
general-purpose representations of PDEs from heterogeneous data by implementing
joint embedding methods for self-supervised learning (SSL), a framework for
unsupervised representation learning that has had notable success in computer
vision. Our representation outperforms baseline approaches to invariant tasks,
such as regressing the coefficients of a PDE, while also improving the
time-stepping performance of neural solvers. We hope that our proposed
methodology will prove useful in the eventual development of general-purpose
foundation models for PDEs
Advances in Hyperspectral Image Classification: Earth monitoring with statistical learning methods
Hyperspectral images show similar statistical properties to natural grayscale
or color photographic images. However, the classification of hyperspectral
images is more challenging because of the very high dimensionality of the
pixels and the small number of labeled examples typically available for
learning. These peculiarities lead to particular signal processing problems,
mainly characterized by indetermination and complex manifolds. The framework of
statistical learning has gained popularity in the last decade. New methods have
been presented to account for the spatial homogeneity of images, to include
user's interaction via active learning, to take advantage of the manifold
structure with semisupervised learning, to extract and encode invariances, or
to adapt classifiers and image representations to unseen yet similar scenes.
This tutuorial reviews the main advances for hyperspectral remote sensing image
classification through illustrative examples.Comment: IEEE Signal Processing Magazine, 201
Manifold Contrastive Learning with Variational Lie Group Operators
Self-supervised learning of deep neural networks has become a prevalent
paradigm for learning representations that transfer to a variety of downstream
tasks. Similar to proposed models of the ventral stream of biological vision,
it is observed that these networks lead to a separation of category manifolds
in the representations of the penultimate layer. Although this observation
matches the manifold hypothesis of representation learning, current
self-supervised approaches are limited in their ability to explicitly model
this manifold. Indeed, current approaches often only apply augmentations from a
pre-specified set of "positive pairs" during learning. In this work, we propose
a contrastive learning approach that directly models the latent manifold using
Lie group operators parameterized by coefficients with a sparsity-promoting
prior. A variational distribution over these coefficients provides a generative
model of the manifold, with samples which provide feature augmentations
applicable both during contrastive training and downstream tasks. Additionally,
learned coefficient distributions provide a quantification of which
transformations are most likely at each point on the manifold while preserving
identity. We demonstrate benefits in self-supervised benchmarks for image
datasets, as well as a downstream semi-supervised task. In the former case, we
demonstrate that the proposed methods can effectively apply manifold feature
augmentations and improve learning both with and without a projection head. In
the latter case, we demonstrate that feature augmentations sampled from learned
Lie group operators can improve classification performance when using few
labels
- …