4,056 research outputs found
Scalable and Sustainable Deep Learning via Randomized Hashing
Current deep learning architectures are growing larger in order to learn from
complex datasets. These architectures require giant matrix multiplication
operations to train millions of parameters. Conversely, there is another
growing trend to bring deep learning to low-power, embedded devices. The matrix
operations, associated with both training and testing of deep networks, are
very expensive from a computational and energy standpoint. We present a novel
hashing based technique to drastically reduce the amount of computation needed
to train and test deep networks. Our approach combines recent ideas from
adaptive dropouts and randomized hashing for maximum inner product search to
select the nodes with the highest activation efficiently. Our new algorithm for
deep learning reduces the overall computational cost of forward and
back-propagation by operating on significantly fewer (sparse) nodes. As a
consequence, our algorithm uses only 5% of the total multiplications, while
keeping on average within 1% of the accuracy of the original model. A unique
property of the proposed hashing based back-propagation is that the updates are
always sparse. Due to the sparse gradient updates, our algorithm is ideally
suited for asynchronous and parallel training leading to near linear speedup
with increasing number of cores. We demonstrate the scalability and
sustainability (energy efficiency) of our proposed algorithm via rigorous
experimental evaluations on several real datasets
End-to-end Sampling Patterns
Sample patterns have many uses in Computer Graphics, ranging from procedural
object placement over Monte Carlo image synthesis to non-photorealistic
depiction. Their properties such as discrepancy, spectra, anisotropy, or
progressiveness have been analyzed extensively. However, designing methods to
produce sampling patterns with certain properties can require substantial
hand-crafting effort, both in coding, mathematical derivation and compute time.
In particular, there is no systematic way to derive the best sampling algorithm
for a specific end-task.
Tackling this issue, we suggest another level of abstraction: a toolkit to
end-to-end optimize over all sampling methods to find the one producing
user-prescribed properties such as discrepancy or a spectrum that best fit the
end-task. A user simply implements the forward losses and the sampling method
is found automatically -- without coding or mathematical derivation -- by
making use of back-propagation abilities of modern deep learning frameworks.
While this optimization takes long, at deployment time the sampling method is
quick to execute as iterated unstructured non-linear filtering using radial
basis functions (RBFs) to represent high-dimensional kernels. Several important
previous methods are special cases of this approach, which we compare to
previous work and demonstrate its usefulness in several typical Computer
Graphics applications. Finally, we propose sampling patterns with properties
not shown before, such as high-dimensional blue noise with projective
properties
Connectionist-Symbolic Machine Intelligence using Cellular Automata based Reservoir-Hyperdimensional Computing
We introduce a novel framework of reservoir computing, that is capable of
both connectionist machine intelligence and symbolic computation. Cellular
automaton is used as the reservoir of dynamical systems. Input is randomly
projected onto the initial conditions of automaton cells and nonlinear
computation is performed on the input via application of a rule in the
automaton for a period of time. The evolution of the automaton creates a
space-time volume of the automaton state space, and it is used as the
reservoir. The proposed framework is capable of long short-term memory and it
requires orders of magnitude less computation compared to Echo State Networks.
We prove that cellular automaton reservoir holds a distributed representation
of attribute statistics, which provides a more effective computation than local
representation. It is possible to estimate the kernel for linear cellular
automata via metric learning, that enables a much more efficient distance
computation in support vector machine framework. Also, binary reservoir feature
vectors can be combined using Boolean operations as in hyperdimensional
computing, paving a direct way for concept building and symbolic processing.Comment: Corrected Typos. Responded some comments on section 8. Added appendix
for details. Recurrent architecture emphasize
Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders
Multivariate time series (MTS) imputation is a widely studied problem in
recent years. Existing methods can be divided into two main groups, including
(1) deep recurrent or generative models that primarily focus on time series
features, and (2) graph neural networks (GNNs) based models that utilize the
topological information from the inherent graph structure of MTS as relational
inductive bias for imputation. Nevertheless, these methods either neglect
topological information or assume the graph structure is fixed and accurately
known. Thus, they fail to fully utilize the graph dynamics for precise
imputation in more challenging MTS data such as networked time series (NTS),
where the underlying graph is constantly changing and might have missing edges.
In this paper, we propose a novel approach to overcome these limitations.
First, we define the problem of imputation over NTS which contains missing
values in both node time series features and graph structures. Then, we design
a new model named PoGeVon which leverages variational autoencoder (VAE) to
predict missing values over both node time series features and graph
structures. In particular, we propose a new node position embedding based on
random walk with restart (RWR) in the encoder with provable higher expressive
power compared with message-passing based graph neural networks (GNNs). We
further design a decoder with 3-stage predictions from the perspective of
multi-task learning to impute missing values in both time series and graph
structures reciprocally. Experiment results demonstrate the effectiveness of
our model over baselines.Comment: KDD 202
Graph Interpolation via Fast Fused-Gromovization
Graph data augmentation has proven to be effective in enhancing the
generalizability and robustness of graph neural networks (GNNs) for graph-level
classifications. However, existing methods mainly focus on augmenting the graph
signal space and the graph structure space independently, overlooking their
joint interaction. This paper addresses this limitation by formulating the
problem as an optimal transport problem that aims to find an optimal strategy
for matching nodes between graphs considering the interactions between graph
structures and signals. To tackle this problem, we propose a novel graph mixup
algorithm dubbed FGWMixup, which leverages the Fused Gromov-Wasserstein (FGW)
metric space to identify a "midpoint" of the source graphs. To improve the
scalability of our approach, we introduce a relaxed FGW solver that accelerates
FGWMixup by enhancing the convergence rate from to
. Extensive experiments conducted on five datasets,
utilizing both classic (MPNNs) and advanced (Graphormers) GNN backbones,
demonstrate the effectiveness of FGWMixup in improving the generalizability and
robustness of GNNs
- …