106,347 research outputs found
A tree-based kernel for graphs with continuous attributes
The availability of graph data with node attributes that can be either
discrete or real-valued is constantly increasing. While existing kernel methods
are effective techniques for dealing with graphs having discrete node labels,
their adaptation to non-discrete or continuous node attributes has been
limited, mainly for computational issues. Recently, a few kernels especially
tailored for this domain, and that trade predictive performance for
computational efficiency, have been proposed. In this paper, we propose a graph
kernel for complex and continuous nodes' attributes, whose features are tree
structures extracted from specific graph visits. The kernel manages to keep the
same complexity of state-of-the-art kernels while implicitly using a larger
feature space. We further present an approximated variant of the kernel which
reduces its complexity significantly. Experimental results obtained on six
real-world datasets show that the kernel is the best performing one on most of
them. Moreover, in most cases the approximated version reaches comparable
performances to current state-of-the-art kernels in terms of classification
accuracy while greatly shortening the running times.Comment: This work has been submitted to the IEEE Transactions on Neural
Networks and Learning Systems for possible publication. Copyright may be
transferred without notice, after which this version may no longer be
accessibl
The Computational Cost of Asynchronous Neural Communication
Biological neural computation is inherently asynchronous due to large variations in neuronal spike timing and transmission delays. So-far, most theoretical work on neural networks assumes the synchronous setting where neurons fire simultaneously in discrete rounds. In this work we aim at understanding the barriers of asynchronous neural computation from an algorithmic perspective. We consider an extension of the widely studied model of synchronized spiking neurons [Maass, Neural Networks 97] to the asynchronous setting by taking into account edge and node delays.
- Edge Delays: We define an asynchronous model for spiking neurons in which the latency values (i.e., transmission delays) of non self-loop edges vary adversarially over time. This extends the recent work of [Hitron and Parter, ESA\u2719] in which the latency values are restricted to be fixed over time. Our first contribution is an impossibility result that implies that the assumption that self-loop edges have no delays (as assumed in Hitron and Parter) is indeed necessary. Interestingly, in real biological networks self-loop edges (a.k.a. autapse) are indeed free of delays, and the latter has been noted by neuroscientists to be crucial for network synchronization.
To capture the computational challenges in this setting, we first consider the implementation of a single NOT gate. This simple function already captures the fundamental difficulties in the asynchronous setting. Our key technical results are space and time upper and lower bounds for the NOT function, our time bounds are tight. In the spirit of the distributed synchronizers [Awerbuch and Peleg, FOCS\u2790] and following [Hitron and Parter, ESA\u2719], we then provide a general synchronizer machinery. Our construction is very modular and it is based on efficient circuit implementation of threshold gates. The complexity of our scheme is measured by the overhead in the number of neurons and the computation time, both are shown to be polynomial in the largest latency value, and the largest incoming degree ? of the original network.
- Node Delays: We introduce the study of asynchronous communication due to variations in the response rates of the neurons in the network. In real brain networks, the round duration varies between different neurons in the network. Our key result is a simulation methodology that allows one to transform the above mentioned synchronized solution under edge delays into a synchronized under node delays while incurring a small overhead w.r.t space and time
A comprehensive study of spike and slab shrinkage priors for structurally sparse Bayesian neural networks
Network complexity and computational efficiency have become increasingly
significant aspects of deep learning. Sparse deep learning addresses these
challenges by recovering a sparse representation of the underlying target
function by reducing heavily over-parameterized deep neural networks.
Specifically, deep neural architectures compressed via structured sparsity
(e.g. node sparsity) provide low latency inference, higher data throughput, and
reduced energy consumption. In this paper, we explore two well-established
shrinkage techniques, Lasso and Horseshoe, for model compression in Bayesian
neural networks. To this end, we propose structurally sparse Bayesian neural
networks which systematically prune excessive nodes with (i) Spike-and-Slab
Group Lasso (SS-GL), and (ii) Spike-and-Slab Group Horseshoe (SS-GHS) priors,
and develop computationally tractable variational inference including
continuous relaxation of Bernoulli variables. We establish the contraction
rates of the variational posterior of our proposed models as a function of the
network topology, layer-wise node cardinalities, and bounds on the network
weights. We empirically demonstrate the competitive performance of our models
compared to the baseline models in prediction accuracy, model compression, and
inference latency
Minimalist Traffic Prediction: Linear Layer Is All You Need
Traffic prediction is essential for the progression of Intelligent
Transportation Systems (ITS) and the vision of smart cities. While
Spatial-Temporal Graph Neural Networks (STGNNs) have shown promise in this
domain by leveraging Graph Neural Networks (GNNs) integrated with either RNNs
or Transformers, they present challenges such as computational complexity,
gradient issues, and resource-intensiveness. This paper addresses these
challenges, advocating for three main solutions: a node-embedding approach,
time series decomposition, and periodicity learning. We introduce STLinear, a
minimalist model architecture designed for optimized efficiency and
performance. Unlike traditional STGNNs, STlinear operates fully locally,
avoiding inter-node data exchanges, and relies exclusively on linear layers,
drastically cutting computational demands. Our empirical studies on real-world
datasets confirm STLinear's prowess, matching or exceeding the accuracy of
leading STGNNs, but with significantly reduced complexity and computation
overhead (more than 95% reduction in MACs per epoch compared to
state-of-the-art STGNN baseline published in 2023). In summary, STLinear
emerges as a potent, efficient alternative to conventional STGNNs, with
profound implications for the future of ITS and smart city initiatives.Comment: 9 page
Exponentially Improving the Complexity of Simulating the Weisfeiler-Lehman Test with Graph Neural Networks
Recent work shows that the expressive power of Graph Neural Networks (GNNs)
in distinguishing non-isomorphic graphs is exactly the same as that of the
Weisfeiler-Lehman (WL) graph test. In particular, they show that the WL test
can be simulated by GNNs. However, those simulations involve neural networks
for the 'combine' function of size polynomial or even exponential in the number
of graph nodes , as well as feature vectors of length linear in .
We present an improved simulation of the WL test on GNNs with
\emph{exponentially} lower complexity. In particular, the neural network
implementing the combine function in each node has only a polylogarithmic
number of parameters in , and the feature vectors exchanged by the nodes of
GNN consists of only bits. We also give logarithmic lower bounds
for the feature vector length and the size of the neural networks, showing the
(near)-optimality of our construction.Comment: 22 pages,5 figures, accepted at NeurIPS 202
- …