3 research outputs found
Robust Graph Neural Networks using Weighted Graph Laplacian
Graph neural network (GNN) is achieving remarkable performances in a variety
of application domains. However, GNN is vulnerable to noise and adversarial
attacks in input data. Making GNN robust against noises and adversarial attacks
is an important problem. The existing defense methods for GNNs are
computationally demanding and are not scalable. In this paper, we propose a
generic framework for robustifying GNN known as Weighted Laplacian GNN
(RWL-GNN). The method combines Weighted Graph Laplacian learning with the GNN
implementation. The proposed method benefits from the positive
semi-definiteness property of Laplacian matrix, feature smoothness, and latent
features via formulating a unified optimization framework, which ensures the
adversarial/noisy edges are discarded and connections in the graph are
appropriately weighted. For demonstration, the experiments are conducted with
Graph convolutional neural network(GCNN) architecture, however, the proposed
framework is easily amenable to any existing GNN architecture. The simulation
results with benchmark dataset establish the efficacy of the proposed method,
both in accuracy and computational efficiency. Code can be accessed at
https://github.com/Bharat-Runwal/RWL-GNN.Comment: Accepted at IEEE International Conference on Signal Processing and
Communications (SPCOM), 202
Reprogramming under constraints: Revisiting efficient and reliable transferability of lottery tickets
In the era of foundation models with huge pre-training budgets, the
downstream tasks have been shifted to the narrative of efficient and fast
adaptation. For classification-based tasks in the domain of computer vision,
the two most efficient approaches have been linear probing (LP) and visual
prompting/reprogramming (VP); the former aims to learn a classifier in the form
of a linear head on the features extracted by the pre-trained model, while the
latter maps the input data to the domain of the source data on which the model
was originally pre-trained on. Although extensive studies have demonstrated the
differences between LP and VP in terms of downstream performance, we explore
the capabilities of the two aforementioned methods via the sparsity axis: (a)
Data sparsity: the impact of few-shot adaptation and (b) Model sparsity: the
impact of lottery tickets (LT). We demonstrate that LT are not universal
reprogrammers, i.e., for certain target datasets, reprogramming an LT yields
significantly lower performance than the reprogrammed dense model although
their corresponding upstream performance is similar. Further, we demonstrate
that the calibration of dense models is always superior to that of their
lottery ticket counterparts under both LP and VP regimes. Our empirical study
opens a new avenue of research into VP for sparse models and encourages further
understanding of the performance beyond the accuracy achieved by VP under
constraints of sparsity. Code and logs can be accessed at
\url{https://github.com/landskape-ai/Reprogram_LT}.Comment: Preprin
APP: Anytime Progressive Pruning
With the latest advances in deep learning, there has been a lot of focus on
the online learning paradigm due to its relevance in practical settings.
Although many methods have been investigated for optimal learning settings in
scenarios where the data stream is continuous over time, sparse networks
training in such settings have often been overlooked. In this paper, we explore
the problem of training a neural network with a target sparsity in a particular
case of online learning: the anytime learning at macroscale paradigm (ALMA). We
propose a novel way of progressive pruning, referred to as \textit{Anytime
Progressive Pruning} (APP); the proposed approach significantly outperforms the
baseline dense and Anytime OSP models across multiple architectures and
datasets under short, moderate, and long-sequence training. Our method, for
example, shows an improvement in accuracy of and a reduction in
the generalization gap by , while being rd the size
of the dense baseline model in few-shot restricted imagenet training. We
further observe interesting nonmonotonic transitions in the generalization gap
in the high number of megabatches-based ALMA. The code and experiment
dashboards can be accessed at
\url{https://github.com/landskape-ai/Progressive-Pruning} and
\url{https://wandb.ai/landskape/APP}, respectively.Comment: 21 pages including 4 pages of references. Preprint versio