69,870 research outputs found
Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks
Recently, graph neural networks (GNNs) have shown its unprecedented success
in many graph-related tasks. However, GNNs face the label scarcity issue as
other neural networks do. Thus, recent efforts try to pre-train GNNs on a
large-scale unlabeled graph and adapt the knowledge from the unlabeled graph to
the target downstream task. The adaptation is generally achieved by fine-tuning
the pre-trained GNNs with a limited number of labeled data. Despite the
importance of fine-tuning, current GNNs pre-training works often ignore
designing a good fine-tuning strategy to better leverage transferred knowledge
and improve the performance on downstream tasks. Only few works start to
investigate a better fine-tuning strategy for pre-trained GNNs. But their
designs either have strong assumptions or overlook the data-aware issue for
various downstream datasets. Therefore, we aim to design a better fine-tuning
strategy for pre-trained GNNs to improve the model performance in this paper.
Given a pre-trained GNN, we propose to search to fine-tune pre-trained graph
neural networks for graph-level tasks (S2PGNN), which adaptively design a
suitable fine-tuning framework for the given labeled data on the downstream
task. To ensure the improvement brought by searching fine-tuning strategy, we
carefully summarize a proper search space of fine-tuning framework that is
suitable for GNNs. The empirical studies show that S2PGNN can be implemented on
the top of 10 famous pre-trained GNNs and consistently improve their
performance. Besides, S2PGNN achieves better performance than existing
fine-tuning strategies within and outside the GNN area. Our code is publicly
available at \url{https://anonymous.4open.science/r/code_icde2024-A9CB/}
Pore-GNN: A graph neural network-based framework for predicting flow properties of porous media from micro-CT images
This paper presents a hybrid deep learning framework that combines graph neural networks with convolutional neural networks to predict porous media properties. This approach capitalizes on the capabilities of pre-trained convolutional neural networks to extract n-dimensional feature vectors from processed three dimensional micro computed tomography porous media images obtained from seven different sandstone rock samples. Subsequently, two strategies for embedding the computed feature vectors into graphs were explored: extracting a single feature vector per sample (image) and treating each sample as a node in the training graph, and representing each sample as a graph by extracting a fixed number of feature vectors, which form the nodes of each training graph. Various types of graph convolutional layers were examined to evaluate the capabilities and limitations of spectral and spatial approaches. The dataset was divided into 70/20/10 for training, validation, and testing. The models were trained to predict the absolute permeability of porous media. Notably, the proposed architectures further reduce the selected objective loss function to values below 35 mD, with improvements in the coefficient of determination reaching 9%. Moreover, the generalizability of the networks was evaluated by testing their performance on unseen sandstone and carbonate rock samples that were not encountered during training. Finally, a sensitivity analysis is conducted to investigate the influence of various hyperparameters on the performance of the models. The findings highlight the potential of graph neural networks as promising deep learning-based alternatives for characterizing porous media properties. The proposed architectures efficiently predict the permeability, which is more than 500 times faster than that of numerical solvers.Document Type: Original articleCited as: Alzahrani, M. K., Shapoval, A., Chen, Z., Rahman, S. S. Pore-GNN: A graph neural network-based framework for predicting flow properties of porous media from micro-CT images. Advances in Geo-Energy Research, 2023, 10(1):39-55. https://doi.org/10.46690/ager.2023.10.0
Motif-aware Attribute Masking for Molecular Graph Pre-training
Attribute reconstruction is used to predict node or edge features in the
pre-training of graph neural networks. Given a large number of molecules, they
learn to capture structural knowledge, which is transferable for various
downstream property prediction tasks and vital in chemistry, biomedicine, and
material science. Previous strategies that randomly select nodes to do
attribute masking leverage the information of local neighbors However, the
over-reliance of these neighbors inhibits the model's ability to learn from
higher-level substructures. For example, the model would learn little from
predicting three carbon atoms in a benzene ring based on the other three but
could learn more from the inter-connections between the functional groups, or
called chemical motifs. In this work, we propose and investigate motif-aware
attribute masking strategies to capture inter-motif structures by leveraging
the information of atoms in neighboring motifs. Once each graph is decomposed
into disjoint motifs, the features for every node within a sample motif are
masked. The graph decoder then predicts the masked features of each node within
the motif for reconstruction. We evaluate our approach on eight molecular
property prediction datasets and demonstrate its advantages
Efficient Deep Feature Learning and Extraction via StochasticNets
Deep neural networks are a powerful tool for feature learning and extraction
given their ability to model high-level abstractions in highly complex data.
One area worth exploring in feature learning and extraction using deep neural
networks is efficient neural connectivity formation for faster feature learning
and extraction. Motivated by findings of stochastic synaptic connectivity
formation in the brain as well as the brain's uncanny ability to efficiently
represent information, we propose the efficient learning and extraction of
features via StochasticNets, where sparsely-connected deep neural networks can
be formed via stochastic connectivity between neurons. To evaluate the
feasibility of such a deep neural network architecture for feature learning and
extraction, we train deep convolutional StochasticNets to learn abstract
features using the CIFAR-10 dataset, and extract the learned features from
images to perform classification on the SVHN and STL-10 datasets. Experimental
results show that features learned using deep convolutional StochasticNets,
with fewer neural connections than conventional deep convolutional neural
networks, can allow for better or comparable classification accuracy than
conventional deep neural networks: relative test error decrease of ~4.5% for
classification on the STL-10 dataset and ~1% for classification on the SVHN
dataset. Furthermore, it was shown that the deep features extracted using deep
convolutional StochasticNets can provide comparable classification accuracy
even when only 10% of the training data is used for feature learning. Finally,
it was also shown that significant gains in feature extraction speed can be
achieved in embedded applications using StochasticNets. As such, StochasticNets
allow for faster feature learning and extraction performance while facilitate
for better or comparable accuracy performances.Comment: 10 pages. arXiv admin note: substantial text overlap with
arXiv:1508.0546
- …