268 research outputs found
Efficient Computation Sharing for Multi-Task Visual Scene Understanding
Solving multiple visual tasks using individual models can be
resource-intensive, while multi-task learning can conserve resources by sharing
knowledge across different tasks. Despite the benefits of multi-task learning,
such techniques can struggle with balancing the loss for each task, leading to
potential performance degradation. We present a novel computation- and
parameter-sharing framework that balances efficiency and accuracy to perform
multiple visual tasks utilizing individually-trained single-task transformers.
Our method is motivated by transfer learning schemes to reduce computational
and parameter storage costs while maintaining the desired performance. Our
approach involves splitting the tasks into a base task and the other sub-tasks,
and sharing a significant portion of activations and parameters/weights between
the base and sub-tasks to decrease inter-task redundancies and enhance
knowledge sharing. The evaluation conducted on NYUD-v2 and PASCAL-context
datasets shows that our method is superior to the state-of-the-art
transformer-based multi-task learning techniques with higher accuracy and
reduced computational resources. Moreover, our method is extended to video
stream inputs, further reducing computational costs by efficiently sharing
information across the temporal domain as well as the task domain. Our codes
and models will be publicly available.Comment: Camera-Ready version. Accepted to ICCV 202
Simple and Efficient Heterogeneous Graph Neural Network
Heterogeneous graph neural networks (HGNNs) have powerful capability to embed
rich structural and semantic information of a heterogeneous graph into node
representations. Existing HGNNs inherit many mechanisms from graph neural
networks (GNNs) over homogeneous graphs, especially the attention mechanism and
the multi-layer structure. These mechanisms bring excessive complexity, but
seldom work studies whether they are really effective on heterogeneous graphs.
This paper conducts an in-depth and detailed study of these mechanisms and
proposes Simple and Efficient Heterogeneous Graph Neural Network (SeHGNN). To
easily capture structural information, SeHGNN pre-computes the neighbor
aggregation using a light-weight mean aggregator, which reduces complexity by
removing overused neighbor attention and avoiding repeated neighbor aggregation
in every training epoch. To better utilize semantic information, SeHGNN adopts
the single-layer structure with long metapaths to extend the receptive field,
as well as a transformer-based semantic fusion module to fuse features from
different metapaths. As a result, SeHGNN exhibits the characteristics of simple
network structure, high prediction accuracy, and fast training speed. Extensive
experiments on five real-world heterogeneous graphs demonstrate the superiority
of SeHGNN over the state-of-the-arts on both accuracy and training speed.Comment: Accepted by AAAI 202
Rethinking Efficiency and Redundancy in Training Large-scale Graphs
Large-scale graphs are ubiquitous in real-world scenarios and can be trained
by Graph Neural Networks (GNNs) to generate representation for downstream
tasks. Given the abundant information and complex topology of a large-scale
graph, we argue that redundancy exists in such graphs and will degrade the
training efficiency. Unfortunately, the model scalability severely restricts
the efficiency of training large-scale graphs via vanilla GNNs. Despite recent
advances in sampling-based training methods, sampling-based GNNs generally
overlook the redundancy issue. It still takes intolerable time to train these
models on large-scale graphs. Thereby, we propose to drop redundancy and
improve efficiency of training large-scale graphs with GNNs, by rethinking the
inherent characteristics in a graph.
In this paper, we pioneer to propose a once-for-all method, termed DropReef,
to drop the redundancy in large-scale graphs. Specifically, we first conduct
preliminary experiments to explore potential redundancy in large-scale graphs.
Next, we present a metric to quantify the neighbor heterophily of all nodes in
a graph. Based on both experimental and theoretical analysis, we reveal the
redundancy in a large-scale graph, i.e., nodes with high neighbor heterophily
and a great number of neighbors. Then, we propose DropReef to detect and drop
the redundancy in large-scale graphs once and for all, helping reduce the
training time while ensuring no sacrifice in the model accuracy. To demonstrate
the effectiveness of DropReef, we apply it to recent state-of-the-art
sampling-based GNNs for training large-scale graphs, owing to the high
precision of such models. With DropReef leveraged, the training efficiency of
models can be greatly promoted. DropReef is highly compatible and is offline
performed, benefiting the state-of-the-art sampling-based GNNs in the present
and future to a significant extent.Comment: 11 Page
- …