48 research outputs found
Rethinking Knowledge Graph Propagation for Zero-Shot Learning
Graph convolutional neural networks have recently shown great potential for
the task of zero-shot learning. These models are highly sample efficient as
related concepts in the graph structure share statistical strength allowing
generalization to new classes when faced with a lack of data. However,
multi-layer architectures, which are required to propagate knowledge to distant
nodes in the graph, dilute the knowledge by performing extensive Laplacian
smoothing at each layer and thereby consequently decrease performance. In order
to still enjoy the benefit brought by the graph structure while preventing
dilution of knowledge from distant nodes, we propose a Dense Graph Propagation
(DGP) module with carefully designed direct links among distant nodes. DGP
allows us to exploit the hierarchical graph structure of the knowledge graph
through additional connections. These connections are added based on a node's
relationship to its ancestors and descendants. A weighting scheme is further
used to weigh their contribution depending on the distance to the node to
improve information propagation in the graph. Combined with finetuning of the
representations in a two-stage training approach our method outperforms
state-of-the-art zero-shot learning approaches.Comment: The first two authors contributed equally. Code at
https://github.com/cyvius96/adgpm. To appear in CVPR 201
Bilinear Graph Neural Network with Neighbor Interactions
Graph Neural Network (GNN) is a powerful model to learn representations and
make predictions on graph data. Existing efforts on GNN have largely defined
the graph convolution as a weighted sum of the features of the connected nodes
to form the representation of the target node. Nevertheless, the operation of
weighted sum assumes the neighbor nodes are independent of each other, and
ignores the possible interactions between them. When such interactions exist,
such as the co-occurrence of two neighbor nodes is a strong signal of the
target node's characteristics, existing GNN models may fail to capture the
signal. In this work, we argue the importance of modeling the interactions
between neighbor nodes in GNN. We propose a new graph convolution operator,
which augments the weighted sum with pairwise interactions of the
representations of neighbor nodes. We term this framework as Bilinear Graph
Neural Network (BGNN), which improves GNN representation ability with bilinear
interactions between neighbor nodes. In particular, we specify two BGNN models
named BGCN and BGAT, based on the well-known GCN and GAT, respectively.
Empirical results on three public benchmarks of semi-supervised node
classification verify the effectiveness of BGNN -- BGCN (BGAT) outperforms GCN
(GAT) by 1.6% (1.5%) in classification accuracy.Codes are available at:
https://github.com/zhuhm1996/bgnn.Comment: Accepted by IJCAI 2020. SOLE copyright holder is IJCAI (International
Joint Conferences on Artificial Intelligence), all rights reserve
Self-Constructing Graph Convolutional Networks for Semantic Labeling
Graph Neural Networks (GNNs) have received increasing attention in many
fields. However, due to the lack of prior graphs, their use for semantic
labeling has been limited. Here, we propose a novel architecture called the
Self-Constructing Graph (SCG), which makes use of learnable latent variables to
generate embeddings and to self-construct the underlying graphs directly from
the input features without relying on manually built prior knowledge graphs.
SCG can automatically obtain optimized non-local context graphs from
complex-shaped objects in aerial imagery. We optimize SCG via an adaptive
diagonal enhancement method and a variational lower bound that consists of a
customized graph reconstruction term and a Kullback-Leibler divergence
regularization term. We demonstrate the effectiveness and flexibility of the
proposed SCG on the publicly available ISPRS Vaihingen dataset and our model
SCG-Net achieves competitive results in terms of F1-score with much fewer
parameters and at a lower computational cost compared to related pure-CNN based
work. Our code will be made public soon.Comment: IGARSS-2020, code at: github.com/samleoqh/MSCG-Ne
Context-Aware Zero-Shot Recognition
We present a novel problem setting in zero-shot learning, zero-shot object
recognition and detection in the context. Contrary to the traditional zero-shot
learning methods, which simply infers unseen categories by transferring
knowledge from the objects belonging to semantically similar seen categories,
we aim to understand the identity of the novel objects in an image surrounded
by the known objects using the inter-object relation prior. Specifically, we
leverage the visual context and the geometric relationships between all pairs
of objects in a single image, and capture the information useful to infer
unseen categories. We integrate our context-aware zero-shot learning framework
into the traditional zero-shot learning techniques seamlessly using a
Conditional Random Field (CRF). The proposed algorithm is evaluated on both
zero-shot region classification and zero-shot detection tasks. The results on
Visual Genome (VG) dataset show that our model significantly boosts performance
with the additional visual context compared to traditional methods
Multi-view Self-Constructing Graph Convolutional Networks with Adaptive Class Weighting Loss for Semantic Segmentation
We propose a novel architecture called the Multi-view Self-Constructing Graph
Convolutional Networks (MSCG-Net) for semantic segmentation. Building on the
recently proposed Self-Constructing Graph (SCG) module, which makes use of
learnable latent variables to self-construct the underlying graphs directly
from the input features without relying on manually built prior knowledge
graphs, we leverage multiple views in order to explicitly exploit the
rotational invariance in airborne images. We further develop an adaptive class
weighting loss to address the class imbalance. We demonstrate the effectiveness
and flexibility of the proposed method on the Agriculture-Vision challenge
dataset and our model achieves very competitive results (0.547 mIoU) with much
fewer parameters and at a lower computational cost compared to related pure-CNN
based work. Code will be available at: github.com/samleoqh/MSCG-NetComment: 7-page, MSCG-Net, CVPRW-202
Graph Neural Networks and Reinforcement Learning for Behavior Generation in Semantic Environments
Most reinforcement learning approaches used in behavior generation utilize
vectorial information as input. However, this requires the network to have a
pre-defined input-size -- in semantic environments this means assuming the
maximum number of vehicles. Additionally, this vectorial representation is not
invariant to the order and number of vehicles. To mitigate the above-stated
disadvantages, we propose combining graph neural networks with actor-critic
reinforcement learning. As graph neural networks apply the same network to
every vehicle and aggregate incoming edge information, they are invariant to
the number and order of vehicles. This makes them ideal candidates to be used
as networks in semantic environments -- environments consisting of objects
lists. Graph neural networks exhibit some other advantages that make them
favorable to be used in semantic environments. The relational information is
explicitly given and does not have to be inferred. Moreover, graph neural
networks propagate information through the network and can gather higher-degree
information. We demonstrate our approach using a highway lane-change scenario
and compare the performance of graph neural networks to conventional ones. We
show that graph neural networks are capable of handling scenarios with a
varying number and order of vehicles during training and application