2,966 research outputs found
Multi-Modal Bayesian Embeddings for Learning Social Knowledge Graphs
We study the extent to which online social networks can be connected to open
knowledge bases. The problem is referred to as learning social knowledge
graphs. We propose a multi-modal Bayesian embedding model, GenVector, to learn
latent topics that generate word and network embeddings. GenVector leverages
large-scale unlabeled data with embeddings and represents data of two
modalities---i.e., social network users and knowledge concepts---in a shared
latent topic space. Experiments on three datasets show that the proposed method
clearly outperforms state-of-the-art methods. We then deploy the method on
AMiner, a large-scale online academic search system with a network of
38,049,189 researchers with a knowledge base with 35,415,011 concepts. Our
method significantly decreases the error rate in an online A/B test with live
users
AliGraph: A Comprehensive Graph Neural Network Platform
An increasing number of machine learning tasks require dealing with large
graph datasets, which capture rich and complex relationship among potentially
billions of elements. Graph Neural Network (GNN) becomes an effective way to
address the graph learning problem by converting the graph data into a low
dimensional space while keeping both the structural and property information to
the maximum extent and constructing a neural network for training and
referencing. However, it is challenging to provide an efficient graph storage
and computation capabilities to facilitate GNN training and enable development
of new GNN algorithms. In this paper, we present a comprehensive graph neural
network system, namely AliGraph, which consists of distributed graph storage,
optimized sampling operators and runtime to efficiently support not only
existing popular GNNs but also a series of in-house developed ones for
different scenarios. The system is currently deployed at Alibaba to support a
variety of business scenarios, including product recommendation and
personalized search at Alibaba's E-Commerce platform. By conducting extensive
experiments on a real-world dataset with 492.90 million vertices, 6.82 billion
edges and rich attributes, AliGraph performs an order of magnitude faster in
terms of graph building (5 minutes vs hours reported from the state-of-the-art
PowerGraph platform). At training, AliGraph runs 40%-50% faster with the novel
caching strategy and demonstrates around 12 times speed up with the improved
runtime. In addition, our in-house developed GNN models all showcase their
statistically significant superiorities in terms of both effectiveness and
efficiency (e.g., 4.12%-17.19% lift by F1 scores)
Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
Despite its great success, machine learning can have its limits when dealing
with insufficient training data. A potential solution is the additional
integration of prior knowledge into the training process which leads to the
notion of informed machine learning. In this paper, we present a structured
overview of various approaches in this field. We provide a definition and
propose a concept for informed machine learning which illustrates its building
blocks and distinguishes it from conventional machine learning. We introduce a
taxonomy that serves as a classification framework for informed machine
learning approaches. It considers the source of knowledge, its representation,
and its integration into the machine learning pipeline. Based on this taxonomy,
we survey related research and describe how different knowledge representations
such as algebraic equations, logic rules, or simulation results can be used in
learning systems. This evaluation of numerous papers on the basis of our
taxonomy uncovers key methods in the field of informed machine learning.Comment: Accepted at IEEE Transactions on Knowledge and Data Engineering:
https://ieeexplore.ieee.org/document/942998
FILDNE: A Framework for Incremental Learning of Dynamic Networks Embeddings
Representation learning on graphs has emerged as a powerful mechanism to
automate feature vector generation for downstream machine learning tasks. The
advances in representation on graphs have centered on both homogeneous and
heterogeneous graphs, where the latter presenting the challenges associated
with multi-typed nodes and/or edges. In this paper, we consider the additional
challenge of evolving graphs. We ask the question of whether the advances in
representation learning for static graphs can be leveraged for dynamic graphs
and how? It is important to be able to incorporate those advances to maximize
the utility and generalization of methods. To that end, we propose the
Framework for Incremental Learning of Dynamic Networks Embedding (FILDNE),
which can utilize any existing static representation learning method for
learning node embeddings, while keeping the computational costs low. FILDNE
integrates the feature vectors computed using the standard methods over
different timesteps into a single representation by developing a convex
combination function and alignment mechanism. Experimental results on several
downstream tasks, over seven real-world data sets, show that FILDNE is able to
reduce memory and computational time costs while providing competitive quality
measure gains with respect to the contemporary methods for representation
learning on dynamic graphs
Representation Learning for Dynamic Graphs: A Survey
Graphs arise naturally in many real-world applications including social
networks, recommender systems, ontologies, biology, and computational finance.
Traditionally, machine learning models for graphs have been mostly designed for
static graphs. However, many applications involve evolving graphs. This
introduces important challenges for learning and inference since nodes,
attributes, and edges change over time. In this survey, we review the recent
advances in representation learning for dynamic graphs, including dynamic
knowledge graphs. We describe existing models from an encoder-decoder
perspective, categorize these encoders and decoders based on the techniques
they employ, and analyze the approaches in each category. We also review
several prominent applications and widely used datasets and highlight
directions for future research.Comment: Accepted at JMLR, 73 pages, 2 figure
Machine Learning with World Knowledge: The Position and Survey
Machine learning has become pervasive in multiple domains, impacting a wide
variety of applications, such as knowledge discovery and data mining, natural
language processing, information retrieval, computer vision, social and health
informatics, ubiquitous computing, etc. Two essential problems of machine
learning are how to generate features and how to acquire labels for machines to
learn. Particularly, labeling large amount of data for each domain-specific
problem can be very time consuming and costly. It has become a key obstacle in
making learning protocols realistic in applications. In this paper, we will
discuss how to use the existing general-purpose world knowledge to enhance
machine learning processes, by enriching the features or reducing the labeling
work. We start from the comparison of world knowledge with domain-specific
knowledge, and then introduce three key problems in using world knowledge in
learning processes, i.e., explicit and implicit feature representation,
inference for knowledge linking and disambiguation, and learning with direct or
indirect supervision. Finally we discuss the future directions of this research
topic
Stochastic Shared Embeddings: Data-driven Regularization of Embedding Layers
In deep neural nets, lower level embedding layers account for a large portion
of the total number of parameters. Tikhonov regularization, graph-based
regularization, and hard parameter sharing are approaches that introduce
explicit biases into training in a hope to reduce statistical complexity.
Alternatively, we propose stochastically shared embeddings (SSE), a data-driven
approach to regularizing embedding layers, which stochastically transitions
between embeddings during stochastic gradient descent (SGD). Because SSE
integrates seamlessly with existing SGD algorithms, it can be used with only
minor modifications when training large scale neural networks. We develop two
versions of SSE: SSE-Graph using knowledge graphs of embeddings; SSE-SE using
no prior information. We provide theoretical guarantees for our method and show
its empirical effectiveness on 6 distinct tasks, from simple neural networks
with one hidden layer in recommender systems, to the transformer and BERT in
natural languages. We find that when used along with widely-used regularization
methods such as weight decay and dropout, our proposed SSE can further reduce
overfitting, which often leads to more favorable generalization results.Comment: Accepted to 2019 Conference on Neural Information Processing System
Recent Advances in Zero-shot Recognition
With the recent renaissance of deep convolution neural networks, encouraging
breakthroughs have been achieved on the supervised recognition tasks, where
each class has sufficient training data and fully annotated training data.
However, to scale the recognition to a large number of classes with few or now
training samples for each class remains an unsolved problem. One approach to
scaling up the recognition is to develop models capable of recognizing unseen
categories without any training instances, or zero-shot recognition/ learning.
This article provides a comprehensive review of existing zero-shot recognition
techniques covering various aspects ranging from representations of models, and
from datasets and evaluation settings. We also overview related recognition
tasks including one-shot and open set recognition which can be used as natural
extensions of zero-shot recognition when limited number of class samples become
available or when zero-shot recognition is implemented in a real-world setting.
Importantly, we highlight the limitations of existing approaches and point out
future research directions in this existing new research area.Comment: accepted by IEEE Signal Processing Magazin
Review on Graph Feature Learning and Feature Extraction Techniques for Link Prediction
The problem of link prediction has recently attracted considerable attention
by research community. Given a graph, which is an abstraction of the
relationships among entities, the task of link prediction is to anticipate
future connections among entities in the graph, concerning its current state.
Extensive studies have examined this problem from different aspects and
proposed various methods, some of which might work very well for a specific
application but not as a global solution. This work presents an extensive
review of state-of-art methods and algorithms proposed on this subject and
categorizes them into four main categories: similarity-based methods,
probabilistic methods, relational models, and learning-based methods.
Additionally, a collection of network data sets has been presented in this
paper, which can be used to study link prediction. To the best of our
knowledge, this survey is the first comprehensive study that considers all of
the mentioned challenges and solutions for link prediction in graphs with the
improvements in the recent years, including the unsupervised and supervised
techniques and their evolution over the recent years.Comment: 31 pages, 7 figure
Compositional Fairness Constraints for Graph Embeddings
Learning high-quality node embeddings is a key building block for machine
learning models that operate on graph data, such as social networks and
recommender systems. However, existing graph embedding techniques are unable to
cope with fairness constraints, e.g., ensuring that the learned representations
do not correlate with certain attributes, such as age or gender. Here, we
introduce an adversarial framework to enforce fairness constraints on graph
embeddings. Our approach is compositional---meaning that it can flexibly
accommodate different combinations of fairness constraints during inference.
For instance, in the context of social recommendations, our framework would
allow one user to request that their recommendations are invariant to both
their age and gender, while also allowing another user to request invariance to
just their age. Experiments on standard knowledge graph and recommender system
benchmarks highlight the utility of our proposed framework.Comment: Proceedings of the 36th International Conference on Machine Learning,
Long Beach, California, PMLR 97, 201
- …