12 research outputs found
An Online Word Vector Generation Method Based on Incremental Huffman Tree Merging
Aiming at high real-time performance processing requirements for large amounts of online text data in natural language processing applications, an online word vector model generation method based on incremental Huffman tree merging is proposed. Maintaining the inherited word Huffman tree in existing word vector model unchanged, a new Huffman tree of incoming words is constructed and ensures that there is no leaf node identical to the inherited Huffman tree. Then the Huffman tree is updated by a method of node merging. Thus based on the existing word vector model, each word still has a unique encoding for the calculation of the hierarchical softmax model. Finally, the generation of incremental word vector model is realized by using neural network on the basis of hierarchical softmax model. The experimental results show that the method could realize the word vector model generation online based on incremental learning with faster time and better performance
LASAGNE: Locality And Structure Aware Graph Node Embedding
In this work we propose Lasagne, a methodology to learn locality and
structure aware graph node embeddings in an unsupervised way. In particular, we
show that the performance of existing random-walk based approaches depends
strongly on the structural properties of the graph, e.g., the size of the
graph, whether the graph has a flat or upward-sloping Network Community Profile
(NCP), whether the graph is expander-like, whether the classes of interest are
more k-core-like or more peripheral, etc. For larger graphs with flat NCPs that
are strongly expander-like, existing methods lead to random walks that expand
rapidly, touching many dissimilar nodes, thereby leading to lower-quality
vector representations that are less useful for downstream tasks. Rather than
relying on global random walks or neighbors within fixed hop distances, Lasagne
exploits strongly local Approximate Personalized PageRank stationary
distributions to more precisely engineer local information into node
embeddings. This leads, in particular, to more meaningful and more useful
vector representations of nodes in poorly-structured graphs. We show that
Lasagne leads to significant improvement in downstream multi-label
classification for larger graphs with flat NCPs, that it is comparable for
smaller graphs with upward-sloping NCPs, and that is comparable to existing
methods for link prediction tasks
On Sampling Strategies for Neural Network-based Collaborative Filtering
Recent advances in neural networks have inspired people to design hybrid
recommendation algorithms that can incorporate both (1) user-item interaction
information and (2) content information including image, audio, and text.
Despite their promising results, neural network-based recommendation algorithms
pose extensive computational costs, making it challenging to scale and improve
upon. In this paper, we propose a general neural network-based recommendation
framework, which subsumes several existing state-of-the-art recommendation
algorithms, and address the efficiency issue by investigating sampling
strategies in the stochastic gradient descent training for the framework. We
tackle this issue by first establishing a connection between the loss functions
and the user-item interaction bipartite graph, where the loss function terms
are defined on links while major computation burdens are located at nodes. We
call this type of loss functions "graph-based" loss functions, for which varied
mini-batch sampling strategies can have different computational costs. Based on
the insight, three novel sampling strategies are proposed, which can
significantly improve the training efficiency of the proposed framework (up to
times speedup in our experiments), as well as improving the
recommendation performance. Theoretical analysis is also provided for both the
computational cost and the convergence. We believe the study of sampling
strategies have further implications on general graph-based loss functions, and
would also enable more research under the neural network-based recommendation
framework.Comment: This is a longer version (with supplementary attached) of the KDD'17
pape
NetSMF: Large-Scale Network Embedding as Sparse Matrix Factorization
We study the problem of large-scale network embedding, which aims to learn
latent representations for network mining applications. Previous research shows
that 1) popular network embedding benchmarks, such as DeepWalk, are in essence
implicitly factorizing a matrix with a closed form, and 2)the explicit
factorization of such matrix generates more powerful embeddings than existing
methods. However, directly constructing and factorizing this matrix---which is
dense---is prohibitively expensive in terms of both time and space, making it
not scalable for large networks.
In this work, we present the algorithm of large-scale network embedding as
sparse matrix factorization (NetSMF). NetSMF leverages theories from spectral
sparsification to efficiently sparsify the aforementioned dense matrix,
enabling significantly improved efficiency in embedding learning. The
sparsified matrix is spectrally close to the original dense one with a
theoretically bounded approximation error, which helps maintain the
representation power of the learned embeddings. We conduct experiments on
networks of various scales and types. Results show that among both popular
benchmarks and factorization based methods, NetSMF is the only method that
achieves both high efficiency and effectiveness. We show that NetSMF requires
only 24 hours to generate effective embeddings for a large-scale academic
collaboration network with tens of millions of nodes, while it would cost
DeepWalk months and is computationally infeasible for the dense matrix
factorization solution. The source code of NetSMF is publicly available
(https://github.com/xptree/NetSMF).Comment: 11 pages, in Proceedings of the Web Conference 2019 (WWW 19