999 research outputs found
Improving Negative Sampling for Word Representation using Self-embedded Features
Although the word-popularity based negative sampler has shown superb
performance in the skip-gram model, the theoretical motivation behind
oversampling popular (non-observed) words as negative samples is still not well
understood. In this paper, we start from an investigation of the gradient
vanishing issue in the skipgram model without a proper negative sampler. By
performing an insightful analysis from the stochastic gradient descent (SGD)
learning perspective, we demonstrate that, both theoretically and intuitively,
negative samples with larger inner product scores are more informative than
those with lower scores for the SGD learner in terms of both convergence rate
and accuracy. Understanding this, we propose an alternative sampling algorithm
that dynamically selects informative negative samples during each SGD update.
More importantly, the proposed sampler accounts for multi-dimensional
self-embedded features during the sampling process, which essentially makes it
more effective than the original popularity-based (one-dimensional) sampler.
Empirical experiments further verify our observations, and show that our
fine-grained samplers gain significant improvement over the existing ones
without increasing computational complexity.Comment: Accepted in WSDM 201
Molding CNNs for text: non-linear, non-consecutive convolutions
The success of deep learning often derives from well-chosen operational
building blocks. In this work, we revise the temporal convolution operation in
CNNs to better adapt it to text processing. Instead of concatenating word
representations, we appeal to tensor algebra and use low-rank n-gram tensors to
directly exploit interactions between words already at the convolution stage.
Moreover, we extend the n-gram convolution to non-consecutive words to
recognize patterns with intervening words. Through a combination of low-rank
tensors, and pattern weighting, we can efficiently evaluate the resulting
convolution operation via dynamic programming. We test the resulting
architecture on standard sentiment classification and news categorization
tasks. Our model achieves state-of-the-art performance both in terms of
accuracy and training speed. For instance, we obtain 51.2% accuracy on the
fine-grained sentiment classification task
Recovery Guarantees for Quadratic Tensors with Limited Observations
We consider the tensor completion problem of predicting the missing entries
of a tensor. The commonly used CP model has a triple product form, but an
alternate family of quadratic models which are the sum of pairwise products
instead of a triple product have emerged from applications such as
recommendation systems. Non-convex methods are the method of choice for
learning quadratic models, and this work examines their sample complexity and
error guarantee. Our main result is that with the number of samples being only
linear in the dimension, all local minima of the mean squared error objective
are global minima and recover the original tensor accurately. The techniques
lead to simple proofs showing that convex relaxation can recover quadratic
tensors provided with linear number of samples. We substantiate our theoretical
results with experiments on synthetic and real-world data, showing that
quadratic models have better performance than CP models in scenarios where
there are limited amount of observations available
Improved Relation Extraction with Feature-Rich Compositional Embedding Models
Compositional embedding models build a representation (or embedding) for a
linguistic structure based on its component word embeddings. We propose a
Feature-rich Compositional Embedding Model (FCM) for relation extraction that
is expressive, generalizes to new domains, and is easy-to-implement. The key
idea is to combine both (unlexicalized) hand-crafted features with learned word
embeddings. The model is able to directly tackle the difficulties met by
traditional compositional embeddings models, such as handling arbitrary types
of sentence annotations and utilizing global information for composition. We
test the proposed model on two relation extraction tasks, and demonstrate that
our model outperforms both previous compositional models and traditional
feature rich models on the ACE 2005 relation extraction task, and the SemEval
2010 relation classification task. The combination of our model and a
log-linear classifier with hand-crafted features gives state-of-the-art
results.Comment: 12 pages for EMNLP 201
- …