12,548 research outputs found
Improving Negative Sampling for Word Representation using Self-embedded Features
Although the word-popularity based negative sampler has shown superb
performance in the skip-gram model, the theoretical motivation behind
oversampling popular (non-observed) words as negative samples is still not well
understood. In this paper, we start from an investigation of the gradient
vanishing issue in the skipgram model without a proper negative sampler. By
performing an insightful analysis from the stochastic gradient descent (SGD)
learning perspective, we demonstrate that, both theoretically and intuitively,
negative samples with larger inner product scores are more informative than
those with lower scores for the SGD learner in terms of both convergence rate
and accuracy. Understanding this, we propose an alternative sampling algorithm
that dynamically selects informative negative samples during each SGD update.
More importantly, the proposed sampler accounts for multi-dimensional
self-embedded features during the sampling process, which essentially makes it
more effective than the original popularity-based (one-dimensional) sampler.
Empirical experiments further verify our observations, and show that our
fine-grained samplers gain significant improvement over the existing ones
without increasing computational complexity.Comment: Accepted in WSDM 201
Adaptive high-order splitting schemes for large-scale differential Riccati equations
We consider high-order splitting schemes for large-scale differential Riccati
equations. Such equations arise in many different areas and are especially
important within the field of optimal control. In the large-scale case, it is
critical to employ structural properties of the matrix-valued solution, or the
computational cost and storage requirements become infeasible. Our main
contribution is therefore to formulate these high-order splitting schemes in a
efficient way by utilizing a low-rank factorization. Previous results indicated
that this was impossible for methods of order higher than 2, but our new
approach overcomes these difficulties. In addition, we demonstrate that the
proposed methods contain natural embedded error estimates. These may be used
e.g. for time step adaptivity, and our numerical experiments in this direction
show promising results.Comment: 23 pages, 7 figure
- …