2 research outputs found
Memory-efficient Embedding for Recommendations
Practical large-scale recommender systems usually contain thousands of
feature fields from users, items, contextual information, and their
interactions. Most of them empirically allocate a unified dimension to all
feature fields, which is memory inefficient. Thus it is highly desired to
assign different embedding dimensions to different feature fields according to
their importance and predictability. Due to the large amounts of feature fields
and the nuanced relationship between embedding dimensions with feature
distributions and neural network architectures, manually allocating embedding
dimensions in practical recommender systems can be very difficult. To this end,
we propose an AutoML based framework (AutoDim) in this paper, which can
automatically select dimensions for different feature fields in a data-driven
fashion. Specifically, we first proposed an end-to-end differentiable framework
that can calculate the weights over various dimensions for feature fields in a
soft and continuous manner with an AutoML based optimization algorithm; then we
derive a hard and discrete embedding component architecture according to the
maximal weights and retrain the whole recommender framework. We conduct
extensive experiments on benchmark datasets to validate the effectiveness of
the AutoDim framework
Learnable Embedding Sizes for Recommender Systems
The embedding-based representation learning is commonly used in deep learning
recommendation models to map the raw sparse features to dense vectors. The
traditional embedding manner that assigns a uniform size to all features has
two issues. First, the numerous features inevitably lead to a gigantic
embedding table that causes a high memory usage cost. Second, it is likely to
cause the over-fitting problem for those features that do not require too large
representation capacity. Existing works that try to address the problem always
cause a significant drop in recommendation performance or suffers from the
limitation of unaffordable training time cost. In this paper, we proposed a
novel approach, named PEP (short for Plug-in Embedding Pruning), to reduce the
size of the embedding table while avoiding the drop of recommendation accuracy.
PEP prunes embedding parameter where the pruning threshold(s) can be adaptively
learned from data. Therefore we can automatically obtain a mixed-dimension
embedding-scheme by pruning redundant parameters for each feature. PEP is a
general framework that can plug in various base recommendation models.
Extensive experiments demonstrate it can efficiently cut down embedding
parameters and boost the base model's performance. Specifically, it achieves
strong recommendation performance while reducing 97-99% parameters. As for the
computation cost, PEP only brings an additional 20-30% time cost compared with
base models. Codes are available at
https://github.com/ssui-liu/learnable-embed-sizes-for-RecSys.Comment: International Conference on Learning Representations (ICLR), 202