8,648 research outputs found
Implicit Regularization in Deep Matrix Factorization
Efforts to understand the generalization mystery in deep learning have led to
the belief that gradient-based optimization induces a form of implicit
regularization, a bias towards models of low "complexity." We study the
implicit regularization of gradient descent over deep linear neural networks
for matrix completion and sensing, a model referred to as deep matrix
factorization. Our first finding, supported by theory and experiments, is that
adding depth to a matrix factorization enhances an implicit tendency towards
low-rank solutions, oftentimes leading to more accurate recovery. Secondly, we
present theoretical and empirical arguments questioning a nascent view by which
implicit regularization in matrix factorization can be captured using simple
mathematical norms. Our results point to the possibility that the language of
standard regularizers may not be rich enough to fully encompass the implicit
regularization brought forth by gradient-based optimization.Comment: Published at the conference on Neural Information Processing Systems
(NeurIPS) 201
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks
In the pursuit of explaining implicit regularization in deep learning,
prominent focus was given to matrix and tensor factorizations, which correspond
to simplified neural networks. It was shown that these models exhibit an
implicit tendency towards low matrix and tensor ranks, respectively. Drawing
closer to practical deep learning, the current paper theoretically analyzes the
implicit regularization in hierarchical tensor factorization, a model
equivalent to certain deep convolutional neural networks. Through a dynamical
systems lens, we overcome challenges associated with hierarchy, and establish
implicit regularization towards low hierarchical tensor rank. This translates
to an implicit regularization towards locality for the associated convolutional
networks. Inspired by our theory, we design explicit regularization
discouraging locality, and demonstrate its ability to improve the performance
of modern convolutional networks on non-local tasks, in defiance of
conventional wisdom by which architectural changes are needed. Our work
highlights the potential of enhancing neural networks via theoretical analysis
of their implicit regularization.Comment: Accepted to ICML 202
Regularizing Matrix Factorization with User and Item Embeddings for Recommendation
Following recent successes in exploiting both latent factor and word
embedding models in recommendation, we propose a novel Regularized
Multi-Embedding (RME) based recommendation model that simultaneously
encapsulates the following ideas via decomposition: (1) which items a user
likes, (2) which two users co-like the same items, (3) which two items users
often co-liked, and (4) which two items users often co-disliked. In
experimental validation, the RME outperforms competing state-of-the-art models
in both explicit and implicit feedback datasets, significantly improving
Recall@5 by 5.9~7.0%, NDCG@20 by 4.3~5.6%, and MAP@10 by 7.9~8.9%. In addition,
under the cold-start scenario for users with the lowest number of interactions,
against the competing models, the RME outperforms NDCG@5 by 20.2% and 29.4% in
MovieLens-10M and MovieLens-20M datasets, respectively. Our datasets and source
code are available at: https://github.com/thanhdtran/RME.git.Comment: CIKM 201
- …