CORE
🇺🇦Â
 make metadata, not war
Services
Services overview
Explore all CORE services
Access to raw data
API
Dataset
FastSync
Content discovery
Recommender
Discovery
OAI identifiers
OAI Resolver
Managing content
Dashboard
Bespoke contracts
Consultancy services
Support us
Support us
Membership
Sponsorship
Community governance
Advisory Board
Board of supporters
Research network
About
About us
Our mission
Team
Blog
FAQs
Contact us
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
Authors
Sanjeev Arora
Mikhail Khodak
Nikunj Saunshi
Yi Zhang
Publication date
1 January 2020
Publisher
View
on
arXiv
Abstract
One popular trend in meta-learning is to learn from many training tasks a common initialization for a gradient-based method that can be used to solve a new task with few samples. The theory of meta-learning is still in its early stages, with several recent learning-theoretic analyses of methods such as Reptile [Nichol et al., 2018] being for convex models. This work shows that convex-case analysis might be insufficient to understand the success of meta-learning, and that even for non-convex models it is important to look inside the optimization black-box, specifically at properties of the optimization trajectory. We construct a simple meta-learning instance that captures the problem of one-dimensional subspace learning. For the convex formulation of linear regression on this instance, we show that the new task sample complexity of any initialization-based meta-learning algorithm is
Ω
(
d
)
\Omega(d)
Ω
(
d
)
, where
d
d
d
is the input dimension. In contrast, for the non-convex formulation of a two layer linear network on the same instance, we show that both Reptile and multi-task representation learning can have new task sample complexity of
O
(
1
)
\mathcal{O}(1)
O
(
1
)
, demonstrating a separation from convex meta-learning. Crucially, analyses of the training dynamics of these methods reveal that they can meta-learn the correct subspace onto which the data should be projected.Comment: 34 page
Similar works
Full text
Available Versions
Sustaining member
Princeton University Open Access Repository
See this paper in CORE
Go to the repository landing page
Download from data provider
oai:oar.princeton.edu:88435/pr...
Last time updated on 14/02/2024