12 research outputs found
Quantum-Inspired Algorithms from Randomized Numerical Linear Algebra
We create classical (non-quantum) dynamic data structures supporting queries
for recommender systems and least-squares regression that are comparable to
their quantum analogues. De-quantizing such algorithms has received a flurry of
attention in recent years; we obtain sharper bounds for these problems. More
significantly, we achieve these improvements by arguing that the previous
quantum-inspired algorithms for these problems are doing leverage or
ridge-leverage score sampling in disguise; these are powerful and standard
techniques in randomized numerical linear algebra. With this recognition, we
are able to employ the large body of work in numerical linear algebra to obtain
algorithms for these problems that are simpler or faster (or both) than
existing approaches.Comment: Adding new numerical experiment
Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recovery
The practice of deep learning has shown that neural networks generalize
remarkably well even with an extreme number of learned parameters. This appears
to contradict traditional statistical wisdom, in which a trade-off between
model complexity and fit to the data is essential. We set out to resolve this
discrepancy from a convex optimization and sparse recovery perspective. We
consider the training and generalization properties of two-layer ReLU networks
with standard weight decay regularization. Under certain regularity assumptions
on the data, we show that ReLU networks with an arbitrary number of parameters
learn only simple models that explain the data. This is analogous to the
recovery of the sparsest linear model in compressed sensing. For ReLU networks
and their variants with skip connections or normalization layers, we present
isometry conditions that ensure the exact recovery of planted neurons. For
randomly generated data, we show the existence of a phase transition in
recovering planted neural network models. The situation is simple: whenever the
ratio between the number of samples and the dimension exceeds a numerical
threshold, the recovery succeeds with high probability; otherwise, it fails
with high probability. Surprisingly, ReLU networks learn simple and sparse
models even when the labels are noisy. The phase transition phenomenon is
confirmed through numerical experiments