26 research outputs found
Matrix Scaling and Balancing via Box Constrained Newton's Method and Interior Point Methods
In this paper, we study matrix scaling and balancing, which are fundamental
problems in scientific computing, with a long line of work on them that dates
back to the 1960s. We provide algorithms for both these problems that, ignoring
logarithmic factors involving the dimension of the input matrix and the size of
its entries, both run in time where is the amount of error we are willing to
tolerate. Here, represents the ratio between the largest and the
smallest entries of the optimal scalings. This implies that our algorithms run
in nearly-linear time whenever is quasi-polynomial, which includes, in
particular, the case of strictly positive matrices. We complement our results
by providing a separate algorithm that uses an interior-point method and runs
in time .
In order to establish these results, we develop a new second-order
optimization framework that enables us to treat both problems in a unified and
principled manner. This framework identifies a certain generalization of linear
system solving that we can use to efficiently minimize a broad class of
functions, which we call second-order robust. We then show that in the context
of the specific functions capturing matrix scaling and balancing, we can
leverage and generalize the work on Laplacian system solving to make the
algorithms obtained via this framework very efficient.Comment: To appear in FOCS 201
How Does Batch Normalization Help Optimization?
Batch Normalization (BatchNorm) is a widely adopted technique that enables
faster and more stable training of deep neural networks (DNNs). Despite its
pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly
understood. The popular belief is that this effectiveness stems from
controlling the change of the layers' input distributions during training to
reduce the so-called "internal covariate shift". In this work, we demonstrate
that such distributional stability of layer inputs has little to do with the
success of BatchNorm. Instead, we uncover a more fundamental impact of
BatchNorm on the training process: it makes the optimization landscape
significantly smoother. This smoothness induces a more predictive and stable
behavior of the gradients, allowing for faster training.Comment: In NeurIPS'1
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
In-context learning refers to the ability of a model to condition on a prompt
sequence consisting of in-context examples (input-output pairs corresponding to
some task) along with a new query input, and generate the corresponding output.
Crucially, in-context learning happens only at inference time without any
parameter updates to the model. While large language models such as GPT-3
exhibit some ability to perform in-context learning, it is unclear what the
relationship is between tasks on which this succeeds and what is present in the
training data. To make progress towards understanding in-context learning, we
consider the well-defined problem of training a model to in-context learn a
function class (e.g., linear functions): that is, given data derived from some
functions in the class, can we train a model to in-context learn "most"
functions from this class? We show empirically that standard Transformers can
be trained from scratch to perform in-context learning of linear functions --
that is, the trained model is able to learn unseen linear functions from
in-context examples with performance comparable to the optimal least squares
estimator. In fact, in-context learning is possible even under two forms of
distribution shift: (i) between the training data of the model and
inference-time prompts, and (ii) between the in-context examples and the query
input during inference. We also show that we can train Transformers to
in-context learn more complex function classes -- namely sparse linear
functions, two-layer neural networks, and decision trees -- with performance
that matches or exceeds task-specific learning algorithms. Our code and models
are available at https://github.com/dtsip/in-context-learning
Adversarially Robust Generalization Requires More Data
Machine learning models are often susceptible to adversarial perturbations of
their inputs. Even small perturbations can cause state-of-the-art classifiers
with high "standard" accuracy to produce an incorrect prediction with high
confidence. To better understand this phenomenon, we study adversarially robust
learning from the viewpoint of generalization. We show that already in a simple
natural data model, the sample complexity of robust learning can be
significantly larger than that of "standard" learning. This gap is information
theoretic and holds irrespective of the training algorithm or the model family.
We complement our theoretical results with experiments on popular image
classification datasets and show that a similar gap exists here as well. We
postulate that the difficulty of training robust classifiers stems, at least
partially, from this inherently larger sample complexity.Comment: Small changes for biblatex compatibilit