2 research outputs found
Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization
In this paper, we present some theoretical work to explain why simple
gradient descent methods are so successful in solving non-convex optimization
problems in learning large-scale neural networks (NN). After introducing a
mathematical tool called canonical space, we have proved that the objective
functions in learning NNs are convex in the canonical model space. We further
elucidate that the gradients between the original NN model space and the
canonical space are related by a pointwise linear transformation, which is
represented by the so-called disparity matrix. Furthermore, we have proved that
gradient descent methods surely converge to a global minimum of zero loss
provided that the disparity matrices maintain full rank. If this full-rank
condition holds, the learning of NNs behaves in the same way as normal convex
optimization. At last, we have shown that the chance to have singular disparity
matrices is extremely slim in large NNs. In particular, when over-parameterized
NNs are randomly initialized, the gradient decent algorithms converge to a
global minimum of zero loss in probability.Comment: 10 page
A Latent Space Theory for Emergent Abilities in Large Language Models
Languages are not created randomly but rather to communicate information.
There is a strong association between languages and their underlying meanings,
resulting in a sparse joint distribution that is heavily peaked according to
their correlations. Moreover, these peak values happen to match with the
marginal distribution of languages due to the sparsity. With the advent of LLMs
trained on big data and large models, we can now precisely assess the marginal
distribution of languages, providing a convenient means of exploring the sparse
structures in the joint distribution for effective inferences. In this paper,
we categorize languages as either unambiguous or {\epsilon}-ambiguous and
present quantitative results to demonstrate that the emergent abilities of
LLMs, such as language understanding, in-context learning, chain-of-thought
prompting, and effective instruction fine-tuning, can all be attributed to
Bayesian inference on the sparse joint distribution of languages.Comment: 17 pages, 3 figure