Statistical Mechanics of Dictionary Learning

Finding a basis matrix (dictionary) by which objective signals are represented sparsely is of major relevance in various scientific and technological fields. We consider a problem to learn a dictionary from a set of training signals. We employ techniques of statistical mechanics of disordered systems to evaluate the size of the training set necessary to typically succeed in the dictionary learning. The results indicate that the necessary size is much smaller than previously estimated, which theoretically supports and/or encourages the use of dictionary learning in practical situations.Comment: 6 pages, 4 figure

Cavity approach to the first eigenvalue problem in a family of symmetric random sparse matrices

A methodology to analyze the properties of the first (largest) eigenvalue and its eigenvector is developed for large symmetric random sparse matrices utilizing the cavity method of statistical mechanics. Under a tree approximation, which is plausible for infinitely large systems, in conjunction with the introduction of a Lagrange multiplier for constraining the length of the eigenvector, the eigenvalue problem is reduced to a bunch of optimization problems of a quadratic function of a single variable, and the coefficients of the first and the second order terms of the functions act as cavity fields that are handled in cavity analysis. We show that the first eigenvalue is determined in such a way that the distribution of the cavity fields has a finite value for the second order moment with respect to the cavity fields of the first order coefficient. The validity and utility of the developed methodology are examined by applying it to two analytically solvable and one simple but non-trivial examples in conjunction with numerical justification.Comment: 11 pages, 4 figures, to be presented at IW-SMI2010, Kyoto, March 7-10, 201

On analyticity with respect to the replica number in random energy models I: an exact expression of the moment of the partition function

We provide an exact expression of the moment of the partition function for random energy models of finite system size, generalizing an earlier expression for a grand canonical version of the discrete random energy model presented by the authors in Prog. Theor. Phys. 111, 661 (2004). The expression can be handled both analytically and numerically, which is useful for examining how the analyticity of the moment with respect to the replica numbers, which play the role of powers of the moment, can be broken in the thermodynamic limit. A comparison with a replica method analysis indicates that the analyticity breaking can be regarded as the origin of the one-step replica symmetry breaking. The validity of the expression is also confirmed by numerical methods for finite systems.Comment: 16 pages, 4 figure