Statistical Mechanics of Dictionary Learning

Finding a basis matrix (dictionary) by which objective signals are represented sparsely is of major relevance in various scientific and technological fields. We consider a problem to learn a dictionary from a set of training signals. We employ techniques of statistical mechanics of disordered systems to evaluate the size of the training set necessary to typically succeed in the dictionary learning. The results indicate that the necessary size is much smaller than previously estimated, which theoretically supports and/or encourages the use of dictionary learning in practical situations.Comment: 6 pages, 4 figure

Statistical mechanics of typical set decoding

The performance of ``typical set (pairs) decoding'' for ensembles of Gallager's linear code is investigated using statistical physics. In this decoding, error happens when the information transmission is corrupted by an untypical noise or two or more typical sequences satisfy the parity check equation provided by the received codeword for which a typical noise is added. We show that the average error rate for the latter case over a given code ensemble can be tightly evaluated using the replica method, including the sensitivity to the message length. Our approach generally improves the existing analysis known in information theory community, which was reintroduced by MacKay (1999) and believed as most accurate to date.Comment: 7 page

Numerical Study of TAP Metastable States in 3-body Ising Spin Glasses

The distribution of solutions of the Thouless-Anderson-Palmer equation is studied by extensive numerical experiments for fully connected 3-body interaction Ising spin glass models in a level of annealed calculation. A recent study predicted that when the equilibrium state of the system is characterized by one-step replica symmetry breaking, the distribution is described by a Becchi-Rouet-Stora-Tyutin (BRST) supersymmetric solution in the relatively low free energy region, whereas the BRST supersymmetry is broken for higher values of free energy (Crisanti et al., Phys. Rev. B 71 (2005) 094202). Our experiments qualitatively reproduce the discriminative behavior of macroscopic variables predicted by the theoretical assessment.Comment: 13 pages, 4 figure

Analysis of CDMA systems that are characterized by eigenvalue spectrum

An approach by which to analyze the performance of the code division multiple access (CDMA) scheme, which is a core technology used in modern wireless communication systems, is provided. The approach characterizes the objective system by the eigenvalue spectrum of a cross-correlation matrix composed of signature sequences used in CDMA communication, which enables us to handle a wider class of CDMA systems beyond the basic model reported by Tanaka. The utility of the novel scheme is shown by analyzing a system in which the generation of signature sequences is designed for enhancing the orthogonality.Comment: 7 pages, 2 figure

Parallel dynamics of continuous Hopfield model revisited

We have applied the generating functional analysis (GFA) to the continuous Hopfield model. We have also confirmed that the GFA predictions in some typical cases exhibit good consistency with computer simulation results. When a retarded self-interaction term is omitted, the GFA result becomes identical to that obtained using the statistical neurodynamics as well as the case of the sequential binary Hopfield model.Comment: 4 pages, 2 figure

Analysis of common attacks in LDPCC-based public-key cryptosystems

We analyze the security and reliability of a recently proposed class of public-key cryptosystems against attacks by unauthorized parties who have acquired partial knowledge of one or more of the private key components and/or of the plaintext. Phase diagrams are presented, showing critical partial knowledge levels required for unauthorized decryptionComment: 14 pages, 6 figure

Typical Performance of Gallager-type Error-Correcting Codes

The performance of Gallager's error-correcting code is investigated via methods of statistical physics. In this approach, the transmitted codeword comprises products of the original message bits selected by two randomly-constructed sparse matrices; the number of non-zero row/column elements in these matrices constitutes a family of codes. We show that Shannon's channel capacity is saturated for many of the codes while slightly lower performance is obtained for others which may be of higher practical relevance. Decoding aspects are considered by employing the TAP approach which is identical to the commonly used belief-propagation-based decoding.Comment: 6 pages, latex, 1 figur

On-line learning of non-monotonic rules by simple perceptron

We study the generalization ability of a simple perceptron which learns unlearnable rules. The rules are presented by a teacher perceptron with a non-monotonic transfer function. The student is trained in the on-line mode. The asymptotic behaviour of the generalization error is estimated under various conditions. Several learning strategies are proposed and improved to obtain the theoretical lower bound of the generalization error.Comment: LaTeX 20 pages using IOP LaTeX preprint style file, 14 figure

The Statistical Physics of Regular Low-Density Parity-Check Error-Correcting Codes

A variation of Gallager error-correcting codes is investigated using statistical mechanics. In codes of this type, a given message is encoded into a codeword which comprises Boolean sums of message bits selected by two randomly constructed sparse matrices. The similarity of these codes to Ising spin systems with random interaction makes it possible to assess their typical performance by analytical methods developed in the study of disordered systems. The typical case solutions obtained via the replica method are consistent with those obtained in simulations using belief propagation (BP) decoding. We discuss the practical implications of the results obtained and suggest a computationally efficient construction for one of the more practical configurations.Comment: 35 pages, 4 figure

Cryptographical Properties of Ising Spin Systems

The relation between Ising spin systems and public-key cryptography is investigated using methods of statistical physics. The insight gained from the analysis is used for devising a matrix-based cryptosystem whereby the ciphertext comprises products of the original message bits; these are selected by employing two predetermined randomly-constructed sparse matrices. The ciphertext is decrypted using methods of belief-propagation. The analyzed properties of the suggested cryptosystem show robustness against various attacks and competitive performance to modern cyptographical methods.Comment: 4 pages, 2 figure