29 research outputs found
Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons
A lossy data compression scheme for uniformly biased Boolean messages is
investigated via statistical mechanics techniques. We utilize tree-like
committee machine (committee tree) and tree-like parity machine (parity tree)
whose transfer functions are non-monotonic. The scheme performance at the
infinite code length limit is analyzed using the replica method. Both committee
and parity treelike networks are shown to saturate the Shannon bound. The AT
stability of the Replica Symmetric solution is analyzed, and the tuning of the
non-monotonic transfer function is also discussed.Comment: 29 pages, 7 figure
Error correcting code using tree-like multilayer perceptron
An error correcting code using a tree-like multilayer perceptron is proposed.
An original message \mbi{s}^0 is encoded into a codeword \boldmath{y}_0
using a tree-like committee machine (committee tree) or a tree-like parity
machine (parity tree). Based on these architectures, several schemes featuring
monotonic or non-monotonic units are introduced. The codeword \mbi{y}_0 is
then transmitted via a Binary Asymmetric Channel (BAC) where it is corrupted by
noise. The analytical performance of these schemes is investigated using the
replica method of statistical mechanics. Under some specific conditions, some
of the proposed schemes are shown to saturate the Shannon bound at the infinite
codeword length limit. The influence of the monotonicity of the units on the
performance is also discussed.Comment: 23 pages, 3 figures, Content has been extended and revise
Statistical mechanics of lossy compression using multilayer perceptrons
Statistical mechanics is applied to lossy compression using multilayer
perceptrons for unbiased Boolean messages. We utilize a tree-like committee
machine (committee tree) and tree-like parity machine (parity tree) whose
transfer functions are monotonic. For compression using committee tree, a lower
bound of achievable distortion becomes small as the number of hidden units K
increases. However, it cannot reach the Shannon bound even where K -> infty.
For a compression using a parity tree with K >= 2 hidden units, the rate
distortion function, which is known as the theoretical limit for compression,
is derived where the code length becomes infinity.Comment: 12 pages, 5 figure
Belief Propagation for Error Correcting Codes and Lossy Compression Using Multilayer Perceptrons
The belief propagation (BP) based algorithm is investigated as a potential
decoder for both of error correcting codes and lossy compression, which are
based on non-monotonic tree-like multilayer perceptron encoders. We discuss
that whether the BP can give practical algorithms or not in these schemes. The
BP implementations in those kind of fully connected networks unfortunately
shows strong limitation, while the theoretical results seems a bit promising.
Instead, it reveals it might have a rich and complex structure of the solution
space via the BP-based algorithms.Comment: 18 pages, 18 figure
Computational capabilities of multilayer committee machines
We obtained an analytical expression for the computational complexity of many layered committee machines with a finite number of hidden layers (L < 8) using the generalization complexity measure introduced by Franco et al (2006) IEEE Trans. Neural Netw. 17 578. Although our result is valid in the large-size limit and for an overlap synaptic matrix that is ultrametric, it provides a useful tool for inferring the appropriate architecture a network must have to reproduce an arbitrary realizable Boolean function
Analysis of Mismatched Estimation Errors Using Gradients of Partition Functions
We consider the problem of signal estimation (denoising) from a
statistical-mechanical perspective, in continuation to a recent work on the
analysis of mean-square error (MSE) estimation using a direct relationship
between optimum estimation and certain partition functions. The paper consists
of essentially two parts. In the first part, using the aforementioned
relationship, we derive single-letter expressions of the mismatched MSE of a
codeword (from a randomly selected code), corrupted by a Gaussian vector
channel. In the second part, we provide several examples to demonstrate phase
transitions in the behavior of the MSE. These examples enable us to understand
more deeply and to gather intuition regarding the roles of the real and the
mismatched probability measures in creating these phase transitions.Comment: 58 pages;Submitted to IEEE Trans. on Information Theor
Technology Directions for the 21st Century
The Office of Space Communications (OSC) is tasked by NASA to conduct a planning process to meet NASA's science mission and other communications and data processing requirements. A set of technology trend studies was undertaken by Science Applications International Corporation (SAIC) for OSC to identify quantitative data that can be used to predict performance of electronic equipment in the future to assist in the planning process. Only commercially available, off-the-shelf technology was included. For each technology area considered, the current state of the technology is discussed, future applications that could benefit from use of the technology are identified, and likely future developments of the technology are described. The impact of each technology area on NASA operations is presented together with a discussion of the feasibility and risk associated with its development. An approximate timeline is given for the next 15 to 25 years to indicate the anticipated evolution of capabilities within each of the technology areas considered. This volume contains four chapters: one each on technology trends for database systems, computer software, neural and fuzzy systems, and artificial intelligence. The principal study results are summarized at the beginning of each chapter
Self-organising maps : statistical analysis, treatment and applications.
This thesis presents some substantial theoretical analyses and optimal treatments
of Kohonen's self-organising map (SOM) algorithm, and explores the practical
application potential of the algorithm for vector quantisation, pattern classification,
and image processing. It consists of two major parts. In the first part, the SOM
algorithm is investigated and analysed from a statistical viewpoint. The proof of its
universal convergence for any dimensionality is obtained using a novel and
extended form of the Central Limit Theorem. Its feature space is shown to be an
approximate multivariate Gaussian process, which will eventually converge and
form a mapping, which minimises the mean-square distortion between the feature
and input spaces. The diminishing effect of the initial states and implicit effects of
the learning rate and neighbourhood function on its convergence and ordering are
analysed and discussed. Distinct and meaningful definitions, and associated
measures, of its ordering are presented in relation to map's fault-tolerance. The
SOM algorithm is further enhanced by incorporating a proposed constraint, or
Bayesian modification, in order to achieve optimal vector quantisation or pattern
classification. The second part of this thesis addresses the task of unsupervised
texture-image segmentation by means of SOM networks and model-based
descriptions. A brief review of texture analysis in terms of definitions, perceptions,
and approaches is given. Markov random field model-based approaches are
discussed in detail. Arising from this a hierarchical self-organised segmentation
structure, which consists of a local MRF parameter estimator, a SOM network, and
a simple voting layer, is proposed and is shown, by theoretical analysis and
practical experiment, to achieve a maximum likelihood or maximum a posteriori
segmentation. A fast, simple, but efficient boundary relaxation algorithm is
proposed as a post-processor to further refine the resulting segmentation. The class
number validation problem in a fully unsupervised segmentation is approached by
a classical, simple, and on-line minimum mean-square-error method. Experimental
results indicate that this method is very efficient for texture segmentation
problems. The thesis concludes with some suggestions for further work on SOM
neural networks