26,936 research outputs found
Fast optimization of Multithreshold Entropy Linear Classifier
Multithreshold Entropy Linear Classifier (MELC) is a density based model
which searches for a linear projection maximizing the Cauchy-Schwarz Divergence
of dataset kernel density estimation. Despite its good empirical results, one
of its drawbacks is the optimization speed. In this paper we analyze how one
can speed it up through solving an approximate problem. We analyze two methods,
both similar to the approximate solutions of the Kernel Density Estimation
querying and provide adaptive schemes for selecting a crucial parameters based
on user-specified acceptable error. Furthermore we show how one can exploit
well known conjugate gradients and L-BFGS optimizers despite the fact that the
original optimization problem should be solved on the sphere. All above methods
and modifications are tested on 10 real life datasets from UCI repository to
confirm their practical usability.Comment: Presented at Theoretical Foundations of Machine Learning 2015
(http://tfml.gmum.net), final version published in Schedae Informaticae
Journa
Calculation of Densities of States and Spectral Functions by Chebyshev Recursion and Maximum Entropy
We present an efficient algorithm for calculating spectral properties of
large sparse Hamiltonian matrices such as densities of states and spectral
functions. The combination of Chebyshev recursion and maximum entropy achieves
high energy resolution without significant roundoff error, machine precision or
numerical instability limitations. If controlled statistical or systematic
errors are acceptable, cpu and memory requirements scale linearly in the number
of states. The inference of spectral properties from moments is much better
conditioned for Chebyshev moments than for power moments. We adapt concepts
from the kernel polynomial approximation, a linear Chebyshev approximation with
optimized Gibbs damping, to control the accuracy of Fourier integrals of
positive non-analytic functions. We compare the performance of kernel
polynomial and maximum entropy algorithms for an electronic structure example.Comment: 8 pages RevTex, 3 postscript figure
Communications-Inspired Projection Design with Application to Compressive Sensing
We consider the recovery of an underlying signal x \in C^m based on
projection measurements of the form y=Mx+w, where y \in C^l and w is
measurement noise; we are interested in the case l < m. It is assumed that the
signal model p(x) is known, and w CN(w;0,S_w), for known S_W. The objective is
to design a projection matrix M \in C^(l x m) to maximize key
information-theoretic quantities with operational significance, including the
mutual information between the signal and the projections I(x;y) or the Renyi
entropy of the projections h_a(y) (Shannon entropy is a special case). By
capitalizing on explicit characterizations of the gradients of the information
measures with respect to the projections matrix, where we also partially extend
the well-known results of Palomar and Verdu from the mutual information to the
Renyi entropy domain, we unveil the key operations carried out by the optimal
projections designs: mode exposure and mode alignment. Experiments are
considered for the case of compressive sensing (CS) applied to imagery. In this
context, we provide a demonstration of the performance improvement possible
through the application of the novel projection designs in relation to
conventional ones, as well as justification for a fast online projections
design method with which state-of-the-art adaptive CS signal recovery is
achieved.Comment: 25 pages, 7 figures, parts of material published in IEEE ICASSP 2012,
submitted to SIIM
Extreme Entropy Machines: Robust information theoretic classification
Most of the existing classification methods are aimed at minimization of
empirical risk (through some simple point-based error measured with loss
function) with added regularization. We propose to approach this problem in a
more information theoretic way by investigating applicability of entropy
measures as a classification model objective function. We focus on quadratic
Renyi's entropy and connected Cauchy-Schwarz Divergence which leads to the
construction of Extreme Entropy Machines (EEM).
The main contribution of this paper is proposing a model based on the
information theoretic concepts which on the one hand shows new, entropic
perspective on known linear classifiers and on the other leads to a
construction of very robust method competetitive with the state of the art
non-information theoretic ones (including Support Vector Machines and Extreme
Learning Machines).
Evaluation on numerous problems spanning from small, simple ones from UCI
repository to the large (hundreads of thousands of samples) extremely
unbalanced (up to 100:1 classes' ratios) datasets shows wide applicability of
the EEM in real life problems and that it scales well
Designing labeled graph classifiers by exploiting the R\'enyi entropy of the dissimilarity representation
Representing patterns as labeled graphs is becoming increasingly common in
the broad field of computational intelligence. Accordingly, a wide repertoire
of pattern recognition tools, such as classifiers and knowledge discovery
procedures, are nowadays available and tested for various datasets of labeled
graphs. However, the design of effective learning procedures operating in the
space of labeled graphs is still a challenging problem, especially from the
computational complexity viewpoint. In this paper, we present a major
improvement of a general-purpose classifier for graphs, which is conceived on
an interplay between dissimilarity representation, clustering,
information-theoretic techniques, and evolutionary optimization algorithms. The
improvement focuses on a specific key subroutine devised to compress the input
data. We prove different theorems which are fundamental to the setting of the
parameters controlling such a compression operation. We demonstrate the
effectiveness of the resulting classifier by benchmarking the developed
variants on well-known datasets of labeled graphs, considering as distinct
performance indicators the classification accuracy, computing time, and
parsimony in terms of structural complexity of the synthesized classification
models. The results show state-of-the-art standards in terms of test set
accuracy and a considerable speed-up for what concerns the computing time.Comment: Revised versio
Information Theoretical Estimators Toolbox
We present ITE (information theoretical estimators) a free and open source,
multi-platform, Matlab/Octave toolbox that is capable of estimating many
different variants of entropy, mutual information, divergence, association
measures, cross quantities, and kernels on distributions. Thanks to its highly
modular design, ITE supports additionally (i) the combinations of the
estimation techniques, (ii) the easy construction and embedding of novel
information theoretical estimators, and (iii) their immediate application in
information theoretical optimization problems. ITE also includes a prototype
application in a central problem class of signal processing, independent
subspace analysis and its extensions.Comment: 5 pages; ITE toolbox: https://bitbucket.org/szzoli/ite
- …