5,085 research outputs found
L0 Sparse Inverse Covariance Estimation
Recently, there has been focus on penalized log-likelihood covariance
estimation for sparse inverse covariance (precision) matrices. The penalty is
responsible for inducing sparsity, and a very common choice is the convex
norm. However, the best estimator performance is not always achieved with this
penalty. The most natural sparsity promoting "norm" is the non-convex
penalty but its lack of convexity has deterred its use in sparse maximum
likelihood estimation. In this paper we consider non-convex penalized
log-likelihood inverse covariance estimation and present a novel cyclic descent
algorithm for its optimization. Convergence to a local minimizer is proved,
which is highly non-trivial, and we demonstrate via simulations the reduced
bias and superior quality of the penalty as compared to the
penalty
Pairwise MRF Calibration by Perturbation of the Bethe Reference Point
We investigate different ways of generating approximate solutions to the
pairwise Markov random field (MRF) selection problem. We focus mainly on the
inverse Ising problem, but discuss also the somewhat related inverse Gaussian
problem because both types of MRF are suitable for inference tasks with the
belief propagation algorithm (BP) under certain conditions. Our approach
consists in to take a Bethe mean-field solution obtained with a maximum
spanning tree (MST) of pairwise mutual information, referred to as the
\emph{Bethe reference point}, for further perturbation procedures. We consider
three different ways following this idea: in the first one, we select and
calibrate iteratively the optimal links to be added starting from the Bethe
reference point; the second one is based on the observation that the natural
gradient can be computed analytically at the Bethe point; in the third one,
assuming no local field and using low temperature expansion we develop a dual
loop joint model based on a well chosen fundamental cycle basis. We indeed
identify a subclass of planar models, which we refer to as \emph{Bethe-dual
graph models}, having possibly many loops, but characterized by a singly
connected dual factor graph, for which the partition function and the linear
response can be computed exactly in respectively O(N) and operations,
thanks to a dual weight propagation (DWP) message passing procedure that we set
up. When restricted to this subclass of models, the inverse Ising problem being
convex, becomes tractable at any temperature. Experimental tests on various
datasets with refined or regularization procedures indicate that
these approaches may be competitive and useful alternatives to existing ones.Comment: 54 pages, 8 figure. section 5 and refs added in V
Conditional Gradient Algorithms for Rank-One Matrix Approximations with a Sparsity Constraint
The sparsity constrained rank-one matrix approximation problem is a difficult
mathematical optimization problem which arises in a wide array of useful
applications in engineering, machine learning and statistics, and the design of
algorithms for this problem has attracted intensive research activities. We
introduce an algorithmic framework, called ConGradU, that unifies a variety of
seemingly different algorithms that have been derived from disparate
approaches, and allows for deriving new schemes. Building on the old and
well-known conditional gradient algorithm, ConGradU is a simplified version
with unit step size and yields a generic algorithm which either is given by an
analytic formula or requires a very low computational complexity. Mathematical
properties are systematically developed and numerical experiments are given.Comment: Minor changes. Final version. To appear in SIAM Revie
- …