51,423 research outputs found
Block stochastic gradient iteration for convex and nonconvex optimization
The stochastic gradient (SG) method can minimize an objective function
composed of a large number of differentiable functions, or solve a stochastic
optimization problem, to a moderate accuracy. The block coordinate
descent/update (BCD) method, on the other hand, handles problems with multiple
blocks of variables by updating them one at a time; when the blocks of
variables are easier to update individually than together, BCD has a lower
per-iteration cost. This paper introduces a method that combines the features
of SG and BCD for problems with many components in the objective and with
multiple (blocks of) variables.
Specifically, a block stochastic gradient (BSG) method is proposed for
solving both convex and nonconvex programs. At each iteration, BSG approximates
the gradient of the differentiable part of the objective by randomly sampling a
small set of data or sampling a few functions from the sum term in the
objective, and then, using those samples, it updates all the blocks of
variables in either a deterministic or a randomly shuffled order. Its
convergence for both convex and nonconvex cases are established in different
senses. In the convex case, the proposed method has the same order of
convergence rate as the SG method. In the nonconvex case, its convergence is
established in terms of the expected violation of a first-order optimality
condition. The proposed method was numerically tested on problems including
stochastic least squares and logistic regression, which are convex, as well as
low-rank tensor recovery and bilinear logistic regression, which are nonconvex
An Alternating Direction Algorithm for Matrix Completion with Nonnegative Factors
This paper introduces an algorithm for the nonnegative matrix
factorization-and-completion problem, which aims to find nonnegative low-rank
matrices X and Y so that the product XY approximates a nonnegative data matrix
M whose elements are partially known (to a certain accuracy). This problem
aggregates two existing problems: (i) nonnegative matrix factorization where
all entries of M are given, and (ii) low-rank matrix completion where
nonnegativity is not required. By taking the advantages of both nonnegativity
and low-rankness, one can generally obtain superior results than those of just
using one of the two properties. We propose to solve the non-convex constrained
least-squares problem using an algorithm based on the classic alternating
direction augmented Lagrangian method. Preliminary convergence properties of
the algorithm and numerical simulation results are presented. Compared to a
recent algorithm for nonnegative matrix factorization, the proposed algorithm
produces factorizations of similar quality using only about half of the matrix
entries. On tasks of recovering incomplete grayscale and hyperspectral images,
the proposed algorithm yields overall better qualities than those produced by
two recent matrix-completion algorithms that do not exploit nonnegativity
A fast patch-dictionary method for whole image recovery
Various algorithms have been proposed for dictionary learning. Among those
for image processing, many use image patches to form dictionaries. This paper
focuses on whole-image recovery from corrupted linear measurements. We address
the open issue of representing an image by overlapping patches: the overlapping
leads to an excessive number of dictionary coefficients to determine. With very
few exceptions, this issue has limited the applications of image-patch methods
to the local kind of tasks such as denoising, inpainting, cartoon-texture
decomposition, super-resolution, and image deblurring, for which one can
process a few patches at a time. Our focus is global imaging tasks such as
compressive sensing and medical image recovery, where the whole image is
encoded together, making it either impossible or very ineffective to update a
few patches at a time.
Our strategy is to divide the sparse recovery into multiple subproblems, each
of which handles a subset of non-overlapping patches, and then the results of
the subproblems are averaged to yield the final recovery. This simple strategy
is surprisingly effective in terms of both quality and speed. In addition, we
accelerate computation of the learned dictionary by applying a recent block
proximal-gradient method, which not only has a lower per-iteration complexity
but also takes fewer iterations to converge, compared to the current
state-of-the-art. We also establish that our algorithm globally converges to a
stationary point. Numerical results on synthetic data demonstrate that our
algorithm can recover a more faithful dictionary than two state-of-the-art
methods.
Combining our whole-image recovery and dictionary-learning methods, we
numerically simulate image inpainting, compressive sensing recovery, and
deblurring. Our recovery is more faithful than those of a total variation
method and a method based on overlapping patches
On the Computation Power of Name Parameterization in Higher-order Processes
Parameterization extends higher-order processes with the capability of
abstraction (akin to that in lambda-calculus), and is known to be able to
enhance the expressiveness. This paper focuses on the parameterization of
names, i.e. a construct that maps a name to a process, in the higher-order
setting. We provide two results concerning its computation capacity. First,
name parameterization brings up a complete model, in the sense that it can
express an elementary interactive model with built-in recursive functions.
Second, we compare name parameterization with the well-known pi-calculus, and
provide two encodings between them.Comment: In Proceedings ICE 2015, arXiv:1508.0459
- …