24 research outputs found
Globally Convergent Accelerated Algorithms for Multilinear Sparse Logistic Regression with -constraints
Tensor data represents a multidimensional array. Regression methods based on
low-rank tensor decomposition leverage structural information to reduce the
parameter count. Multilinear logistic regression serves as a powerful tool for
the analysis of multidimensional data. To improve its efficacy and
interpretability, we present a Multilinear Sparse Logistic Regression model
with -constraints (-MLSR). In contrast to the -norm and
-norm, the -norm constraint is better suited for feature
selection. However, due to its nonconvex and nonsmooth properties, solving it
is challenging and convergence guarantees are lacking. Additionally, the
multilinear operation in -MLSR also brings non-convexity. To tackle
these challenges, we propose an Accelerated Proximal Alternating Linearized
Minimization with Adaptive Momentum (APALM) method to solve the
-MLSR model. We provide a proof that APALM can ensure the
convergence of the objective function of -MLSR. We also demonstrate
that APALM is globally convergent to a first-order critical point as well
as establish convergence rate by using the Kurdyka-Lojasiewicz property.
Empirical results obtained from synthetic and real-world datasets validate the
superior performance of our algorithm in terms of both accuracy and speed
compared to other state-of-the-art methods.Comment: arXiv admin note: text overlap with arXiv:2308.1212
Low Complexity Regularization of Linear Inverse Problems
Inverse problems and regularization theory is a central theme in contemporary
signal processing, where the goal is to reconstruct an unknown signal from
partial indirect, and possibly noisy, measurements of it. A now standard method
for recovering the unknown signal is to solve a convex optimization problem
that enforces some prior knowledge about its structure. This has proved
efficient in many problems routinely encountered in imaging sciences,
statistics and machine learning. This chapter delivers a review of recent
advances in the field where the regularization prior promotes solutions
conforming to some notion of simplicity/low-complexity. These priors encompass
as popular examples sparsity and group sparsity (to capture the compressibility
of natural signals and images), total variation and analysis sparsity (to
promote piecewise regularity), and low-rank (as natural extension of sparsity
to matrix-valued data). Our aim is to provide a unified treatment of all these
regularizations under a single umbrella, namely the theory of partial
smoothness. This framework is very general and accommodates all low-complexity
regularizers just mentioned, as well as many others. Partial smoothness turns
out to be the canonical way to encode low-dimensional models that can be linear
spaces or more general smooth manifolds. This review is intended to serve as a
one stop shop toward the understanding of the theoretical properties of the
so-regularized solutions. It covers a large spectrum including: (i) recovery
guarantees and stability to noise, both in terms of -stability and
model (manifold) identification; (ii) sensitivity analysis to perturbations of
the parameters involved (in particular the observations), with applications to
unbiased risk estimation ; (iii) convergence properties of the forward-backward
proximal splitting scheme, that is particularly well suited to solve the
corresponding large-scale regularized optimization problem
An Accelerated Block Proximal Framework with Adaptive Momentum for Nonconvex and Nonsmooth Optimization
We propose an accelerated block proximal linear framework with adaptive
momentum (ABPL) for nonconvex and nonsmooth optimization. We analyze the
potential causes of the extrapolation step failing in some algorithms, and
resolve this issue by enhancing the comparison process that evaluates the
trade-off between the proximal gradient step and the linear extrapolation step
in our algorithm. Furthermore, we extends our algorithm to any scenario
involving updating block variables with positive integers, allowing each cycle
to randomly shuffle the update order of the variable blocks. Additionally,
under mild assumptions, we prove that ABPL can monotonically decrease the
function value without strictly restricting the extrapolation parameters and
step size, demonstrates the viability and effectiveness of updating these
blocks in a random order, and we also more obviously and intuitively
demonstrate that the derivative set of the sequence generated by our algorithm
is a critical point set. Moreover, we demonstrate the global convergence as
well as the linear and sublinear convergence rates of our algorithm by
utilizing the Kurdyka-Lojasiewicz (K{\L}) condition. To enhance the
effectiveness and flexibility of our algorithm, we also expand the study to the
imprecise version of our algorithm and construct an adaptive extrapolation
parameter strategy, which improving its overall performance. We apply our
algorithm to multiple non-negative matrix factorization with the norm,
nonnegative tensor decomposition with the norm, and perform extensive
numerical experiments to validate its effectiveness and efficiency
Audio inpainting algorithms
Tato práce se zabývá doplňováním chybějících dat do audio signálů a algoritmy řešícími problém založenými na řídké reprezentaci audio signálu. Práce se zaměřuje na některé algoritmy, které řeší doplňování chybějících dat do audio signálů pomocí řídké reprezentace signálů. Součástí práce je také návrh algoritmu, který používá řídkou reprezentaci signálu a také nízkou hodnost signálu ve spektrogramu audio signálu. Dále práce uvádí implementaci tohoto algoritmu v programu Matlab a jeho vyhodnocení.The thesis deals with audio inpainting problem and sparse representation approaches to this problem. It focuses on some of recent approaches to solving audio inpainting problem with respect to sparse representation algorithms. It proposes solving audio inapinting problem based on sparse representation of signal and low rank structure in spectrogram of audio signal. Thesis also describes implementation in program Matlab and evaluation of the proposed method.
Compressive Wave Computation
This paper considers large-scale simulations of wave propagation phenomena.
We argue that it is possible to accurately compute a wavefield by decomposing
it onto a largely incomplete set of eigenfunctions of the Helmholtz operator,
chosen at random, and that this provides a natural way of parallelizing wave
simulations for memory-intensive applications.
This paper shows that L1-Helmholtz recovery makes sense for wave computation,
and identifies a regime in which it is provably effective: the one-dimensional
wave equation with coefficients of small bounded variation. Under suitable
assumptions we show that the number of eigenfunctions needed to evolve a sparse
wavefield defined on N points, accurately with very high probability, is
bounded by C log(N) log(log(N)), where C is related to the desired accuracy and
can be made to grow at a much slower rate than N when the solution is sparse.
The PDE estimates that underlie this result are new to the authors' knowledge
and may be of independent mathematical interest; they include an L1 estimate
for the wave equation, an estimate of extension of eigenfunctions, and a bound
for eigenvalue gaps in Sturm-Liouville problems.
Numerical examples are presented in one spatial dimension and show that as
few as 10 percents of all eigenfunctions can suffice for accurate results.
Finally, we argue that the compressive viewpoint suggests a competitive
parallel algorithm for an adjoint-state inversion method in reflection
seismology.Comment: 45 pages, 4 figure
Non-Smooth Regularization: Improvement to Learning Framework through Extrapolation
International audienceDeep learning architectures employ various regularization terms to handle different types of priors. Non-smooth regularization terms have shown promising performance in the deep learning architectures and a learning framework has recently been proposed to train autoencoders with such regularization terms. While this framework efficiently manages the non-smooth term during training through proximal operators, it is limited to autoencoders and suffers from low convergence speed due to several optimization sub-problems that must be solved in a row. In this paper, we address these issues by extending the framework to general feed-forward neural networks and introducing variable extrapolation which can dramatically increase the convergence speed in each sub-problem. We show that the proposed update rules converge to a critical point of the objective function under mild conditions. To compare the resulting framework with the previously proposed one, we consider the problem of training sparse autoencoders and robustifying deep neural architectures against both targeted and untargeted attacks. Simulations show superior performance in both convergence speed and final objective function value