40,286 research outputs found
Adaptive Momentum for Neural Network Optimization
In this thesis, we develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyaks Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental results on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. We believe that our research will open up new directions for high-dimensional neural network optimization where combining the efficiency of first-order methods and the effectiveness of second-order methods proves a promising avenue to explore
A Non-Monotone Conjugate Subgradient Type Method for Minimization of Convex Functions
We suggest a conjugate subgradient type method without any line-search for
minimization of convex non differentiable functions. Unlike the custom methods
of this class, it does not require monotone decrease of the goal function and
reduces the implementation cost of each iteration essentially. At the same
time, its step-size procedure takes into account behavior of the method along
the iteration points. Preliminary results of computational experiments confirm
efficiency of the proposed modification.Comment: 11 page
Bridging Proper Orthogonal Decomposition methods and augmented Newton-Krylov algorithms: an adaptive model order reduction for highly nonlinear mechanical problems
This article describes a bridge between POD-based model order reduction
techniques and the classical Newton/Krylov solvers. This bridge is used to
derive an efficient algorithm to correct, "on-the-fly", the reduced order
modelling of highly nonlinear problems undergoing strong topological changes.
Damage initiation problems are addressed and tackle via a corrected
hyperreduction method. It is shown that the relevancy of reduced order model
can be significantly improved with reasonable additional costs when using this
algorithm, even when strong topological changes are involved
Efficient Gaussian Sampling for Solving Large-Scale Inverse Problems using MCMC Methods
The resolution of many large-scale inverse problems using MCMC methods
requires a step of drawing samples from a high dimensional Gaussian
distribution. While direct Gaussian sampling techniques, such as those based on
Cholesky factorization, induce an excessive numerical complexity and memory
requirement, sequential coordinate sampling methods present a low rate of
convergence. Based on the reversible jump Markov chain framework, this paper
proposes an efficient Gaussian sampling algorithm having a reduced computation
cost and memory usage. The main feature of the algorithm is to perform an
approximate resolution of a linear system with a truncation level adjusted
using a self-tuning adaptive scheme allowing to achieve the minimal computation
cost. The connection between this algorithm and some existing strategies is
discussed and its efficiency is illustrated on a linear inverse problem of
image resolution enhancement.Comment: 20 pages, 10 figures, under review for journal publicatio
Low-complexity RLS algorithms using dichotomous coordinate descent iterations
In this paper, we derive low-complexity recursive least squares (RLS) adaptive filtering algorithms. We express the RLS problem in terms of auxiliary normal equations with respect to increments of the filter weights and apply this approach to the exponentially weighted and sliding window cases to derive new RLS techniques. For solving the auxiliary equations, line search methods are used. We first consider conjugate gradient iterations with a complexity of O(N-2) operations per sample; N being the number of the filter weights. To reduce the complexity and make the algorithms more suitable for finite precision implementation, we propose a new dichotomous coordinate descent (DCD) algorithm and apply it to the auxiliary equations. This results in a transversal RLS adaptive filter with complexity as low as 3N multiplications per sample, which is only slightly higher than the complexity of the least mean squares (LMS) algorithm (2N multiplications). Simulations are used to compare the performance of the proposed algorithms against the classical RLS and known advanced adaptive algorithms. Fixed-point FPGA implementation of the proposed DCD-based RLS algorithm is also discussed and results of such implementation are presented
An adaptive finite element method in reconstruction of coefficients in Maxwell's equations from limited observations
We propose an adaptive finite element method for the solution of a
coefficient inverse problem of simultaneous reconstruction of the dielectric
permittivity and magnetic permeability functions in the Maxwell's system using
limited boundary observations of the electric field in 3D. We derive a
posteriori error estimates in the Tikhonov functional to be minimized and in
the regularized solution of this functional, as well as formulate corresponding
adaptive algorithm. Our numerical experiments justify the efficiency of our a
posteriori estimates and show significant improvement of the reconstructions
obtained on locally adaptively refined meshes.Comment: Corrected typo
Robust Adaptive LCMV Beamformer Based On An Iterative Suboptimal Solution
The main drawback of closed-form solution of linearly constrained minimum variance (CF-LCMV) beamformer is the dilemma of acquiring long observation time for stable covariance matrix estimates and short observation time to track dynamic behavior of targets, leading to poor performance including low signal-noise-ratio (SNR), low jammer-to-noise ratios (JNRs) and small number of snapshots. Additionally, CF-LCMV suffers from heavy computational burden which mainly comes from two matrix inverse operations for computing the optimal weight vector. In this paper, we derive a low-complexity Robust Adaptive LCMV beamformer based on an Iterative Suboptimal solution (RAIS-LCMV) using conjugate gradient (CG) optimization method. The merit of our proposed method is threefold. Firstly, RAIS-LCMV beamformer can reduce the complexity of CF-LCMV remarkably. Secondly, RAIS-LCMV beamformer can adjust output adaptively based on measurement and its convergence speed is comparable. Finally, RAIS-LCMV algorithm has robust performance against low SNR, JNRs, and small number of snapshots. Simulation results demonstrate the superiority of our proposed algorithms
- …