Improved Hessian approximation with modified quasi-Cauchy relation for a gradient-type method

In this work we develop a new gradient-type method with improved Hessian approximation for unconstrained optimization problems. The new method resembles the Barzilai-Borwein (BB) method, except that the Hessian matrix is approximated by a diagonal matrix rather than the multiple of the identity matrix in the BB method. Then the diagonal Hessian approximation is derived based on the quasi-Cauchy relation. To further improve the Hessian approximation, we modify the quasi-Cauchy relation to carry some additional information from the values and gradients of the objective function. Numerical experiments show that the proposed method yields desirable improvement

Globalization of Barzilai and Borwein Method for Unconstrained Optimization

The focus of this thesis is on finding the unconstrained minimizer of a function. Specifically, we will focus on the Barzilai and Borwein (BB) method that is a famous two-point stepsize gradient method. First we briefly give some mathematical background. Then we discuss the (BB) method that is important in the area of optimization. A review of the minimization methods currently available that can be used to solve unconstrained optimization is also given. Due to BB method’s simplicity, low storage and numerical efficiency, the Barzilai and Borwein method has received a good deal of attention in the optimization community but despite all these advances, stepsize of BB method is computed by means of simple approximation of Hessian in the form of scalar multiple of identity and especially the BB method is not monotone, and it is not easy to generalize the method to general nonlinear functions. Due to the presence of these deficiencies, we introduce new gradient-type methods in the frame of BB method including a new gradient method via weak secant equation (quasi-Cauchy relation), improved Hessian approximation and scaling the diagonal updating. The proposed methods are a kind of fixed step gradient method like that of Barzilai and Borwein method. In contrast with the Barzilai and Borwein approach’s in which stepsize is computed by means of simple approximation of the Hessian in the form of scalar multiple of identity, the proposed methods consider approximation of Hessian in diagonal matrix. Incorporate with monotone strategies, the resulting algorithms belong to the class of monotone gradient methods with globally convergence. Numerical results suggest that for non-quadratic minimization problem, the new methods clearly outperform the Barzilai- Borwein method. Finally we comment on some achievement in our researches. Possible extensions are also given to conclude this thesis

On Functions of quasi Toeplitz matrices

Let $a(z)=\sum_{i\in\mathbb Z}a_iz^i$ be a complex valued continuous function, defined for $|z|=1$, such that $\sum_{i=-\infty}^{+\infty}|ia_i|<\infty$. Consider the semi-infinite Toeplitz matrix $T(a)=(t_{i,j})_{i,j\in\mathbb Z^+}$ associated with the symbol $a(z)$ such that $t_{i,j}=a_{j-i}$. A quasi-Toeplitz matrix associated with the continuous symbol $a(z)$ is a matrix of the form $A=T(a)+E$ where $E=(e_{i,j})$, $\sum_{i,j\in\mathbb Z^+}|e_{i,j}|<\infty$, and is called a CQT-matrix. Given a function $f(x)$ and a CQT matrix $M$, we provide conditions under which $f(M)$ is well defined and is a CQT matrix. Moreover, we introduce a parametrization of CQT matrices and algorithms for the computation of $f(M)$. We treat the case where $f(x)$ is assigned in terms of power series and the case where $f(x)$ is defined in terms of a Cauchy integral. This analysis is applied also to finite matrices which can be written as the sum of a Toeplitz matrix and of a low rank correction

A quasi-Newton proximal splitting method

A new result in convex analysis on the calculation of proximity operators in certain scaled norms is derived. We describe efficient implementations of the proximity calculation for a useful class of functions; the implementations exploit the piece-wise linear nature of the dual problem. The second part of the paper applies the previous result to acceleration of convex minimization problems, and leads to an elegant quasi-Newton method. The optimization method compares favorably against state-of-the-art alternatives. The algorithm has extensive applications including signal processing, sparse recovery and machine learning and classification

Improved Diagonal Hessian Approximations for Large-Scale Unconstrained Optimization

We consider some diagonal quasi-Newton methods for solving large-scale unconstrained optimization problems. A simple and effective approach for diagonal quasi-Newton algorithms is presented by proposing new updates of diagonal entries of the Hessian. Moreover, we suggest employing an extra BFGS update of the diagonal updating matrix and use its diagonal again. Numerical experiments on a collection of standard test problems show, in particular, that the proposed diagonal quasi-Newton methods perform substantially better than certain available diagonal methods

Newton based Stochastic Optimization using q-Gaussian Smoothed Functional Algorithms

We present the first q-Gaussian smoothed functional (SF) estimator of the Hessian and the first Newton-based stochastic optimization algorithm that estimates both the Hessian and the gradient of the objective function using q-Gaussian perturbations. Our algorithm requires only two system simulations (regardless of the parameter dimension) and estimates both the gradient and the Hessian at each update epoch using these. We also present a proof of convergence of the proposed algorithm. In a related recent work (Ghoshdastidar et al., 2013), we presented gradient SF algorithms based on the q-Gaussian perturbations. Our work extends prior work on smoothed functional algorithms by generalizing the class of perturbation distributions as most distributions reported in the literature for which SF algorithms are known to work and turn out to be special cases of the q-Gaussian distribution. Besides studying the convergence properties of our algorithm analytically, we also show the results of several numerical simulations on a model of a queuing network, that illustrate the significance of the proposed method. In particular, we observe that our algorithm performs better in most cases, over a wide range of q-values, in comparison to Newton SF algorithms with the Gaussian (Bhatnagar, 2007) and Cauchy perturbations, as well as the gradient q-Gaussian SF algorithms (Ghoshdastidar et al., 2013).Comment: This is a longer of version of the paper with the same title accepted in Automatic

Constraint-preserving Sommerfeld conditions for the harmonic Einstein equations

The principle part of Einstein equations in the harmonic gauge consists of a constrained system of 10 curved space wave equations for the components of the space-time metric. A new formulation of constraint-preserving boundary conditions of the Sommerfeld type for such systems has recently been proposed. We implement these boundary conditions in a nonlinear 3D evolution code and test their accuracy.Comment: 16 pages, 17 figures, submitted to Phys. Rev.

VIPSCAL: A combined vector ideal point model for preference data

In this paper, we propose a new model that combines the vector model and theideal point model of unfolding. An algorithm is developed, called VIPSCAL, thatminimizes the combined loss both for ordinal and interval transformations. As such,mixed representations including both vectors and ideal points can be obtained butthe algorithm also allows for the unmixed cases, giving either a complete idealpointanalysis or a complete vector analysis. On the basis of previous research,the mixed representations were expected to be nondegenerate. However, degeneratesolutions still occurred as the common belief that distant ideal points can be represented by vectors does not hold true. The occurrence of these distant ideal points was solved by adding certain length and orthogonality restrictions on the configuration. The restrictions can be used both for the mixed and unmixed cases in several ways such that a number of different models can be fitted by VIPSCAL.unfolding;ideal point model;vector model