The relationship between the Jacobi and the successive overrelaxation (SOR) matrices of a k-cyclic matrix

AbstractLet A be a (k−l, l)-generalized consistently ordered matrix with T and Lω its associated Jacobi and SOR matrices whose eigenvalues μ and λ satisfy the well-known relationship (λ+ω−1)k=ωkμkλk−1. For a subclass of the above matrices A we prove that the matrix analogue of the previous relationship holds. Exploiting the matrix relationship we show that the SOR method is equivalent to a certain monoparametric k-step iterative one when used for the solution of the fixed-point problem x=Tx+c

On some extensions of the accelerated overrelaxation (AOR) theory

This paper extends the convergence theory of the Accelerated Overrelaxation (AOR) method to cases analogous to those considered first by Ostrowski and then by Varga in connection with the Successive Overrelaxation (SOR) method. Among others, the Ostrowski Theorem, some of the theorems by Varga on the extensions of the SOR theory, and some recent results by Niethammer and by the authors are obtained as special cases of the work presented in this paper. In addition, several points are raised which suggest further research

Deep Bilevel Learning

We present a novel regularization approach to train neural networks that enjoys better generalization and test error than standard stochastic gradient descent. Our approach is based on the principles of cross-validation, where a validation set is used to limit the model overfitting. We formulate such principles as a bilevel optimization problem. This formulation allows us to define the optimization of a cost on the validation set subject to another optimization on the training set. The overfitting is controlled by introducing weights on each mini-batch in the training set and by choosing their values so that they minimize the error on the validation set. In practice, these weights define mini-batch learning rates in a gradient descent update equation that favor gradients with better generalization capabilities. Because of its simplicity, this approach can be integrated with other regularization methods and training schemes. We evaluate extensively our proposed algorithm on several neural network architectures and datasets, and find that it consistently improves the generalization of the model, especially when labels are noisy.Comment: ECCV 201

Optimal p-Cyclic SOR for Complex Spectra

In this work we consider the Successive Overrelaxation (SOR) method for the so-lution of a linear system Ax = b, when the matrix A has a block p X P partitioned p-cyclic form and its associated block Jacobi matrix Jp 1s weaJdy cyclic of index p. Following the pioneering work by Young and Varga in the 50s many researchers have considered various cases for the spectrum 0'(Jp) and have determlned (optimal) values for the relaxation factor w E (0,2) so that the SOR method converges as fast as pos-sible. After l.he most recent work on the best block p-cyclic repartitionlng and that on the solution of large scale systems arising in queueing network problems in Markov analysis, the optimization of the convergence of the p-cyclic SOR for more complex spectra (J'(Jp) has become more demanding. Here we state the "one-point " problem for the general p-cyclic complex SOR case. The existence and the uniqueness of its solu-tion are established by analyzing and developing further the theory of the associated hypocycloidal curves. For the determination of the optimal parameter(s) an algorithm is presented and a number of illustrative numerical examples are given