11 research outputs found
Heuristic parameter-choice rules for convex variational regularization based on error estimates
In this paper, we are interested in heuristic parameter choice rules for
general convex variational regularization which are based on error estimates.
Two such rules are derived and generalize those from quadratic regularization,
namely the Hanke-Raus rule and quasi-optimality criterion. A posteriori error
estimates are shown for the Hanke-Raus rule, and convergence for both rules is
also discussed. Numerical results for both rules are presented to illustrate
their applicability
A new approach to nonlinear constrained Tikhonov regularization
We present a novel approach to nonlinear constrained Tikhonov regularization
from the viewpoint of optimization theory. A second-order sufficient optimality
condition is suggested as a nonlinearity condition to handle the nonlinearity
of the forward operator. The approach is exploited to derive convergence rates
results for a priori as well as a posteriori choice rules, e.g., discrepancy
principle and balancing principle, for selecting the regularization parameter.
The idea is further illustrated on a general class of parameter identification
problems, for which (new) source and nonlinearity conditions are derived and
the structural property of the nonlinearity term is revealed. A number of
examples including identifying distributed parameters in elliptic differential
equations are presented.Comment: 21 pages, to appear in Inverse Problem
On the Regularizing Property of Stochastic Gradient Descent
Stochastic gradient descent is one of the most successful approaches for
solving large-scale problems, especially in machine learning and statistics. At
each iteration, it employs an unbiased estimator of the full gradient computed
from one single randomly selected data point. Hence, it scales well with
problem size and is very attractive for truly massive dataset, and holds
significant potentials for solving large-scale inverse problems. In the recent
literature of machine learning, it was empirically observed that when equipped
with early stopping, it has regularizing property. In this work, we rigorously
establish its regularizing property (under \textit{a priori} early stopping
rule), and also prove convergence rates under the canonical sourcewise
condition, for minimizing the quadratic functional for linear inverse problems.
This is achieved by combining tools from classical regularization theory and
stochastic analysis. Further, we analyze the preasymptotic weak and strong
convergence behavior of the algorithm. The theoretical findings shed insights
into the performance of the algorithm, and are complemented with illustrative
numerical experiments.Comment: 22 pages, better presentatio
On the regularizing property of stochastic gradient descent
Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments