Stochastic Optimization with Importance Sampling

Uniform sampling of training data has been commonly used in traditional stochastic optimization algorithms such as Proximal Stochastic Gradient Descent (prox-SGD) and Proximal Stochastic Dual Coordinate Ascent (prox-SDCA). Although uniform sampling can guarantee that the sampled stochastic quantity is an unbiased estimate of the corresponding true quantity, the resulting estimator may have a rather high variance, which negatively affects the convergence of the underlying optimization procedure. In this paper we study stochastic optimization with importance sampling, which improves the convergence rate by reducing the stochastic variance. Specifically, we study prox-SGD (actually, stochastic mirror descent) with importance sampling and prox-SDCA with importance sampling. For prox-SGD, instead of adopting uniform sampling throughout the training process, the proposed algorithm employs importance sampling to minimize the variance of the stochastic gradient. For prox-SDCA, the proposed importance sampling scheme aims to achieve higher expected dual value at each dual coordinate ascent step. We provide extensive theoretical analysis to show that the convergence rates with the proposed importance sampling methods can be significantly improved under suitable conditions both for prox-SGD and for prox-SDCA. Experiments are provided to verify the theoretical analysis.Comment: 29 page

Sweeping process by prox-regular sets in Riemannian Hilbert manifolds

In this paper, we deal with sweeping processes on (possibly infinite-dimensional) Riemannian Hilbert manifolds. We extend the useful notions (proximal normal cone, prox-regularity) already defined in the setting of a Hilbert space to the framework of such manifolds. Especially we introduce the concept of local prox-regularity of a closed subset in accordance with the geometrical features of the ambient manifold and we check that this regularity implies a property of hypomonotonicity for the proximal normal cone. Moreover we show that the metric projection onto a locally prox-regular set is single-valued in its neighborhood. Then under some assumptions, we prove the well-posedness of perturbed sweeping processes by locally prox-regular sets.Comment: 27 page

Thermal food processing computation software

The objective of this research consisted of developing the two following thermal food processing software: “F-CALC” is software developed to carry out thermal process calculations based on the well-known Ball's formula method, and “OPT-PROx” is software for thermal food processing optimization based on variable retort temperature processing and global optimization technique. Time-temperature data loaded from Excel-file is used by “F-CALC” software to evaluate the heat penetration parameters jh and fh, as well as to compute process lethality for given process time or vice versa. The possibility of computing the process time and lethality for broken heating curves is included. The diversity of thermal food processing optimization problems with different objectives and required constraints are solvable by “OPT-PROx” software. The adaptive random search algorithm coupled with penalty functions approach, and the finite difference method with cubic spline approximation are utilized by “OPT-PROx” for simulation and optimization thermal food processes. The possibility of estimating the thermal diffusivity coefficient based on the mean squared error function minimization is included. The “OPT-PROx” software was successfully tested on the real thermal food processing problems, namely in the case of total process time minimization with a constraint for average and surface retentions the “OPT-PROx” demonstrates significant advantage over the traditional constant temperature processes in terms of process time and final product quality. The developed user friendly dialogue and used numerical procedures make the “F-CALC” and “OPT-PROx” software extremely useful for food scientists (research and education) and engineers (real thermal food process evaluation and optimization)

Proximal Stochastic Newton-type Gradient Descent Methods for Minimizing Regularized Finite Sums

In this work, we generalized and unified recent two completely different works of Jascha \cite{sohl2014fast} and Lee \cite{lee2012proximal} respectively into one by proposing the \textbf{prox}imal s\textbf{to}chastic \textbf{N}ewton-type gradient (PROXTONE) method for optimizing the sums of two convex functions: one is the average of a huge number of smooth convex functions, and the other is a non-smooth convex function. While a set of recently proposed proximal stochastic gradient methods, include MISO, Prox-SDCA, Prox-SVRG, and SAG, converge at linear rates, the PROXTONE incorporates second order information to obtain stronger convergence results, that it achieves a linear convergence rate not only in the value of the objective function, but also in the \emph{solution}. The proof is simple and intuitive, and the results and technique can be served as a initiate for the research on the proximal stochastic methods that employ second order information.Comment: arXiv admin note: text overlap with arXiv:1309.2388, arXiv:1403.4699 by other author

Solving Variational Inequalities with Monotone Operators on Domains Given by Linear Minimization Oracles

The standard algorithms for solving large-scale convex-concave saddle point problems, or, more generally, variational inequalities with monotone operators, are proximal type algorithms which at every iteration need to compute a prox-mapping, that is, to minimize over problem's domain $X$ the sum of a linear form and the specific convex distance-generating function underlying the algorithms in question. Relative computational simplicity of prox-mappings, which is the standard requirement when implementing proximal algorithms, clearly implies the possibility to equip $X$ with a relatively computationally cheap Linear Minimization Oracle (LMO) able to minimize over $X$ linear forms. There are, however, important situations where a cheap LMO indeed is available, but where no proximal setup with easy-to-compute prox-mappings is known. This fact motivates our goal in this paper, which is to develop techniques for solving variational inequalities with monotone operators on domains given by Linear Minimization Oracles. The techniques we develope can be viewed as a substantial extension of the proposed in [5] method of nonsmooth convex minimization over an LMO-represented domain