Stochastic Optimization with Importance Sampling

Uniform sampling of training data has been commonly used in traditional stochastic optimization algorithms such as Proximal Stochastic Gradient Descent (prox-SGD) and Proximal Stochastic Dual Coordinate Ascent (prox-SDCA). Although uniform sampling can guarantee that the sampled stochastic quantity is an unbiased estimate of the corresponding true quantity, the resulting estimator may have a rather high variance, which negatively affects the convergence of the underlying optimization procedure. In this paper we study stochastic optimization with importance sampling, which improves the convergence rate by reducing the stochastic variance. Specifically, we study prox-SGD (actually, stochastic mirror descent) with importance sampling and prox-SDCA with importance sampling. For prox-SGD, instead of adopting uniform sampling throughout the training process, the proposed algorithm employs importance sampling to minimize the variance of the stochastic gradient. For prox-SDCA, the proposed importance sampling scheme aims to achieve higher expected dual value at each dual coordinate ascent step. We provide extensive theoretical analysis to show that the convergence rates with the proposed importance sampling methods can be significantly improved under suitable conditions both for prox-SGD and for prox-SDCA. Experiments are provided to verify the theoretical analysis.Comment: 29 page

Feasibility studies of a converter-free grid-connected offshore hydrostatic wind turbine

Owing to the increasing penetration of renewable power generation, the modern power system faces great challenges in frequency regulations and reduced system inertia. Hence, renewable energy is expected to take over part of the frequency regulation responsibilities from the gas or hydro plants and contribute to the system inertia. In this article, we investigate the feasibility of frequency regulation by the offshore hydrostatic wind turbine (HWT). The simulation model is transformed from NREL (National Renewable Energy Laboratory) 5-MW gearbox-equipped wind turbine model within FAST (fatigue, aerodynamics, structures, and turbulence) code. With proposed coordinated control scheme and the hydrostatic transmission configuration of the HWT, the `continuously variable gearbox ratio' in turbulent wind conditions can be realised to maintain the constant generator speed, so that the HWT can be connected to the grid without power converters in-between. To test the performances of the control scheme, the HWT is connected to a 5-bus grid model and operates with different frequency events. The simulation results indicate that the proposed control scheme is a promising solution for offshore HWT to participated in frequency response in the modern power system

Pathwise Coordinate Optimization for Sparse Learning: Algorithm and Theory

The pathwise coordinate optimization is one of the most important computational frameworks for high dimensional convex and nonconvex sparse learning problems. It differs from the classical coordinate optimization algorithms in three salient features: {\it warm start initialization}, {\it active set updating}, and {\it strong rule for coordinate preselection}. Such a complex algorithmic structure grants superior empirical performance, but also poses significant challenge to theoretical analysis. To tackle this long lasting problem, we develop a new theory showing that these three features play pivotal roles in guaranteeing the outstanding statistical and computational performance of the pathwise coordinate optimization framework. Particularly, we analyze the existing pathwise coordinate optimization algorithms and provide new theoretical insights into them. The obtained insights further motivate the development of several modifications to improve the pathwise coordinate optimization framework, which guarantees linear convergence to a unique sparse local optimum with optimal statistical properties in parameter estimation and support recovery. This is the first result on the computational and statistical guarantees of the pathwise coordinate optimization framework in high dimensions. Thorough numerical experiments are provided to support our theory.Comment: Accepted by the Annals of Statistics, 2016