66 research outputs found
Accelerating Deep Learning with Shrinkage and Recall
Deep Learning is a very powerful machine learning model. Deep Learning trains
a large number of parameters for multiple layers and is very slow when data is
in large scale and the architecture size is large. Inspired from the shrinking
technique used in accelerating computation of Support Vector Machines (SVM)
algorithm and screening technique used in LASSO, we propose a shrinking Deep
Learning with recall (sDLr) approach to speed up deep learning computation. We
experiment shrinking Deep Learning with recall (sDLr) using Deep Neural Network
(DNN), Deep Belief Network (DBN) and Convolution Neural Network (CNN) on 4 data
sets. Results show that the speedup using shrinking Deep Learning with recall
(sDLr) can reach more than 2.0 while still giving competitive classification
performance.Comment: The 22nd IEEE International Conference on Parallel and Distributed
Systems (ICPADS 2016
On Newton Screening
Screening and working set techniques are important approaches to reducing the
size of an optimization problem. They have been widely used in accelerating
first-order methods for solving large-scale sparse learning problems. In this
paper, we develop a new screening method called Newton screening (NS) which is
a generalized Newton method with a built-in screening mechanism. We derive an
equivalent KKT system for the Lasso and utilize a generalized Newton method to
solve the KKT equations. Based on this KKT system, a built-in working set with
a relatively small size is first determined using the sum of primal and dual
variables generated from the previous iteration, then the primal variable is
updated by solving a least-squares problem on the working set and the dual
variable updated based on a closed-form expression. Moreover, we consider a
sequential version of Newton screening (SNS) with a warm-start strategy. We
show that NS possesses an optimal convergence property in the sense that it
achieves one-step local convergence. Under certain regularity conditions on the
feature matrix, we show that SNS hits a solution with the same signs as the
underlying true target and achieves a sharp estimation error bound with high
probability. Simulation studies and real data analysis support our theoretical
results and demonstrate that SNS is faster and more accurate than several
state-of-the-art methods in our comparative studies
GAP Safe screening rules for sparse multi-task and multi-class models
High dimensional regression benefits from sparsity promoting regularizations.
Screening rules leverage the known sparsity of the solution by ignoring some
variables in the optimization, hence speeding up solvers. When the procedure is
proven not to discard features wrongly the rules are said to be \emph{safe}. In
this paper we derive new safe rules for generalized linear models regularized
with and norms. The rules are based on duality gap
computations and spherical safe regions whose diameters converge to zero. This
allows to discard safely more variables, in particular for low regularization
parameters. The GAP Safe rule can cope with any iterative solver and we
illustrate its performance on coordinate descent for multi-task Lasso, binary
and multinomial logistic regression, demonstrating significant speed ups on all
tested datasets with respect to previous safe rules.Comment: in Proceedings of the 29-th Conference on Neural Information
Processing Systems (NIPS), 201
- …