3 research outputs found
Submodular Batch Selection for Training Deep Neural Networks
Mini-batch gradient descent based methods are the de facto algorithms for training neural network architectures today. We introduce a mini-batch selection strategy based on submodular function maximization. Our novel submodular formulation captures the informativeness of each sample and diversity of the whole subset. We design an efficient, greedy algorithm which can give high-quality solutions to this NP-hard combinatorial optimization problem. Our extensive experiments on standard datasets show that the deep models trained using the proposed batch selection strategy provide better generalization than Stochastic Gradient Descent as well as a popular baseline sampling strategy across different learning rates, batch sizes, and distance metrics
On the Approximation Relationship between Optimizing Ratio of Submodular (RS) and Difference of Submodular (DS) Functions
We demonstrate that from an algorithm guaranteeing an approximation factor
for the ratio of submodular (RS) optimization problem, we can build another
algorithm having a different kind of approximation guarantee -- weaker than the
classical one -- for the difference of submodular (DS) optimization problem,
and vice versa. We also illustrate the link between these two problems by
analyzing a \textsc{Greedy} algorithm which approximately maximizes objective
functions of the form , where are two non-negative, monotone,
submodular functions and is a {quasiconvex} 2-variables function, which
is non decreasing with respect to the first variable. For the choice
, we recover RS, and for the choice
, we recover DS. To the best of our knowledge, this
greedy approach is new for DS optimization. For RS optimization, it reduces to
the standard \textsc{GreedRatio} algorithm that has already been analyzed
previously. However, our analysis is novel for this case