62,067 research outputs found
Adaptive Sequential Optimization with Applications to Machine Learning
A framework is introduced for solving a sequence of slowly changing
optimization problems, including those arising in regression and classification
applications, using optimization algorithms such as stochastic gradient descent
(SGD). The optimization problems change slowly in the sense that the minimizers
change at either a fixed or bounded rate. A method based on estimates of the
change in the minimizers and properties of the optimization algorithm is
introduced for adaptively selecting the number of samples needed from the
distributions underlying each problem in order to ensure that the excess risk,
i.e., the expected gap between the loss achieved by the approximate minimizer
produced by the optimization algorithm and the exact minimizer, does not exceed
a target level. Experiments with synthetic and real data are used to confirm
that this approach performs well.Comment: submitted to ICASSP 2016, extended versio
Adaptive sequential optimization with applications to machine learning
The focus of this thesis is on solving a sequence of optimization problems that change over time in a structured manner. This type of problem naturally arises in contexts as diverse as channel estimation, target tracking, sequential machine learning, and repeated games. Due to the time-varying nature of these problems, it is necessary to determine new solutions as the problems change in order to ensure good solution quality. However, since the problems change over time in a structured manner, it is beneficial to exploit solutions to the previous optimization problems in order to efficiently solve the current optimization problem.
The first problem considered is sequentially solving minimization problems that change slowly, in the sense that the gap between successive minimizers is bounded in norm. The minimization problems are solved by sequentially applying a selected optimization algorithm, such as stochastic gradient descent (SGD), based on drawing a number of samples in order to carry out a desired number of iterations. Two tracking criteria are introduced to evaluate approximate minimizer quality: one based on being accurate with respect to the mean trajectory, and the other based on being accurate in high probability (IHP). Knowledge of the bound on how the minimizers change, combined with properties of the chosen optimization algorithm, is used to select the number of samples needed to meet the desired tracking criterion.
Next, it is not assumed that the bound on how the minimizers change is known. A technique to estimate the change in minimizers is provided along with analysis to show that eventually the estimate upper bounds the change in minimizers. This estimate of the change in minimizers is combined with the previous analysis to provide sample size selection rules to ensure that the mean or IHP tracking criterion is met. Simulations are used to confirm that the estimation approach provides the desired tracking accuracy in practice.
An application of this framework to machine learning problems is considered next. A cost-based approach is introduced to select the number of samples with a cost function for taking a number of samples and a cost budget over a fixed horizon. An extension of this framework is developed to apply cross validation for model selection. Finally, experiments with synthetic and real data are used to confirm that this approach performs well for machine learning problems.
The next model considered is solving a sequence of minimization problems with the possibility that there can be abrupt jumps in the minimizers mixed in with the normal slow changes. Alternative approaches are introduced to estimate the changes in the minimizers and select the number of samples. A simulation experiment demonstrates the effectiveness of this approach.
Finally, a variant of this framework is applied to learning in games. A sequence of repeated games is considered in which the underlying stage games themselves vary slowly over time in the sense that the pure strategy Nash equilibria change slowly. Approximate pure-strategy Nash equilibria are learned for this sequence of zero sum games. A technique is introduced to estimate the change in the Nash equilibiria as for the sequence of minimization problems. Applications to a synthetic game and a game based on a surveillance network problem are introduced to demonstrate the game framework
Non-monotone Submodular Maximization with Nearly Optimal Adaptivity and Query Complexity
Submodular maximization is a general optimization problem with a wide range
of applications in machine learning (e.g., active learning, clustering, and
feature selection). In large-scale optimization, the parallel running time of
an algorithm is governed by its adaptivity, which measures the number of
sequential rounds needed if the algorithm can execute polynomially-many
independent oracle queries in parallel. While low adaptivity is ideal, it is
not sufficient for an algorithm to be efficient in practice---there are many
applications of distributed submodular optimization where the number of
function evaluations becomes prohibitively expensive. Motivated by these
applications, we study the adaptivity and query complexity of submodular
maximization. In this paper, we give the first constant-factor approximation
algorithm for maximizing a non-monotone submodular function subject to a
cardinality constraint that runs in adaptive rounds and makes
oracle queries in expectation. In our empirical study, we use
three real-world applications to compare our algorithm with several benchmarks
for non-monotone submodular maximization. The results demonstrate that our
algorithm finds competitive solutions using significantly fewer rounds and
queries.Comment: 12 pages, 8 figure
Submodular Maximization with Nearly Optimal Approximation, Adaptivity and Query Complexity
Submodular optimization generalizes many classic problems in combinatorial
optimization and has recently found a wide range of applications in machine
learning (e.g., feature engineering and active learning). For many large-scale
optimization problems, we are often concerned with the adaptivity complexity of
an algorithm, which quantifies the number of sequential rounds where
polynomially-many independent function evaluations can be executed in parallel.
While low adaptivity is ideal, it is not sufficient for a distributed algorithm
to be efficient, since in many practical applications of submodular
optimization the number of function evaluations becomes prohibitively
expensive. Motivated by these applications, we study the adaptivity and query
complexity of adaptive submodular optimization.
Our main result is a distributed algorithm for maximizing a monotone
submodular function with cardinality constraint that achieves a
-approximation in expectation. This algorithm runs in
adaptive rounds and makes calls to the function evaluation
oracle in expectation. The approximation guarantee and query complexity are
optimal, and the adaptivity is nearly optimal. Moreover, the number of queries
is substantially less than in previous works. Last, we extend our results to
the submodular cover problem to demonstrate the generality of our algorithm and
techniques.Comment: 30 pages, Proceedings of the Thirtieth Annual ACM-SIAM Symposium on
Discrete Algorithms (SODA 2019
The SUMO toolbox: a tool for automatic regression modeling and active learning
Many complex, real world phenomena are difficult to study directly using controlled experiments. Instead, the use of computer simulations has become commonplace as a feasible alternative. Due to the computational cost of these high fidelity simulations, surrogate models are often employed as a drop-in replacement for the original simulator, in order to reduce evaluation times. In this context, neural networks, kernel methods, and other modeling techniques have become indispensable. Surrogate models have proven to be very useful for tasks such as optimization, design space exploration, visualization, prototyping and sensitivity analysis. We present a fully automated machine learning tool for generating accurate surrogate models, using active learning techniques to minimize the number of simulations and to maximize efficiency
- …