Global Optimization for Value Function Approximation

Existing value function approximation methods have been successfully used in many applications, but they often lack useful a priori error bounds. We propose a new approximate bilinear programming formulation of value function approximation, which employs global optimization. The formulation provides strong a priori guarantees on both robust and expected policy loss by minimizing specific norms of the Bellman residual. Solving a bilinear program optimally is NP-hard, but this is unavoidable because the Bellman-residual minimization itself is NP-hard. We describe and analyze both optimal and approximate algorithms for solving bilinear programs. The analysis shows that this algorithm offers a convergent generalization of approximate policy iteration. We also briefly analyze the behavior of bilinear programming algorithms under incomplete samples. Finally, we demonstrate that the proposed approach can consistently minimize the Bellman residual on simple benchmark problems

Locally Adaptive Optimization: Adaptive Seeding for Monotone Submodular Functions

The Adaptive Seeding problem is an algorithmic challenge motivated by influence maximization in social networks: One seeks to select among certain accessible nodes in a network, and then select, adaptively, among neighbors of those nodes as they become accessible in order to maximize a global objective function. More generally, adaptive seeding is a stochastic optimization framework where the choices in the first stage affect the realizations in the second stage, over which we aim to optimize. Our main result is a $(1-1/e)^2$-approximation for the adaptive seeding problem for any monotone submodular function. While adaptive policies are often approximated via non-adaptive policies, our algorithm is based on a novel method we call \emph{locally-adaptive} policies. These policies combine a non-adaptive global structure, with local adaptive optimizations. This method enables the $(1-1/e)^2$-approximation for general monotone submodular functions and circumvents some of the impossibilities associated with non-adaptive policies. We also introduce a fundamental problem in submodular optimization that may be of independent interest: given a ground set of elements where every element appears with some small probability, find a set of expected size at most $k$ that has the highest expected value over the realization of the elements. We show a surprising result: there are classes of monotone submodular functions (including coverage) that can be approximated almost optimally as the probability vanishes. For general monotone submodular functions we show via a reduction from \textsc{Planted-Clique} that approximations for this problem are not likely to be obtainable. This optimization problem is an important tool for adaptive seeding via non-adaptive policies, and its hardness motivates the introduction of \emph{locally-adaptive} policies we use in the main result

Theory and applications of the Sum-Of-Squares technique

The Sum-of-Squares (SOS) approximation method is a technique used in optimization problems to derive lower bounds to the optimal value of an objective function. By representing the objective function as a sum of squares in a feature space, the SOS method transforms non-convex global optimization problems into solvable semidefinite programs. This note presents an overview of the SOS method. We start with its application in finite-dimensional feature spaces and, subsequently, we extend it to infinite-dimensional feature spaces using kernels (k-SOS). Additionally, we highlight the utilization of SOS for estimating some relevant quantities in information theory, including the log-partition function.Comment: 19 pages, 4 figure

Reservoir Flooding Optimization by Control Polynomial Approximations

In this dissertation, we provide novel parametrization procedures for water-flooding production optimization problems, using polynomial approximation techniques. The methods project the original infinite dimensional controls space into a polynomial subspace. Our contribution includes new parameterization formulations using natural polynomials, orthogonal Chebyshev polynomials and Cubic spline interpolation. We show that the proposed methods are well suited for black-box approach with stochastic global-search method as they tend to produce smooth control trajectories, while reducing the solution space size. We demonstrate their efficiency on synthetic two-dimensional problems and on a realistic 3-dimensional problem. By contributing with a new adjoint method formulation for polynomial approximation, we implemented the methods also with gradient-based algorithms. In addition to fine-scale simulation, we also performed reduced order modeling, where we demonstrated a synergistic effect when combining polynomial approximation with model order reduction, that leads to faster optimization with higher gains in terms of Net Present Value. Finally, we performed gradient-based optimization under uncertainty. We proposed a new multi-objective function with three components, one that maximizes the expected value of all realizations, and two that maximize the averages of distribution tails from both sides. The new objective provides decision makers with the flexibility to choose the amount of risk they are willing to take, while deciding on production strategy or performing reserves estimation (P10;P50;P90)