Global Optimization for Value Function Approximation

Existing value function approximation methods have been successfully used in many applications, but they often lack useful a priori error bounds. We propose a new approximate bilinear programming formulation of value function approximation, which employs global optimization. The formulation provides strong a priori guarantees on both robust and expected policy loss by minimizing specific norms of the Bellman residual. Solving a bilinear program optimally is NP-hard, but this is unavoidable because the Bellman-residual minimization itself is NP-hard. We describe and analyze both optimal and approximate algorithms for solving bilinear programs. The analysis shows that this algorithm offers a convergent generalization of approximate policy iteration. We also briefly analyze the behavior of bilinear programming algorithms under incomplete samples. Finally, we demonstrate that the proposed approach can consistently minimize the Bellman residual on simple benchmark problems

A D-induced duality and its applications

This paper attempts to extend the notion of duality for convex cones, by basing it on a predescribed conic ordering and a fixed bilinear mapping. This is an extension of the standard definition of dual cones, in the sense that the nonnegativity of the inner-product is replaced by a pre-specified conic ordering, defined by a convex cone D, and the inner-product itself is replaced by a general multi-dimensional bilinear mapping. This new type of duality is termed the D-induced duality in the paper. We further introduce the notion of D-induced polar sets within the same framework, which can be viewed as a generalization of the D-induced polar sets within the same framework, which can be viewed as a generalization of the D-induced dual cones and are convenient to use for some practical applications. Properties of the extended duality, including the extended bi-polar theorem, are proven. Furthermore, attention is paid to the computation and approximation of the D-induced dual objects. We discuss, as examples, applications of the newly introduced D-induced duality concepts in robust conic optimization and the duality theory for multi-objective conic optimization.bi-polar theorem;conic optimization;convex cones;duality