Polynomial Time Algorithms for Branching Markov Decision Processes and Probabilistic Min(Max) Polynomial Bellman Equations

We show that one can approximate the least fixed point solution for a multivariate system of monotone probabilistic max(min) polynomial equations, referred to as maxPPSs (and minPPSs, respectively), in time polynomial in both the encoding size of the system of equations and in log(1/epsilon), where epsilon > 0 is the desired additive error bound of the solution. (The model of computation is the standard Turing machine model.) We establish this result using a generalization of Newton's method which applies to maxPPSs and minPPSs, even though the underlying functions are only piecewise-differentiable. This generalizes our recent work which provided a P-time algorithm for purely probabilistic PPSs. These equations form the Bellman optimality equations for several important classes of infinite-state Markov Decision Processes (MDPs). Thus, as a corollary, we obtain the first polynomial time algorithms for computing to within arbitrary desired precision the optimal value vector for several classes of infinite-state MDPs which arise as extensions of classic, and heavily studied, purely stochastic processes. These include both the problem of maximizing and mininizing the termination (extinction) probability of multi-type branching MDPs, stochastic context-free MDPs, and 1-exit Recursive MDPs. Furthermore, we also show that we can compute in P-time an epsilon-optimal policy for both maximizing and minimizing branching, context-free, and 1-exit-Recursive MDPs, for any given desired epsilon > 0. This is despite the fact that actually computing optimal strategies is Sqrt-Sum-hard and PosSLP-hard in this setting. We also derive, as an easy consequence of these results, an FNP upper bound on the complexity of computing the value (within arbitrary desired precision) of branching simple stochastic games (BSSGs)

A Family of Iterative Gauss-Newton Shooting Methods for Nonlinear Optimal Control

This paper introduces a family of iterative algorithms for unconstrained nonlinear optimal control. We generalize the well-known iLQR algorithm to different multiple-shooting variants, combining advantages like straight-forward initialization and a closed-loop forward integration. All algorithms have similar computational complexity, i.e. linear complexity in the time horizon, and can be derived in the same computational framework. We compare the full-step variants of our algorithms and present several simulation examples, including a high-dimensional underactuated robot subject to contact switches. Simulation results show that our multiple-shooting algorithms can achieve faster convergence, better local contraction rates and much shorter runtimes than classical iLQR, which makes them a superior choice for nonlinear model predictive control applications.Comment: 8 page

A Newton Collocation Method for Solving Dynamic Bargaining Games

We develop and implement a collocation method to solve for an equilibrium in the dynamic legislative bargaining game of Duggan and Kalandrakis (2008). We formulate the collocation equations in a quasi-discrete version of the model, and we show that the collocation equations are locally Lipchitz continuous and directionally differentiable. In numerical experiments, we successfully implement a globally convergent variant of Broyden's method on a preconditioned version of the collocation equations, and the method economizes on computation cost by more than 50% compared to the value iteration method. We rely on a continuity property of the equilibrium set to obtain increasingly precise approximations of solutions to the continuum model. We showcase these techniques with an illustration of the dynamic core convergence theorem of Duggan and Kalandrakis (2008) in a nine-player, two-dimensional model with negative quadratic preferences.

An interior-point method for the single-facility location problem with mixed norms using a conic formulation

We consider the single-facility location problem with mixed norms, i.e. the problem of minimizing the sum of the distances from a point to a set of fixed points in R, where each distance can be measured according to a different p-norm.We show how this problem can be expressed into a structured conic format by decomposing the nonlinear components of the objective into a series of constraints involving three-dimensional cones. Using the availability of a self-concordant barrier for these cones, we present a polynomial-time algorithm (a long-step path-following interior-point scheme) to solve the problem up to a given accuracy. Finally, we report computational results for this algorithm and compare with standard nonlinear optimization solvers applied to this problem.nonsymmetric conic optimization, conic reformulation, convex optimization, sum of norm minimization, single-facility location problems, interior-point methods

An Efficient Policy Iteration Algorithm for Dynamic Programming Equations

We present an accelerated algorithm for the solution of static Hamilton-Jacobi-Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. In many cases, this limitation degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a value iteration phase and then switches to a policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computation with the value iteration and, at the same time, to be reasonably sure that the policy iteration method will finally converge to the optimal solution. We analyze the methods and efficient coupling in a number of examples in dimension two, three and four illustrating its properties

New Acceleration of Nearly Optimal Univariate Polynomial Root-findERS

Univariate polynomial root-finding has been studied for four millennia and is still the subject of intensive research. Hundreds of efficient algorithms for this task have been proposed. Two of them are nearly optimal. The first one, proposed in 1995, relies on recursive factorization of a polynomial, is quite involved, and has never been implemented. The second one, proposed in 2016, relies on subdivision iterations, was implemented in 2018, and promises to be practically competitive, although user's current choice for univariate polynomial root-finding is the package MPSolve, proposed in 2000, revised in 2014, and based on Ehrlich's functional iterations. By proposing and incorporating some novel techniques we significantly accelerate both subdivision and Ehrlich's iterations. Moreover our acceleration of the known subdivision root-finders is dramatic in the case of sparse input polynomials. Our techniques can be of some independent interest for the design and analysis of polynomial root-finders.Comment: 89 pages, 5 figures, 2 table

Adaptive Detection of Instabilities: An Experimental Feasibility Study

We present an example of the practical implementation of a protocol for experimental bifurcation detection based on on-line identification and feedback control ideas. The idea is to couple the experiment with an on-line computer-assisted identification/feedback protocol so that the closed-loop system will converge to the open-loop bifurcation points. We demonstrate the applicability of this instability detection method by real-time, computer-assisted detection of period doubling bifurcations of an electronic circuit; the circuit implements an analog realization of the Roessler system. The method succeeds in locating the bifurcation points even in the presence of modest experimental uncertainties, noise and limited resolution. The results presented here include bifurcation detection experiments that rely on measurements of a single state variable and delay-based phase space reconstruction, as well as an example of tracing entire segments of a codimension-1 bifurcation boundary in two parameter space.Comment: 29 pages, Latex 2.09, 10 figures in encapsulated postscript format (eps), need psfig macro to include them. Submitted to Physica