7,001 research outputs found
Factored Value Iteration Converges
In this paper we propose a novel algorithm, factored value iteration (FVI),
for the approximate solution of factored Markov decision processes (fMDPs). The
traditional approximate value iteration algorithm is modified in two ways. For
one, the least-squares projection operator is modified so that it does not
increase max-norm, and thus preserves convergence. The other modification is
that we uniformly sample polynomially many samples from the (exponentially
large) state space. This way, the complexity of our algorithm becomes
polynomial in the size of the fMDP description length. We prove that the
algorithm is convergent. We also derive an upper bound on the difference
between our approximate solution and the optimal one, and also on the error
introduced by sampling. We analyze various projection operators with respect to
their computation complexity and their convergence when combined with
approximate value iteration.Comment: 17 pages, 1 figur
Least-Squares Covariance Matrix Adjustment
We consider the problem of finding the smallest adjustment to a given symmetric matrix, as measured by the Euclidean or Frobenius norm, so that it satisfies some given linear equalities and inequalities, and in addition is positive semidefinite. This least-squares covariance adjustment problem is a convex optimization problem, and can be efficiently solved using standard methods when the number of variables (i.e., entries in the matrix) is modest, say, under . Since the number of variables is , this corresponds to a limit around . Malick [{\it SIAM J. Matrix Anal.\ Appl.,} 26 (2005), pp. 272--284] studies a closely related problem and calls it the semidefinite least-squares problem. In this paper we formulate a dual problem that has no matrix inequality or matrix variables, and a number of (scalar) variables equal to the number of equality and inequality constraints in the original least-squares covariance adjustment problem. This dual problem allows us to solve far larger least-squares covariance adjustment problems than would be possible using standard methods. Assuming a modest number of constraints, problems with are readily solved by the dual method. The dual method coincides with the dual method proposed by Malick when there are no inequality constraints and can be obtained as an extension of his dual method when there are inequality constraints. Using the dual problem, we show that in many cases the optimal solution is a low rank update of the original matrix. When the original matrix has structure, such as sparsity, this observation allows us to solve very large least-squares covariance adjustment problems
Solving Factored MDPs with Hybrid State and Action Variables
Efficient representations and solutions for large decision problems with
continuous and discrete variables are among the most important challenges faced
by the designers of automated decision support systems. In this paper, we
describe a novel hybrid factored Markov decision process (MDP) model that
allows for a compact representation of these problems, and a new hybrid
approximate linear programming (HALP) framework that permits their efficient
solutions. The central idea of HALP is to approximate the optimal value
function by a linear combination of basis functions and optimize its weights by
linear programming. We analyze both theoretical and computational aspects of
this approach, and demonstrate its scale-up potential on several hybrid
optimization problems
- …