82,963 research outputs found
Inverse stochastic optimal controls
We study an inverse problem of the stochastic optimal control of general
diffusions with performance index having the quadratic penalty term of the
control process. Under mild conditions on the drift, the volatility, the cost
functions of the state, and under the assumption that the optimal control
belongs to the interior of the control set, we show that our inverse problem is
well-posed using a stochastic maximum principle. Then, with the well-posedness,
we reduce the inverse problem to some root finding problem of the expectation
of a random variable involved with the value function, which has a unique
solution. Based on this result, we propose a numerical method for our inverse
problem by replacing the expectation above with arithmetic mean of observed
optimal control processes and the corresponding state processes. The recent
progress of numerical analyses of Hamilton-Jacobi-Bellman equations enables the
proposed method to be implementable for multi-dimensional cases. In particular,
with the help of the kernel-based collocation method for
Hamilton-Jacobi-Bellman equations, our method for the inverse problems still
works well even when an explicit form of the value function is unavailable.
Several numerical experiments show that the numerical method recover the
unknown weight parameter with high accuracy
Performance Guarantees for Homomorphisms Beyond Markov Decision Processes
Most real-world problems have huge state and/or action spaces. Therefore, a
naive application of existing tabular solution methods is not tractable on such
problems. Nonetheless, these solution methods are quite useful if an agent has
access to a relatively small state-action space homomorphism of the true
environment and near-optimal performance is guaranteed by the map. A plethora
of research is focused on the case when the homomorphism is a Markovian
representation of the underlying process. However, we show that near-optimal
performance is sometimes guaranteed even if the homomorphism is non-Markovian.
Moreover, we can aggregate significantly more states by lifting the Markovian
requirement without compromising on performance. In this work, we expand
Extreme State Aggregation (ESA) framework to joint state-action aggregations.
We also lift the policy uniformity condition for aggregation in ESA that allows
even coarser modeling of the true environment
- …