2,910 research outputs found
Adaptive traffic signal control using approximate dynamic programming
This paper presents a study on an adaptive traffic signal controller for real-time operation. The controller aims for three operational objectives: dynamic allocation of green time, automatic adjustment to control parameters, and fast revision of signal plans. The control algorithm is built on approximate dynamic programming (ADP). This approach substantially reduces computational burden by using an approximation to the value function of the dynamic programming and reinforcement learning to update the approximation. We investigate temporal-difference learning and perturbation learning as specific learning techniques for the ADP approach. We find in computer simulation that the ADP controllers achieve substantial reduction in vehicle delays in comparison with optimised fixed-time plans. Our results show that substantial benefits can be gained by increasing the frequency at which the signal plans are revised, which can be achieved conveniently using the ADP approach
Path integral policy improvement with differential dynamic programming
Path Integral Policy Improvement with Covariance Matrix Adaptation (PI2-CMA) is a step-based model free reinforcement learning approach that combines statistical estimation techniques with fundamental results from Stochastic Optimal Control. Basically, a policy distribution is improved iteratively using reward weighted averaging of the corresponding rollouts. It was assumed that PI2-CMA somehow exploited gradient information that was contained by the reward weighted statistics. To our knowledge we are the first to expose the principle of this gradient extraction rigorously. Our findings reveal that PI2-CMA essentially obtains gradient information similar to the forward and backward passes in the Differential Dynamic Programming (DDP) method. It is then straightforward to extend the analogy with DDP by introducing a feedback term in the policy update. This suggests a novel algorithm which we coin Path Integral Policy Improvement with Differential Dynamic Programming (PI2-DDP). The resulting algorithm is similar to the previously proposed Sampled Differential Dynamic Programming (SaDDP) but we derive the method independently as a generalization of the framework of PI2-CMA. Our derivations suggest to implement some small variations to SaDDP so to increase performance. We validated our claims on a robot trajectory learning task
Learning scalable and transferable multi-robot/machine sequential assignment planning via graph embedding
Can the success of reinforcement learning methods for simple combinatorial
optimization problems be extended to multi-robot sequential assignment
planning? In addition to the challenge of achieving near-optimal performance in
large problems, transferability to an unseen number of robots and tasks is
another key challenge for real-world applications. In this paper, we suggest a
method that achieves the first success in both challenges for robot/machine
scheduling problems.
Our method comprises of three components. First, we show a robot scheduling
problem can be expressed as a random probabilistic graphical model (PGM). We
develop a mean-field inference method for random PGM and use it for Q-function
inference. Second, we show that transferability can be achieved by carefully
designing two-step sequential encoding of problem state. Third, we resolve the
computational scalability issue of fitted Q-iteration by suggesting a heuristic
auction-based Q-iteration fitting method enabled by transferability we
achieved.
We apply our method to discrete-time, discrete space problems (Multi-Robot
Reward Collection (MRRC)) and scalably achieve 97% optimality with
transferability. This optimality is maintained under stochastic contexts. By
extending our method to continuous time, continuous space formulation, we claim
to be the first learning-based method with scalable performance among
multi-machine scheduling problems; our method scalability achieves comparable
performance to popular metaheuristics in Identical parallel machine scheduling
(IPMS) problems
Algorithm Portfolios for Noisy Optimization
Noisy optimization is the optimization of objective functions corrupted by
noise. A portfolio of solvers is a set of solvers equipped with an algorithm
selection tool for distributing the computational power among them. Portfolios
are widely and successfully used in combinatorial optimization. In this work,
we study portfolios of noisy optimization solvers. We obtain mathematically
proved performance (in the sense that the portfolio performs nearly as well as
the best of its solvers) by an ad hoc portfolio algorithm dedicated to noisy
optimization. A somehow surprising result is that it is better to compare
solvers with some lag, i.e., propose the current recommendation of best solver
based on their performance earlier in the run. An additional finding is a
principled method for distributing the computational power among solvers in the
portfolio.Comment: in Annals of Mathematics and Artificial Intelligence, Springer
Verlag, 201
A survey of random processes with reinforcement
The models surveyed include generalized P\'{o}lya urns, reinforced random
walks, interacting urn models, and continuous reinforced processes. Emphasis is
on methods and results, with sketches provided of some proofs. Applications are
discussed in statistics, biology, economics and a number of other areas.Comment: Published at http://dx.doi.org/10.1214/07-PS094 in the Probability
Surveys (http://www.i-journals.org/ps/) by the Institute of Mathematical
Statistics (http://www.imstat.org
- …