Integration of cost-risk assessment of denial of service within an intelligent maintenance system

As organisations become richer in data the function of asset management will have to increasingly use intelligent systems to control condition monitoring systems and organise maintenance. In the future the UK rail industry is anticipating having to optimize capacity by running trains closer to each other. In this situation maintenance becomes extremely problematic as within such a high-performance network a relatively minor fault will impact more trains and passengers; such denial of service causes reputational damage for the industry and causes fines to be levied against the infrastructure owner, Network Rail. Intelligent systems used to control condition monitoring systems will need to optimize for several factors; optimization for minimizing denial of service will be one such factor. With schedules anticipated to be increasingly complicated detailed estimation methods will be extremely difficult to implement. Cost prediction of maintenance activities tend to be expert driven and require extensive details, making automation of such an activity difficult. Therefore a stochastic process will be needed to approach the problem of predicting the denial of service arising from any required maintenance. Good uncertainty modelling will help to increase the confidence of estimates. This paper seeks to detail the challenges that the UK Railway industry face with regards to cost modelling of maintenance activities and outline an example of a suitable cost model for quantifying cost uncertainty. The proposed uncertainty quantification is based on historical cost data and interpretation of its statistical distributions. These estimates are then integrated in a cost model to obtain accurate uncertainty measurements of outputs through Monte-Carlo simulation methods. An additional criteria of the model was that it be suitable for integration into an existing prototype integrated intelligent maintenance system. It is anticipated that applying an integrated maintenance management system will apply significant downward pressure on maintenance budgets and reduce denial of service. Accurate cost estimation is therefore of great importance if anticipated cost efficiencies are to be achieved. While the rail industry has been the focus of this work, other industries have been considered and it is anticipated that the approach will be applicable to many other organisations across several asset management intensive industrie

Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search

Bayesian model-based reinforcement learning is a formally elegant approach to learning optimal behaviour under model uncertainty, trading off exploration and exploitation in an ideal way. Unfortunately, finding the resulting Bayes-optimal policies is notoriously taxing, since the search space becomes enormous. In this paper we introduce a tractable, sample-based method for approximate Bayes-optimal planning which exploits Monte-Carlo tree search. Our approach outperformed prior Bayesian model-based RL algorithms by a significant margin on several well-known benchmark problems -- because it avoids expensive applications of Bayes rule within the search tree by lazily sampling models from the current beliefs. We illustrate the advantages of our approach by showing it working in an infinite state space domain which is qualitatively out of reach of almost all previous work in Bayesian exploration.Comment: 14 pages, 7 figures, includes supplementary material. Advances in Neural Information Processing Systems (NIPS) 201

Accelerating Cooperative Planning for Automated Vehicles with Learned Heuristics and Monte Carlo Tree Search

Efficient driving in urban traffic scenarios requires foresight. The observation of other traffic participants and the inference of their possible next actions depending on the own action is considered cooperative prediction and planning. Humans are well equipped with the capability to predict the actions of multiple interacting traffic participants and plan accordingly, without the need to directly communicate with others. Prior work has shown that it is possible to achieve effective cooperative planning without the need for explicit communication. However, the search space for cooperative plans is so large that most of the computational budget is spent on exploring the search space in unpromising regions that are far away from the solution. To accelerate the planning process, we combined learned heuristics with a cooperative planning method to guide the search towards regions with promising actions, yielding better solutions at lower computational costs

Optimizing the depth and the direction of prospective planning using information values

Evaluating the future consequences of actions is achievable by simulating a mental search tree into the future. Expanding deep trees, however, is computationally taxing. Therefore, machines and humans use a plan-until-habit scheme that simulates the environment up to a limited depth and then exploits habitual values as proxies for consequences that may arise in the future. Two outstanding questions in this scheme are “in which directions the search tree should be expanded?”, and “when should the expansion stop?”. Here we propose a principled solution to these questions based on a speed/accuracy tradeoff: deeper expansion in the appropriate directions leads to more accurate planning, but at the cost of slower decision-making. Our simulation results show how this algorithm expands the search tree effectively and efficiently in a grid-world environment. We further show that our algorithm can explain several behavioral patterns in animals and humans, namely the effect of time-pressure on the depth of planning, the effect of reward magnitudes on the direction of planning, and the gradual shift from goal-directed to habitual behavior over the course of training. The algorithm also provides several predictions testable in animal/human experiments