A Maintenance Planning Framework using Online and Offline Deep Reinforcement Learning

Cost-effective asset management is an area of interest across several industries. Specifically, this paper develops a deep reinforcement learning (DRL) solution to automatically determine an optimal rehabilitation policy for continuously deteriorating water pipes. We approach the problem of rehabilitation planning in an online and offline DRL setting. In online DRL, the agent interacts with a simulated environment of multiple pipes with distinct lengths, materials, and failure rate characteristics. We train the agent using deep Q-learning (DQN) to learn an optimal policy with minimal average costs and reduced failure probability. In offline learning, the agent uses static data, e.g., DQN replay data, to learn an optimal policy via a conservative Q-learning algorithm without further interactions with the environment. We demonstrate that DRL-based policies improve over standard preventive, corrective, and greedy planning alternatives. Additionally, learning from the fixed DQN replay dataset in an offline setting further improves the performance. The results warrant that the existing deterioration profiles of water pipes consisting of large and diverse states and action trajectories provide a valuable avenue to learn rehabilitation policies in the offline setting, which can be further fine-tuned using the simulator.Comment: Published Neural Comput & Applic (2023), 12 pages, 8 Figur

Multi-agent deep reinforcement learning based Predictive Maintenance on parallel machines

In the context of Industry 4.0, companies understand the advantages of performing Predictive Maintenance (PdM). However, when moving towards PdM, several considerations must be carefully examined. First, they need to have a sufficient number of production machines and relative fault data to generate maintenance predictions. Second, they need to adopt the right maintenance approach, which, ideally, should self-adapt to the machinery, priorities of the organization, technician skills, but also to be able to deal with uncertainty. Reinforcement learning (RL) is envisioned as a key technique in this regard due to its inherent ability to learn by interacting through trials and errors, but very few RL-based maintenance frameworks have been proposed so far in the literature, or are limited in several respects. This paper proposes a new multi-agent approach that learns a maintenance policy performed by technicians, under the uncertainty of multiple machine failures. This approach comprises RL agents that partially observe the state of each machine to coordinate the decision-making in maintenance scheduling, resulting in the dynamic assignment of maintenance tasks to technicians (with different skills) over a set of machines. Experimental evaluation shows that our RL-based maintenance policy outperforms traditional maintenance policies (incl., corrective and preventive ones) in terms of failure prevention and downtime, improving by ≈75% the overall performance

Towards a Deep Reinforcement Learning based approach for real-time decision making and resource allocation for Prognostics and Health Management applications

Industrial operational environments are stochastic and can have complex system dynamics which introduce multiple levels of uncertainty. This uncertainty leads to sub-optimal decision making and resource allocation. Digitalisation and automation of production equipment and the maintenance environment enable predictive maintenance, meaning that equipment can be stopped for maintenance at the optimal time. Resource constraints in maintenance capacity could however result in further undesired downtime if maintenance cannot be performed when scheduled. In this dissertation the applicability of using a Multi-Agent Deep Reinforcement Learning based approach for decision making is investigated to determine the optimal maintenance scheduling policy in a fleet of assets where there are maintenance resource constraints. By considering the underlying system dynamics of maintenance capacity, as well as the health state of individual assets, a near-optimal decision making policy is found that increases equipment availability while also maximising maintenance capacity. The implemented solution is compared to a run-to-failure corrective maintenance strategy, a constant interval preventive maintenance strategy and a condition based predictive maintenance strategy. The proposed approach outperformed traditional maintenance strategies across several asset and operational maintenance performance metrics. It is concluded that Deep Reinforcement Learning based decision making for asset health management and resource allocation is more effective than human based decision making.Dissertation (MEng (Mechanical Engineering))--University of Pretoria, 2020.Mechanical and Aeronautical EngineeringMEng (Mechanical Engineering)Unrestricte

Hybrid Statistical, Machine Learning, and Deep Learning Models for Fault Diagnosis and Prognosis in Condition-based Maintenance

Maintenance has always been an essential and inseparable part of manufacturing and industrial sectors. Generally speaking, maintenance strategies aim to prevent asset failures/downtimes to protect investments and to provide a safe working environment. With the recent growth in sensor and data acquisition technologies, a rich amount of condition monitoring data has become available in manufacturing and industrial sectors. Consequently, there has been a recent surge of interest in using more advanced solutions, especially those based on Machine Learning (ML), Reinforcement Learning (RL), and Deep Learning (DL) models, to utilize such extensive and high-quality data in the maintenance domain. In this context, the thesis proposed different ML and hybrid models for prognostic and health management purposes to further advance the maintenance field. In particular, we conducted the following three studies: In the first work, a hybrid and semi-supervised framework is designed based on the hazard rate of the system. The proposed framework can extract the hidden state of the system without domain knowledge. To evaluate the efficacy of the proposed method, a real dataset is used where optimal maintenance policies are obtained based on the extracted states via RL. In the second study, a DL-based model is proposed to predict the hazard rate of the underlying system. As opposed to its statistical counterparts, the proposed predictive model does not assume any linear relationship between the sensors' measurements, and is capable of learning from censored data. In the last study, we investigated application of the proposed methods on high-dimensional data such as images. The proposed methods achieved promising results illustrating their great potential to be used in real-world applications

JIDOKA. Integration of Human and AI within Industry 4.0 Cyber Physical Manufacturing Systems

This book is about JIDOKA, a Japanese management technique coined by Toyota that consists of imbuing machines with human intelligence. The purpose of this compilation of research articles is to show industrial leaders innovative cases of digitization of value creation processes that have allowed them to improve their performance in a sustainable way. This book shows several applications of JIDOKA in the quest towards an integration of human and AI within Industry 4.0 Cyber Physical Manufacturing Systems. From the use of artificial intelligence to advanced mathematical models or quantum computing, all paths are valid to advance in the process of human–machine integration