Deep Multi-Agent Reinforcement Learning using DNN-Weight Evolution to Optimize Supply Chain Performance

To develop a supply chain management (SCM) system that performs optimally for both each entity in the chain and the entire chain, a multi-agent reinforcement learning (MARL) technique has been developed. To solve two problems of the MARL for SCM (building a Markov decision processes for a supply chain and avoiding learning stagnation in a way similar to the prisoner\u27s dilemma ), a learning management method with deep-neural-network (DNN)-weight evolution (LM-DWE) has been developed. By using a beer distribution game (BDG) as an example of a supply chain, experiments with a four-agent system were performed. Consequently, the LM-DWE successfully solved the above two problems and achieved 80.0% lower total cost than expert players of the BDG

Applications of Machine Learning in Supply Chains

Advances in new technologies have resulted in increasing the speed of data generation and accessing larger data storage. The availability of huge datasets and massive computational power have resulted in the emergence of new algorithms in artificial intelligence and specifically machine learning, with significant research done in fields like computer vision. Although the same amount of data exists in most components of supply chains, there is not much research to utilize the power of raw data to improve efficiency in supply chains.In this dissertation our objective is to propose data-driven non-parametric machine learning algorithms to solve different supply chain problems in data-rich environments.Among wide range of supply chain problems, inventory management has been one of the main challenges in every supply chain. The ability to manage inventories to maximize the service level while minimizing holding costs is a goal of many company. An unbalanced inventory system can easily result in a stopped production line, back-ordered demands, lost sales, and huge extra costs. This dissertation studies three problems and proposes machine learning algorithms to help inventory managers reduce their inventory costs.In the first problem, we consider the newsvendor problem in which an inventory manager needs to determine the order quantity of a perishable product to minimize the sum of shortage and holding costs, while some feature information is available for each product. We propose a neural network approach with a specialized loss function to solve this problem. The neural network gets historical data and is trained to provide the order quantity. We show that our approach works better than the classical separated estimation and optimization approaches as well as other machine learning based algorithms. Especially when the historical data is noisy, and there is little data for each combination of features, our approach works much better than other approaches. Also, to show how this approach can be used in other common inventory policies, we apply it on an $(r,Q)$ policy and provide the results.This algorithm allows inventory managers to quickly determine an order quantity without obtaining the underling demand distribution.Now, assume the order quantities or safety stock levels are obtained for a single or multi-echelon system. Classical inventory optimization models work well in normal conditions, or in other words when all underlying assumptions are valid. Once one of the assumptions or the normal working conditions is violated, unplanned stock-outs or excess inventories arise.To address this issue, in the second problem, a multi-echelon supply network is considered, and the goal is to determine the nodes that might face a stock-out in the next period. Stock-outs are usually expensive and inventory managers try to avoid them, so stock-out prediction might results in averting stock-outs and the corresponding costs.In order to provide such predictions, we propose a neural network model and additionally three naive algorithms. We analyze the performance of the proposed algorithms by comparing them with classical forecasting algorithms and a linear regression model, over five network topologies. Numerical results show that the neural network model is quite accurate and obtains accuracies in $[0.92, 0.99]$ for the hardest to easiest network topologies, with average of 0.950 and standard deviation of 0.023, while the closest competitor, i.e., one of the proposed naive algorithms, obtains accuracies in $[0.91, 0.95]$ with average of 9.26 and standard deviation of .0136. Additionally, we suggest conditions under which each algorithm is the most reliable and additionally apply all algorithms to threshold and multi-period predictions.Although stock-out prediction can be very useful, any inventory manager would like to have a powerful model to optimize the inventory system and balance the holding and shortage costs. The literature on multi-echelon inventory models is quite rich, though it mostly relies on the assumption of accessing a known demand distribution. The demand distribution can be approximated, but even so, in some cases a globally optimal model is not available.In the third problem, we develop a machine learning algorithm to address this issue for multi-period inventory optimization problems in multi-echelon networks. We consider the well-known beer game problem and propose a reinforcement learning algorithm to efficiently learn ordering policies from data.The beer game is a serial supply chain with four agents, i.e. retailer, wholesaler, distributor, and manufacturer, in which each agent replenishes its stock by ordering beer from its predecessor. The retailer satisfies the demand of external customers, and the manufacturer orders from external suppliers. Each of the agents must decide its own order quantity to minimize the summation of holding and shortage cost of the system, while they are not allowed to share any information with other agents. For this setting, a base-stock policy is optimal, if the retailer is the only node with a positive shortage cost and a known demand distribution is available. Outside of this narrow condition, there is not a known optimal policy for this game. Also, from the game theory point of view, the beer game can be modeled as a decentralized multi-agent cooperative problem with partial observability, which is known as a NEXP-complete problem.We propose an extension of deep Q-network for making decisions about order quantities in a single node of the beer game. When the co-players follow a rational policy, it obtains a close-to-optimal solution, and it works much better than a base-stock policy if the other agents play irrationally. Additionally, to reduce the training time of the algorithm, we propose using transfer learning, which reduces the training time by one order of magnitude. This approach can be extended to other inventory optimization and supply chain problems

Artificial Intelligence and Its Application in Optimization under Uncertainty

Nowadays, the increase in data acquisition and availability and complexity around optimization make it imperative to jointly use artificial intelligence (AI) and optimization for devising data-driven and intelligent decision support systems (DSS). A DSS can be successful if large amounts of interactive data proceed fast and robustly and extract useful information and knowledge to help decision-making. In this context, the data-driven approach has gained prominence due to its provision of insights for decision-making and easy implementation. The data-driven approach can discover various database patterns without relying on prior knowledge while also handling flexible objectives and multiple scenarios. This chapter reviews recent advances in data-driven optimization, highlighting the promise of data-driven optimization that integrates mathematical programming and machine learning (ML) for decision-making under uncertainty and identifies potential research opportunities. This chapter provides guidelines and implications for researchers, managers, and practitioners in operations research who want to advance their decision-making capabilities under uncertainty concerning data-driven optimization. Then, a comprehensive review and classification of the relevant publications on the data-driven stochastic program, data-driven robust optimization, and data-driven chance-constrained are presented. This chapter also identifies fertile avenues for future research that focus on deep-data-driven optimization, deep data-driven models, as well as online learning-based data-driven optimization. Perspectives on reinforcement learning (RL)-based data-driven optimization and deep RL for solving NP-hard problems are discussed. We investigate the application of data-driven optimization in different case studies to demonstrate improvements in operational performance over conventional optimization methodology. Finally, some managerial implications and some future directions are provided

A Tale of Two Cities: Data and Configuration Variances in Robust Deep Learning

Deep neural networks (DNNs), are widely used in many industries such as image recognition, supply chain, medical diagnosis, and autonomous driving. However, prior work has shown the high accuracy of a DNN model does not imply high robustness (i.e., consistent performances on new and future datasets) because the input data and external environment (e.g., software and model configurations) for a deployed model are constantly changing. Hence, ensuring the robustness of deep learning is not an option but a priority to enhance business and consumer confidence. Previous studies mostly focus on the data aspect of model variance. In this article, we systematically summarize DNN robustness issues and formulate them in a holistic view through two important aspects, i.e., data and software configuration variances in DNNs. We also provide a predictive framework to generate representative variances (counterexamples) by considering both data and configurations for robust learning through the lens of search-based optimization

Intégration de la blockchain à l'Internet des objets

L'Internet des objets (IdO) est en train de transformer l'industrie traditionnelle en une industrie intelligente où les décisions sont prises en fonction des données. L'IdO interconnecte de nombreux objets (ou dispositifs) qui effectuent des tâches complexes (e.g., la collecte de données, l'optimisation des services, la transmission de données). Toutefois, les caractéristiques intrinsèques de l'IdO entraînent plusieurs problèmes, tels que la décentralisation, une faible interopérabilité, des problèmes de confidentialité et des failles de sécurité. Avec l'évolution attendue de l'IdO dans les années à venir, il est nécessaire d'assurer la confiance dans cette énorme source d'informations entrantes. La blockchain est apparue comme une technologie clé pour relever les défis de l'IdO. En raison de ses caractéristiques saillantes telles que la décentralisation, l'immuabilité, la sécurité et l'auditabilité, la blockchain a été proposée pour établir la confiance dans plusieurs applications, y compris l'IdO. L'intégration de la blockchain a l'IdO ouvre la porte à de nouvelles possibilités qui améliorent intrinsèquement la fiabilité, la réputation, et la transparence pour toutes les parties concernées, tout en permettant la sécurité. Cependant, les blockchains classiques sont coûteuses en calcul, ont une évolutivité limitée, et nécessitent une bande passante élevée, ce qui les rend inadaptées aux environnements IdO à ressources limitées. L'objectif principal de cette thèse est d'utiliser la blockchain comme un outil clé pour améliorer l'IdO. Pour atteindre notre objectif, nous relevons les défis de la fiabilité des données et de la sécurité de l'IdO en utilisant la blockchain ainsi que de nouvelles technologies émergentes, notamment l'intelligence artificielle (IA). Dans la première partie de cette thèse, nous concevons une blockchain qui garantit la fiabilité des données, adaptée à l'IdO. Tout d'abord, nous proposons une architecture blockchain légère qui réalise la décentralisation en formant un réseau superposé où les dispositifs à ressources élevées gèrent conjointement la blockchain. Ensuite, nous présentons un algorithme de consensus léger qui réduit la puissance de calcul, la capacité de stockage, et la latence de la blockchain. Dans la deuxième partie de cette thèse, nous concevons un cadre sécurisé pour l'IdO tirant parti de la blockchain. Le nombre croissant d'attaques sur les réseaux IdO, et leurs graves effets, rendent nécessaire la création d'un IdO avec une sécurité plus sophistiquée. Par conséquent, nous tirons parti des modèles IA pour fournir une intelligence intégrée dans les dispositifs et les réseaux IdO afin de prédire et d'identifier les menaces et les vulnérabilités de sécurité. Nous proposons un système de détection d'intrusion par IA qui peut détecter les comportements malveillants et contribuer à renforcer la sécurité de l'IdO basé sur la blockchain. Ensuite, nous concevons un mécanisme de confiance distribué basé sur des contrats intelligents de blockchain pour inciter les dispositifs IdO à se comporter de manière fiable. Les systèmes IdO existants basés sur la blockchain souffrent d'une bande passante de communication et d’une évolutivité limitée. Par conséquent, dans la troisième partie de cette thèse, nous proposons un apprentissage machine évolutif basé sur la blockchain pour l'IdO. Tout d'abord, nous proposons un cadre IA multi-tâches qui exploite la blockchain pour permettre l'apprentissage parallèle de modèles. Ensuite, nous concevons une technique de partitionnement de la blockchain pour améliorer l'évolutivité de la blockchain. Enfin, nous proposons un algorithme d'ordonnancement des dispositifs pour optimiser l'utilisation des ressources, en particulier la bande passante de communication.Abstract : The Internet of Things (IoT) is reshaping the incumbent industry into a smart industry featured with data-driven decision making. The IoT interconnects many objects (or devices) that perform complex tasks (e.g., data collection, service optimization, data transmission). However, intrinsic features of IoT result in several challenges, such as decentralization, poor interoperability, privacy issues, and security vulnerabilities. With the expected evolution of IoT in the coming years, there is a need to ensure trust in this huge source of incoming information. Blockchain has emerged as a key technology to address the challenges of IoT. Due to its salient features such as decentralization, immutability, security, and auditability, blockchain has been proposed to establish trust in several applications, including IoT. The integration of IoT and blockchain opens the door for new possibilities that inherently improve trustworthiness, reputation, and transparency for all involved parties, while enabling security. However, conventional blockchains are computationally expensive, have limited scalability, and incur significant bandwidth, making them unsuitable for resource-constrained IoT environments. The main objective of this thesis is to leverage blockchain as a key enabler to improve the IoT. Toward our objective, we address the challenges of data reliability and IoT security using the blockchain and new emerging technologies, including machine learning (ML). In the first part of this thesis, we design a blockchain that guarantees data reliability, suitable for IoT. First, we propose a lightweight blockchain architecture that achieves decentralization by forming an overlay network where high-resource devices jointly manage the blockchain. Then, we present a lightweight consensus algorithm that reduces blockchain computational power, storage capability, and latency. In the second part of this thesis, we design a secure framework for IoT leveraging blockchain. The increasing number of attacks on IoT networks, and their serious effects, make it necessary to create an IoT with more sophisticated security. Therefore, we leverage ML models to provide embedded intelligence in the IoT devices and networks to predict and identify security threats and vulnerabilities. We propose a ML intrusion detection system that can detect malicious behaviors and help further bolster the blockchain-based IoT’s security. Then, we design a distributed trust mechanism based on blockchain smart contracts to incite IoT devices to behave reliably. Existing blockchain-based IoT systems suffer from limited communication bandwidth and scalability. Therefore, in the third part of this thesis, we propose a scalable blockchain-based ML for IoT. First, we propose a multi-task ML framework that leverages the blockchain to enable parallel model learning. Then, we design a blockchain partitioning technique to improve the blockchain scalability. Finally, we propose a device scheduling algorithm to optimize resource utilization, in particular communication bandwidth

Deep Learning Techniques for Power System Operation: Modeling and Implementation

The fast development of the deep learning (DL) techniques in the most recent years has drawn attention from both academia and industry. And there have been increasing applications of the DL techniques in many complex real-world situations, including computer vision, medical diagnosis, and natural language processing. The great power and flexibility of DL can be attributed to its hierarchical learning structure that automatically extract features from mass amounts of data. In addition, DL applies an end-to-end solving mechanism, and directly generates the output from the input, where the traditional machine learning methods usually break down the problem and combine the results. The end-to-end mechanism considerably improve the computational efficiency of the DL.The power system is one of the most complex artificial infrastructures, and many power system control and operation problems share the same features as the above mentioned real-world applications, such as time variability and uncertainty, partial observability, which impedes the performance of the conventional model-based methods. On the other hand, with the wide spread implementation of Advanced Metering Infrastructures (AMI), the SCADA, the Wide Area Monitoring Systems (WAMS), and many other measuring system providing massive data from the field, the data-driven deep learning technique is becoming an intriguing alternative method to enable the future development and success of the smart grid. This dissertation aims to explore the potential of utilizing the deep-learning-based approaches to solve a broad range of power system modeling and operation problems. First, a comprehensive literature review is conducted to summarize the existing applications of deep learning techniques in power system area. Second, the prospective application of deep learning techniques in several scenarios in power systems, including contingency screening, cascading outage search, multi-microgrid energy management, residential HVAC system control, and electricity market bidding are discussed in detail in the following 2-6 chapters. The problem formulation, the specific deep learning approaches in use, and the simulation results are all presented, and also compared with the currently used model-based method as a verification of the advantage of deep learning. Finally, the conclusions are provided in the last chapter, as well as the directions for future researches. It’s hoped that this dissertation can work as a single spark of fire to enlighten more innovative ideas and original studies, widening and deepening the application of deep learning technique in the field of power system, and eventually bring some positive impacts to the real-world bulk grid resilient and economic control and operation

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Neural combinatorial optimization as an enabler technology to design real-time virtual network function placement decision systems

158 p.The Fifth Generation of the mobile network (5G) represents a breakthrough technology for thetelecommunications industry. 5G provides a unified infrastructure capable of integrating over thesame physical network heterogeneous services with different requirements. This is achieved thanksto the recent advances in network virtualization, specifically in Network Function Virtualization(NFV) and Software Defining Networks (SDN) technologies. This cloud-based architecture not onlybrings new possibilities to vertical sectors but also entails new challenges that have to be solvedaccordingly. In this sense, it enables to automate operations within the infrastructure, allowing toperform network optimization at operational time (e.g., spectrum optimization, service optimization,traffic optimization). Nevertheless, designing optimization algorithms for this purpose entails somedifficulties. Solving the underlying Combinatorial Optimization (CO) problems that these problemspresent is usually intractable due to their NP-Hard nature. In addition, solutions to these problems arerequired in close to real-time due to the tight time requirements on this dynamic environment. Forthis reason, handwritten heuristic algorithms have been widely used in the literature for achievingfast approximate solutions on this context.However, particularizing heuristics to address CO problems can be a daunting task that requiresexpertise. The ability to automate this resolution processes would be of utmost importance forachieving an intelligent network orchestration. In this sense, Artificial Intelligence (AI) is envisionedas the key technology for autonomously inferring intelligent solutions to these problems. Combining AI with network virtualization can truly transform this industry. Particularly, this Thesis aims at using Neural Combinatorial Optimization (NCO) for inferring endsolutions on CO problems. NCO has proven to be able to learn near optimal solutions on classicalcombinatorial problems (e.g., the Traveler Salesman Problem (TSP), Bin Packing Problem (BPP),Vehicle Routing Problem (VRP)). Specifically, NCO relies on Reinforcement Learning (RL) toestimate a Neural Network (NN) model that describes the relation between the space of instances ofthe problem and the solutions for each of them. In other words, this model for a new instance is ableto infer a solution generalizing from the problem space where it has been trained. To this end, duringthe learning process the model takes instances from the learning space, and uses the reward obtainedfrom evaluating the solution to improve its accuracy.The work here presented, contributes to the NCO theory in two main directions. First, this workargues that the performance obtained by sequence-to-sequence models used for NCO in the literatureis improved presenting combinatorial problems as Constrained Markov Decision Processes (CMDP).Such property can be exploited for building a Markovian model that constructs solutionsincrementally based on interactions with the problem. And second, this formulation enables toaddress general constrained combinatorial problems under this framework. In this context, the modelin addition to the reward signal, relies on penalty signals generated from constraint dissatisfactionthat direct the model toward a competitive policy even in highly constrained environments. Thisstrategy allows to extend the number of problems that can be addressed using this technology.The presented approach is validated in the scope of intelligent network management, specifically inthe Virtual Network Function (VNF) placement problem. This problem consists of efficientlymapping a set of network service requests on top of the physical network infrastructure. Particularly,we seek to obtain the optimal placement for a network service chain considering the state of thevirtual environment, so that a specific resource objective is accomplished, in this case theminimization of the overall power consumption. Conducted experiments prove the capability of theproposal for learning competitive solutions when compared to classical heuristic, metaheuristic, andConstraint Programming (CP) solvers