An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems

Most solutions to the inventory management problem assume a centralization of information that is incompatible with organisational constraints in real supply chain networks. The inventory management problem is a well-known planning problem in operations research, concerned with finding the optimal re-order policy for nodes in a supply chain. While many centralized solutions to the problem exist, they are not applicable to real-world supply chains made up of independent entities. The problem can however be naturally decomposed into sub-problems, each associated with an independent entity, turning it into a multi-agent system. Therefore, a decentralized data-driven solution to inventory management problems using multi-agent reinforcement learning is proposed where each entity is controlled by an agent. Three multi-agent variations of the proximal policy optimization algorithm are investigated through simulations of different supply chain networks and levels of uncertainty. The centralized training decentralized execution framework is deployed, which relies on offline centralization during simulation-based policy identification, but enables decentralization when the policies are deployed online to the real system. Results show that using multi-agent proximal policy optimization with a centralized critic leads to performance very close to that of a centralized data-driven solution and outperforms a distributed model-based solution in most cases while respecting the information constraints of the system

Integrated management of chemical processes in a competitive environment

El objetivo general de esta Tesis es mejorar el proceso de la toma de decisiones en la gestión de cadenas de suministro, tomando en cuenta principalmente dos diferencias: ser competitivo considerando las decisiones propias de la cadena de suministro, y ser competitivo dentro de un entorno global. La estructura de ésta tesis se divide en 4 partes principales: La Parte I consiste en una introducción general de los temas cubiertos en esta Tesis (Capítulo 1). Una revisión de la literatura, que nos permite identificar las problemáticas asociadas al proceso de toma de decisiones (Capítulo 2). El Capítulo 3 presenta una introducción de las técnicas y métodos de optimización utilizados para resolver los problemas propuestos en esta Tesis. La Parte II se enfoca en la integración de los niveles de decisión, buscando mejorar la toma de decisiones de la propia cadena de suministro. El Capítulo 4 presenta una formulación matemática que integra las decisiones de síntesis de procesos y las decisiones operacionales. Además, este capítulo presenta un modelo integrado para la toma de decisiones operacionales incluyendo las características del control de procesos. El Capítulo 5 muestra la integración de las decisiones del nivel táctico y el operacional, dicha propuesta está basada en el conocimiento adquirido capturando la información relacionada al nivel operacional. Una vez obtenida esta información se incluye en la toma de decisiones a nivel táctico. Finalmente en el capítulo 6 se desarrolla un modelo simplificado para integrar múltiples cadenas de suministro. El modelo propuesto incluye la información detallada de las entidades presentes en una cadena de suministro (suministradores, plantas de producción, distribuidores y mercados) introduciéndola en un modelo matemático para su coordinación. La Parte III propone la integración explicita de múltiples cadenas de suministro que tienen que enfrentar numerosas situaciones propias de un mercado global. Asimismo, esta parte presenta una nueva herramienta de optimización basada en el uso integrado de métodos de programación matemática y conceptos relacionados a la Teoría de Juegos. En el Capítulo 7 analiza múltiples cadenas de suministro que cooperan o compiten por la demanda global del mercado. El Capítulo 8 incluye una comparación entre el problema resuelto en el Capítulo anterior y un modelo estocástico, los resultados obtenidos nos permiten situar el comportamiento de los competidores como fuente exógena de la incertidumbre típicamente asociada la demanda del mercado. Además, los resultados de ambos Capítulos muestran una mejora sustancial en el coste total de las cadenas de suministro asociada al hecho de cooperar para atender de forma conjunta la demanda disponible. Es por esto, que el Capítulo 9 presenta una nueva herramienta de negociación, basada en la resolución del mismo problema (Capítulo 7) bajo un análisis multiobjetivo. Finalmente, la parte IV presenta las conclusiones finales y una descripción general del trabajo futuro.This Thesis aims to enhance the decision making process in the SCM, remarking the difference between optimizing the SC to be competitive by its own, and to be competitive in a global market in cooperative and competitive environments. The structure of this work has been divided in four main parts: Part I: consists in a general introduction of the main topics covered in this manuscript (Chapter I); a review of the State of the Art that allows us to identify new open issues in the PSE (Chapter 2). Finally, Chapter 3 introduces the main optimization techniques and methods used in this contribution. Part II focuses on the integration of decision making levels in order to improve the decision making of a single SC: Chapter 4 presents a novel formulation to integrate synthesis and scheduling decision making models, additionally, this chapter also shows an integrated operational and control decision making model for distributed generations systems (EGS). Chapter 5 shows the integration of tactical and operational decision making levels. In this chapter a knowledge based approach has been developed capturing the information related to the operational decision making level. Then, this information has been included in the tactical decision making model. In Chapter 6 a simplified approach for integrated SCs is developed, the detailed information of the typical production‐distribution SC echelons has been introduced in a coordinated SC model. Part III proposes the explicit integration of several SC’s decision making in order to face several real market situations. As well, a novel formulation is developed using an MILP model and Game Theory (GT) as a decision making tool. Chapter 7 includes the tactical and operational analysis of several SC’s cooperating or competing for the global market demand. Moreover, Chapter 8 includes a comparison, based on the previous results (MILP‐GT optimization tool) and a two stage stochastic optimization model. Results from both Chapters show how cooperating for the global demand represent an improvement of the overall total cost. Consequently, Chapter 9 presents a bargaining tool obtained by the Multiobjective (MO) resolution of the model presented in Chapter 7. Finally, final conclusions and further work have been provided in Part IV.Postprint (published version

Optimal management of bio-based energy supply chains under parametric uncertainty through a data-driven decision-support framework

This paper addresses the optimal management of a multi-objective bio-based energy supply chain network subjected to multiple sources of uncertainty. The complexity to obtain an optimal solution using traditional uncertainty management methods dramatically increases with the number of uncertain factors considered. Such a complexity produces that, if tractable, the problem is solved after a large computational effort. Therefore, in this work a data-driven decision-making framework is proposed to address this issue. Such a framework exploits machine learning techniques to efficiently approximate the optimal management decisions considering a set of uncertain parameters that continuously influence the process behavior as an input. A design of computer experiments technique is used in order to combine these parameters and produce a matrix of representative information. These data are used to optimize the deterministic multi-objective bio-based energy network problem through conventional optimization methods, leading to a detailed (but elementary) map of the optimal management decisions based on the uncertain parameters. Afterwards, the detailed data-driven relations are described/identified using an Ordinary Kriging meta-model. The result exhibits a very high accuracy of the parametric meta-models for predicting the optimal decision variables in comparison with the traditional stochastic approach. Besides, and more importantly, a dramatic reduction of the computational effort required to obtain these optimal values in response to the change of the uncertain parameters is achieved. Thus the use of the proposed data-driven decision tool promotes a time-effective optimal decision making, which represents a step forward to use data-driven strategy in large-scale/complex industrial problems.Peer ReviewedPostprint (published version