Hierarchical reinforcement learning for trading agents

Autonomous software agents, the use of which has increased due to the recent growth in computer power, have considerably improved electronic commerce processes by facilitating automated trading actions between the market participants (sellers, brokers and buyers). The rapidly changing market environments pose challenges to the performance of such agents, which are generally developed for specific market settings. To this end, this thesis is concerned with designing agents that can gradually adapt to variable, dynamic and uncertain markets and that are able to reuse the acquired trading skills in new markets. This thesis proposes the use of reinforcement learning techniques to develop adaptive trading agents and puts forward a novel software architecture based on the semi-Markov decision process and on an innovative knowledge transfer framework. To evaluate my approach, the developed trading agents are tested in internationally well-known market simulations and their behaviours when buying or/and selling in the retail and wholesale markets are analysed. The proposed approach has been shown to improve the adaptation of the trading agent in a specific market as well as to enable the portability of the its knowledge in new markets

Production planning of biopharmaceutical manufacture.

Multiproduct manufacturing facilities running on a campaign basis are increasingly becoming the norm for biopharmaceuticals, owing to high risks of clinical failure, regulatory pressures and the increasing number of therapeutics in clinical evaluation. The need for such flexible plants and cost-effective manufacture pose significant challenges for planning and scheduling, which are compounded by long production lead times, intermediate product stability issues and the high cost - low volume nature of biopharmaceutical manufacture. Scheduling and planning decisions are often made in the presence of variable product titres, campaign durations, contamination rates and product demands. Hence this thesis applies mathematical programming techniques to the planning of biopharmaceutical manufacture in order to identify more optimal production plans under different manufacturing scenarios. A deterministic mixed integer linear programming (MILP) medium term planning model which explicitly accounts for upstream and downstream processing is presented. A multiscenario MILP model for the medium term planning of biopharmaceutical manufacture under uncertainty is presented and solved using an iterative solution procedure. An alternative stochastic formulation for the medium term planning of biomanufacture under uncertainty based on the principles of chance constrained programming is also presented. To help manage the risks of long term capacity planning in the biopharmaceutical industry, a goal programming extension is presented which accounts for multiple objectives including cost, risk and customer service level satisfaction. The model is applied to long term capacity analysis of a mix of contractors and owned biopharmaceutical manufacturing facilities. In the final sections of this thesis an example of a commercial application of this work is presented, followed by a discussion on related validation issues in the biopharmaceutical industry. The work in this thesis highlighted the benefits of applying mathematical programming techniques for production planning of biopharmaceutical manufacturing facilities, so as to enhance the biopharmaceutical industry's strategic and operational decision-making towards achieving more cost-effective manufacture

Sustainable Inventory Management Model for High-Volume Material with Limited Storage Space under Stochastic Demand and Supply

Inventory management and control has become an important management function, which is vital in ensuring the efficiency and profitability of a company’s operations. Hence, several research studies attempted to develop models to be used to minimise the quantities of excess inventory, in order to reduce their associated costs without compromising both operational efficiency and customers’ needs. The Economic Order Quantity (EOQ) model is one of the most used of these models; however, this model has a number of limiting assumptions, which led to the development of a number of extensions for this model to increase its applicability to the modern-day business environment. Therefore, in this research study, a sustainable inventory management model is developed based on the EOQ concept to optimise the ordering and storage of large-volume inventory, which deteriorates over time, with limited storage space, such as steel, under stochastic demand, supply and backorders. Two control systems were developed and tested in this research study in order to select the most robust system: an open-loop system, based on direct control through which five different time series for each stochastic variable were generated, before an attempt to optimise the average profit was conducted; and a closed-loop system, which uses a neural network, depicting the different business and economic conditions associated with the steel manufacturing industry, to generate the optimal control parameters for each week across the entire planning horizon. A sensitivity analysis proved that the closed-loop neural network control system was more accurate in depicting real-life business conditions, and more robust in optimising the inventory management process for a large-volume, deteriorating item. Moreover, due to its advantages over other techniques, a meta-heuristic Particle Swarm Optimisation (PSO) algorithm was used to solve this model. This model is implemented throughout the research in the case of a steel manufacturing factory under different operational and extreme economic scenarios. As a result of the case study, the developed model proved its robustness and accuracy in managing the inventory of such a unique industry

Interaction of optimization models and information sharing in a two echelon supply chain

Uncertainty in the manufacturing industry has been a research interest for many years. Deterministic and stochastic optimization methods have been proposed in the past. The objective of this thesis is to study the interaction of these models in a supply chain with a varying error in demand forecast. All the possible combinations of the optimization strategies in a two-echelon supply chain have been considered. Results indicate that the performance of the supply chain is driven by the choice of strategy of the supplier. Stochastic optimization is very efficient in lowering the operational costs and bull-whip effect in most cases. However, in cases where the trend in demand variation is smooth, use of deterministic strategy by both stakeholders is beneficial and it helps in lowering operational cost. Information sharing results in cost saving in most of the cases. It increases with increase in root mean squared error in demand forecast when the supplier uses deterministic strategy

Performance Evaluation of Public Transport Networks and Its Optimal Strategies Under Uncertainty

The study introduces a novel framework to enhance public transportation performance in uncertain situations, incorporating multi-aspiration-level goal programming and Monte Carlo simulation to manage uncertainty. The process involves creating a public transport criteria matrix using an analytic hierarchy process and optimizing the network based on weight results. Three Australian case studies are used to validate the proposed methodology

Computational methods and parallel strategies in dynamic decision making

Cada uno de estos objetivos han sido tratados en un capítulo independiente de esta tesis. En el segundo capítulo, un modelo de programación estocástica es presentado para un problema práctico de planificación de producción de un producto perecedero en un horizonte de tiempo finito. Una política estática es estudiada para el modelo. Tal política ha demostrado ser óptima asumiendo una estrategia de incertidumbre estática, que es considerada para instancias con un tiempo de espera largo. El tercer capítulo trata el uso de computación paralela para los algoritmos desarrollados en el capítulo previo. Dos implementaciones fueron desarrolladas para plataformas heterogéneas: una versión multi-GPU usando CUDA y una versión multinúcleo usando Pthreads y MPI. Para la primera implementación la simulación de Monte Carlo (la tarea más costosa) es paralelizada. La versión multinúcleo mostró una buena escalabilidad, una vez tratada la carga no balanceada entre los procesadores. El cuarto capítulo trata la efectividad de heurísticas para un problemas de tamaño de lote de productos perecederos similar. La clásica heurística de Silver es extendida para productos perecederos y se presentan variantes del procedimiento: una analítica y una basada en simulación. Los resultados de la heurística son comparados con las soluciones óptimas dadas por un modelo SDP generado para el problema, mostrando que los costes de las heurísticas son se presentan, de media, un 5% sobre el coste óptimo para la estrategia basada en simulación y un 6% para la aproximación analítica. En el quinto capítulo, se presenta un modelo MILP para seleccionar la flota de embarcaciones óptima para el mantenimiento de un parque eólico marino. El modelo se presenta como un problema de dos niveles, seleccionando la flota optima en el primer nivel y optimizando la programación de las operaciones, usando dicha flota, en el segundo. Dado que el modelo es determinístico, como otros en la literatura que aspiran a resolver problemas con un horizonte temporal largo usando periodos cortos, el sexto capítulo trata la cuestión de cómo la anticipación de los eventos estocásticos como los fallos en las turbinas o las condiciones meteorológicas afectan la decisión de la flota de embarcaciones óptima. Este capítulo presenta una heurística que ilustra este efecto.Esta tesis analiza aplicaciones de toma de decisiones dinámica para un conjunto de problemas. Pueden diferenciarse dos líneas principales. La primera trata problemas de gestión de la cadena de suministro para productos perecederos, mientras que la segunda estudia el diseño de flotas de embarcaciones para realizar labores de mantenimiento en parques eólicos marinos. Los modelos de inventario para productos perecederos estudiados en esta tesis consideran un único producto, única localización de suministro y una planificación de producción sobre un horizonte de tiempo finito. El problema de toma de decisiones para programar las operaciones de mantenimiento en parques eólicos marinos es tratado como un problema de cadena de suministro: la instalación requiere programar operaciones de mantenimiento y atender los fallos en turbinas durante el horizonte planificado. Una flota de embarcaciones tiene que ser seleccionada para realizar estas operaciones. Para este conjunto de problemas, las decisiones no son solo dinámicas, sino que además se realizan bajo incertidumbre. Los principales objetivos de esta tesis son los siguientes: (1) estudiar que políticas de pedido son las más apropiadas para los problemas de tamaño de lote? ¿En qué casos una política de pedido da una solución óptima?; (2) analizar el efecto del uso de computación paralela para mejorar el rendimiento de los algoritmos derivados para diseñar políticas para problemas de tamaño de lote de productos perecederos; (3) explorar como de efectivas pueden ser las heurísticas para problemas de toma de decisiones dinámica sobre tamaño de lote de productos perecederos; (4) elaborar un modelo MILP para seleccionar una flota de embarcaciones para realizar las operaciones de mantenimiento en parques eólicos marinos; y (5), diseñar una heurística para programar las operaciones de mantenimiento en parques eólicos marinos considerando fallos en turbinas e incertidumbre meteorológica

Toward digital twins for sawmill production planning and control : benefits, opportunities and challenges

Sawmills are key elements of the forest product industry supply chain, and they play important economic, social, and environmental roles. Sawmill production planning and control are, however, challenging owing to severalfactors, including, but not limited to, the heterogeneity of the raw material. The emerging concept of digital twins introduced in the context of Industry 4.0 has generated high interest and has been studied in a variety of domains, including production planning and control. In this paper, we investigate the benefits digital twins would bring to the sawmill industry via a literature review on the wider subject of sawmill production planning and control. Opportunities facilitating their implementation, as well as ongoing challenges from both academic and industrial perspectives, are also studied

Integrated optimisation for production capacity, raw material ordering and production planning under time and quantity uncertainties based on two case studies

Abstract This paper develops a supply chain (SC) model by integrating raw material ordering and production planning, and production capacity decisions based upon two case studies in manufacturing firms. Multiple types of uncertainties are considered; including: time-related uncertainty (that exists in lead-time and delay) and quantity-related uncertainty (that exists in information and material flows). The SC model consists of several sub-models, which are first formulated mathematically. Simulation (simulation-based stochastic approximation) and genetic algorithm tools are then developed to evaluate several non-parameterised strategies and optimise two parameterised strategies. Experiments are conducted to contrast these strategies, quantify their relative performance, and illustrate the value of information and the impact of uncertainties. These case studies provide useful insights into understanding to what degree the integrated planning model including production capacity decisions could benefit economically in different scenarios, which types of data should be shared, and how these data could be utilised to achieve a better SC system. This study provides insights for small and middle-sized firm management to make better decisions regarding production capacity issues with respect to external uncertainty and/or disruptions; e.g. trade wars and pandemics.</jats:p