Reinforcement Learning Approaches to Improve Spatial Reuse in Wireless Local Area Networks

The ubiquitous deployment of IEEE 802.11 based Wireless Local Area Networks (WLANs) or WiFi networks has resulted in dense deployments of Access Points (APs) in an effort to provide wireless links with high data rates to users. This, however, causes APs and users/stations to experience a higher interference level. This is because of the limited spectrum in which WiFi networks operate, resulting in multiple APs operating on the same channel. This in turn affects the signal-tonoise-plus interference ratio (SINR) at APs and users, leading to low data rates that limit their quality of service (QoS). To improve QoS, interference management is critical. To this end, a key metric of interest is spatial reuse. A high spatial reuse means multiple transmissions are able to transmit concurrently, which leads to a high network capacity. One approach to optimize spatial reuse is by tuning the clear channel access (CCA) threshold employed by the carrier sense multiple access with collision avoidance (CSMA/CA) medium access control (MAC) protocol. Specifically, the CCA threshold of a node determines whether it is allowed to transmit after sensing the channel. A node may increase its CCA threshold, causing it to transmit even when there are other ongoing transmissions. Another parameter to be tuned is transmit power. This helps a transmitting node lower its interference to neighboring cells, and thus allows nodes in these neighboring cells to transmit as well. Apart from that, channel bonding can be applied to improve transmission rate. In particular, by combining/aggregating multiple channels together, the resulting channel has a proportionally higher data rate than the case without channel bonding. However, the issue of spatial reuse remains the same whereby the focus is to maximize the number of concurrent transmissions across multiple channels

Allocation of Public Resources: Bringing Order to Chaos

Science Olympiad (SO) is a team-based academic competition involving multiple subject areas (Events) with arcane rules governing the team composition. Add to the mix parental contention over which student(s) get on the “All-Star” team, and you have a potentially explosive situation. This project brings order and logic to school-based SO programs and defuses tense milestones through the implementation of an institutional structure that: assigns students to Events based on solicited student preferences for the Events, collects objective student performance data, composes competitive teams based on student performance (aka “Moneyball”), and brings transparency to the Team Selection process through crowdsourcing. The Event Assignment mechanism is simple, fast, easy to understand, and yields Pareto-optimal results based on student preferences, without the exchange of money or tokens, and with effectively no incentive to game the system. The Team Selection mechanism optimizes student performance data from teachers (Event Coaches) and competitions to compose a tiered series of teams with the greatest potential performance. And the Crowdsource Tool allows any stakeholder to compose a candidate team for advancing to the State competition, where the team with the highest potential performance score advances to State whether the team was composed with the Crowdsource Tool or by the Team Selection algorithm. The end result is that students get more of the Events that they want; Team Selection is transparent and far less contentious; teams are higher quality; and managing the SO program for a school takes considerably less time and effort

Optimization-based decision support systems for planning problems in processing industries

Summary Optimization-based decision support systems for planning problems in processing industries Nowadays, efficient planning of material flows within and between supply chains is of vital importance and has become one of the most challenging problems for decision support in practice. The tremendous progress in hard- and software of the past decades was an important gateway for developing computerized systems that are able to support decision making on different levels within enterprises. The history of such systems started in 1971 when the concept of Decision Support Systems (DSS) emerged. Over the years, the field of DSS has evolved into a broad variety of directions. The described research in this thesis limits to the category of model-driven or optimization-based DSS. Simultaneously with the emergence of DSS, software vendors recognized the high potentials of available data and developed Enterprise Systems to standardize planning problems. Meanwhile, information oriented systems like MRP and its successors are extended by the basic concepts of optimization based decision support. These systems are called Advanced Planning Systems (APS). The main focus of APS is to support decision making at different stages or phases in the material flow, i.e. from procurement, production, distribution to sales (horizontal-axis), on different hierarchical aggregation levels (vertical-axis) ranging from strategic (long-term) to operational (short- term) planning. This framework of building blocks decomposes planning tasks hierarchically into partial planning problems. This basic architecture of the planning processes in APS is known as the Supply Chain Planning Matrix (SCPM). Compared to, for instance, discrete parts manufacturing, planning tasks are much more complicated in processing industries due to a natural variation in the composition of raw materials, the impact of processing operations on properties of material flows, sequence dependent change-over times, the inevitable decline in quality of product flows and relatively low margins. These specific characteristics gave rise to focus on optimization-based decision support in the domain of processing industries. The problems to be addressed in this field call for (inter-related) decisions with respect to the required raw materials, the production quantities to be manufactured, the efficient use of available resources, and the times at which raw materials must be available. Although different APS modules can interact directly, coordination and integration is often restricted to the exchange of data flows between different modules. Given the need for specific integrated decision support, the research presented in this thesis focusses particularly on medium to short term decision support at production stage in processing industry, including the vertical and horizontal integration and coordination with adjacent building blocks in the SCPM. Extensive reviews from literature show that the gap between research and practice of DSS is widening. As the field of DSS was initiated as an application oriented discipline, the strategy of what is referred to as “application-driven theory” was taken as the preferred approach for this thesis. “Application-driven” refers to a bottom-up approach which means that the relevance of the research should both be initiated and obtained from practice. The intended successful use of the proposed approaches should, where possible, be represented by tests of adequacy. Simultaneously, the contribution to “theory” aims to be a recognizable part of the research effort, i.e. obtained understanding and insights from problems in practice should provide the basis for new approaches. Based on the preceding considerations we defined the following general research objective: General research objective To support medium- to short term planning problems by optimization-based models and solution techniques such that: i) The applicability and added value of (prototype) systems is recognized and carried by decision makers in practice ii) The proposed approaches contribute to knowledge, understanding and insights from a model building and – solving point of view. In order to link the general objective with the different studies in the thesis, we defined five, recurring research premises, i.e. Professional relevance and applicability (P1), Aggregation (P2), Decomposition and reformulation (P3), Vertical integration at production level (P4), and Horizontal coordination and integration (P5). The overarching premise P1 refers to the first part of the research objective. All other premises refer to the second part of the research objective, i.e. model building and/or – solving. Several planning issues are studied to give substance to the research objective and each study is connected to at least two research premises. Study 1: Planning and scheduling in food processing industry The main question in Chapter 2 was:” How to apply aggregation, decomposition and reformulation in model-based DSS at planning and scheduling level such that the aspect of decision support is recognized and appreciated by decision makers in practice, and which level of aggregation is needed to integrate production planning (i.e. lot-sizing) and scheduling problems in a single model? The study consists of two parts. The first part of the study refers to a case study for the bottleneck packaging facilities of a large dairy company. The goal was to develop, implement and test a pilot DSS which was able to deliver solutions recognized and carried by decision makers at lower decision levels. The latter aim implied that a straight-forward aggregation on time, product type, resources or product stage, was not preferred. The key to develop an approach for regular use was to identify and take advantage of specific problem characteristics. Clustering of numerous jobs, while retaining information at order level, could be exploited in a reformulation approach. The inclusion of (combined) generalized- and variable upper bound constraints gave very tight lower bounds and sparse search trees. An extensive test phase in daily practice showed that the main benefit of the DSS was the initial quality of the generated plans including the time needed to generate these schedules. Hence, decision makers could i) postpone their planning tasks, ii) conveniently cope with rush orders or planned maintenance and iii) easily generate alternatives or revised plans when unforeseen disturbances occur. Moreover, the graphical presentation and overview of the (future) working schedule enabled order acceptance to make use of remaining capacity. The study also showed that planning problems in practice cannot be captured exhaustively by a (simplified) model. Decision makers need the opportunity to modify automatically generated plans manually and use human judgement and experience such that the solution is tuned to the actual situation. Hence, the DSS should not be considered as an optimizer but rather as a tool for generating high quality plans to be used for further analysis. Within this context the various options of a user-friendly, graphical, and fully interactive user interface, were of major importance. Although the case study clearly demonstrates the validity of earlier case based DSS research for current days APS, the proposed approach is hardly a generic solution for a complete vertical integration between lot-sizing and scheduling. If lot-size decisions are strongly affected by the sequence of jobs, production planning and scheduling should be performed simultaneously. As the described case refers to an earlier study and today’s APS do not provide modules for integrated lot-sizing and scheduling, the second part of the study gives an overview of developments in literature regarding lot-sizing and scheduling models and assess their suitability for addressing sequence-dependent setups, non-triangular setups and product decay. The review shows a tendency in which so-called Big Bucket (BB) models are currently proposed for short term time horizons too. However, we argue that segmentation of the planning horizon is a key issue for simultaneous lot-sizing and scheduling. The advantage of BB models may become a major obstacle for i) the effectiveness of simultaneous lot-sizing and scheduling, and ii) addressing specific characteristics in food processing industry. Study 2: Vertical integration of lot-sizing and scheduling in food processing industry Chapter 3 focused on a complete integration of lot-sizing and scheduling decisions in a single model. The main question was:” How to integrate production planning (i.e. lot- sizing) and scheduling problems in a single model, such that common assumptions regarding the triangular setup conditions are relaxed and issues of product decay and limited shelf lives are taken into account?” The literature research in Chapter 2 revealed that the computational advantage of time oriented aggregation in BB models may become a major obstacle in addressing the identified characteristics in FPI. In addition, product decay is primarily associated with the “age” of products and consequently relates to the segmentation of the time- horizon. Therefore, two SB models are developed to demonstrate the impact of non- triangular setups and product decay on the generated solutions. Small scale examples were used to demonstrate how a small change in the balance between inventory - and changeover costs may generate significantly different solutions, especially when the triangular setup conditions do not hold. The developed models are potentially very large formulations and, as expected, hard to solve. Exploratory research was conducted with a Relax-and-Fix (R&F) heuristic. The heuristic is based on a decomposition of the time horizon. Numerical results of small to medium sized problem instances are promising. However, solving real-size problem instances is not possible yet. Study 3: Integrated planning between procurement and production The case study in Chapter 4 focussed on the need for horizontal coordination and integration between the phases procurement and production, which is of particular importance in inter-organizational supply chains. The main question was:” How to model and solve an integrated planning problem between procurement and production, both on a mid-term and short-term planning level, in an inter-organizational supply chain? The research question was projected on an illustrative milk collection problem in practice. The aim was to develop a pilot DSS that lifted decision support for a “weaker” partner in a food supply chain to a higher level, and to illustrate the importance of horizontal integration between the phases procurement and production in an APS framework. Problem analysis revealed that the problem can be classified as an extension of the Periodic Vehicle Routing Problem (PVRP). The problem was decomposed into more tractable sub problems on different hierarchical levels, i.e. the daily (vehicle) routing problem was separated from a medium-term planning problem. On the higher planning level, numerous suppliers were aggregated such that total supply within a cluster met (multiple) vehicle loading capacities. The continuous supply of relatively small amounts from many suppliers had to be balanced with strict delivery conditions at processing level. A model was developed to assign a single (stable) collection rhythm to each cluster such that the total, weighted deviation of desired processing levels on various days in the planning horizon was minimized. The applied aggregation on the higher planning level turned out to be very beneficial for the required disaggregation at the lower planning level. Once supplier farms were geographically grouped into clusters and the aggregated supply within a cluster was assigned to a single collection rhythm with fixed collection days, the (initial) daily routing problem was considerably easier to solve for vehicle schedulers. The computational complexity of the problem was reduced by exploiting application-based properties algorithmically in a specific branch-and-bound scheme, i.e. a customized approach of Special Ordered Sets type 1 (SOS1) This approach made it possible to solve the generated problems exactly for real-size problem instances. The various facilities of a user-friendly and interactive man-machine interface (i.e. an input, planning, simulation and analysing module) turned out to be essential. Decision makers could easily change the data, and the generated plans, in a separate simulation module. However, the impact of any modification was immediately visualised by several (conflicting) indicators in the output screens, both on supply and demand level. Study 4: Mixed Integer (0-1) Fractional Programming in Paper Production Industry The study in Chapter 5 focussed on the impact of technical settings of production units on material flows. The main question was:” How to support decision-makers in practice if crucial properties of end products simultaneously depend on (endogenous) types of raw materials with different chemical or physical properties and (endogenous) technical settings of processing units? The goal of the study was to revise and upgrade an existing, locally used DSS, to a tailored and flexible tool for decision support within the enterprise. The study revealed that the aimed extension towards multi-objective decision support, together with new physical insight for calculating properties of end products due to process operations, had a substantial impact on the optimization module. The proposed solution procedure takes advantage of the problem characteristics and gives rise i) to apply and extend a classical reformulation approach for continuous linear fractional programming (FP) problems to a more general class of mixed integer (binary) FP problems and ii) to exploit the special structure between the original non- linear mixed integer model and the continuous, linear reformulation by applying the concept of Special Ordered Sets type 1 (SOS1). Although Chapter 5 focusses in particular on the reformulation and solution approach, the DSS consists of four main building blocks, i.e. the user interface, a scenario manager, a simulation- and optimization routine. The optimization module provides a powerful tool to find feasible solutions and the best (unexpected) recipes for any available set of raw materials. Moreover, it provides an innovative way of decision support for purchasing (new) pulps on the market, for assigning available pulps to different paper grades, and for attuning available stock levels of raw materials to (changing) production targets for different paper grades. The results of the optimization routine are mainly used to obtain alternative recipes for different paper grades. Usually, these recipes are stored as base scenarios and adapted to daily practice in the simulation module. Main conclusions and future research Based on the studies in the Chapters 2 and 3 we conclude that no generically applicable models and/or solution approaches exist for simultaneous planning and scheduling in processing industries. More industry-specific solutions are needed incorporating specificities of different production environments into those models. The key to develop solvable approaches for contemporary practice may be i) to use knowledge and experience from practice and take advantage of specific characteristics in different problem domains during model construction, and/or ii) to identify and exploit special problem structures for solving the related models. We conclude that surprisingly little research has been devoted to issues of coordination and integration between “procurement” and “production”. The studies in the chapters 4 and 5 confirm that sourcing of (raw) materials flows needs more attention in processing industries, particularly in push-oriented, inter-organizational networks. The valorisation of raw materials can be improved even more if the composition of raw materials is considered too in future planning problems at production level. In the second part of this thesis we focused on extensions for the applicability of Special Ordered Sets type 1 (SOS1), both from an algorithmic (Chapter 4) and modelling (Chapter 5) point of view. We conclude that the concept of SOS1 can extend a classical reformulation approach for continuous fractional programming (FP) problems, to a specific class of mixed integer (0-1) FP problems. Moreover, we conclude that a natural ordering of the variables within the sets is not necessary to make their use worthwhile. A separate (user defined) reference row or weights associated to the variables in the sets might be omitted for an efficient use of SOS1 in commercially available mathematical programming packages. However, this requires further research and extensive computational tests.</p

UAV-Aided Lifelong Learning for AoI and Energy Optimization in Non-Stationary IoT Networks

In this paper, a novel joint energy and age of information (AoI) optimization framework for IoT devices in a non-stationary environment is presented. In particular, IoT devices that are distributed in the real-world are required to efficiently utilize their computing resources so as to balance the freshness of their data and their energy consumption. To optimize the performance of IoT devices in such a dynamic setting, a novel lifelong reinforcement learning (RL) solution that enables IoT devices to continuously adapt their policies to each newly encountered environment is proposed. Given that IoT devices have limited energy and computing resources, an unmanned aerial vehicle (UAV) is leveraged to visit the IoT devices and update the policy of each device sequentially. As such, the UAV is exploited as a mobile learning agent that can learn a shared knowledge base with a feature base in its training phase, and feature sets of a zero-shot learning method in its testing phase, to generalize between the environments. To optimize the trajectory and flying velocity of the UAV, an actor-critic network is leveraged so as to minimize the UAV energy consumption. Simulation results show that the proposed lifelong RL solution can outperform the state-of-art benchmarks by enhancing the balanced cost of IoT devices by $8.3\%$ when incorporating warm-start policies for unseen environments. In addition, our solution achieves up to $49.38\%$ reduction in terms of energy consumption by the UAV in comparison to the random flying strategy.Comment: 15 pages, 14 figure

Planification socio-responsable du travail dans les chaînes de montage d'aéronefs : comment satisfaire à la fois objectifs ergonomiques et économiques

Dans cette thèse, le problème de planification des tâches dans les chaînes de montage des aéronefs est étudié. Ces lignes de production sont principalement manuelles et tactées. L'échec de la livraison dans les délais pouvant entraîner des pénalités importantes pour le fabricant, il est essentiel de respecter le calendrier de chaque poste de travail en tenant compte à la fois de critères économiques et ergonomiques. Ce problème de planification peut être considéré comme un problème généralisé de planification de projets avec contraintes de ressources (RCPSP). Dans un premier temps, nous passons en revue les méthodes ergonomiques existantes qui peuvent être utilisées pour évaluer la charge de travail physique dans les lignes de production et examinons leur applicabilité au contexte des chaînes de montage d'aéronefs avec des temps de cycle longs. Sur la base de cette évaluation, nous développons des modèles mathématiques à introduire dans les problèmes considérés du RCPSP afin de prendre en compte l'impact ergonomique sur les opérateurs. Tenant compte de ces contraintes ergonomiques, le problème industriel initial est modélisé comme un RCPSP avec des contraintes et des objectifs spéciaux intégrant à la fois des aspects économiques et ergonomiques. Plusieurs formulations avec des opérateurs polyvalents, des ressources avec des capacités dépendantes du temps, des contraintes sur les facteurs ergonomiques et des tâches multimodales ordonnées par des relations de précédence complexes sont considérées. Des modèles de programmation par contraintes et de programmation linéaire en nombres entiers ont été développés pour ces formulations. Afin d'améliorer les procédures de solution, de nouvelles techniques de propagation de contraintes sont proposées et mises en œuvre. Un nouvel algorithme pour le calcul de la borne inférieure est également développé. L'efficacité des modèles et méthodes présentés est validée par des expériences numériques.In this thesis, the scheduling problem of tasks in aircraft assembly lines is studied. These production lines are mainly manual and paced. Since the failure of delivery on time may result in significant penalties for the manufacturer, it is crucial to meet the schedule at each workstation taking into account both economic and ergonomic criteria. This scheduling problem can be considered as a generalized Resource-Constraints Project Scheduling Problem (RCPSP). Firstly, we review the existing ergonomic methods that can be used to evaluate the physical workload in production lines and examine their applicability to the context of aircraft assembly lines with long takt times. On the basis of this evaluation, we develop mathematical models to be introduced in considered RCPSP problems in order to take into account the ergonomic impact on the operators. Taking into consideration these ergonomic constraints, the original industrial problem is modeled as a RCPSP with special constraints and objectives integrating both economic and ergonomic aspects. Several formulations with multi-skilled operators, resources with time-dependent capacities, constraints on ergonomic factors and multi-mode tasks ordered by precedence relations with time lags are considered. Constraint Programming and Integer Linear Programming models are developed for these formulations. In order to enhance the solution procedures, novel constraint propagation techniques are proposed and implemented. A new algorithm for lower bound calculation is developed as well. The efficiency of presented models and methods are validated through numerical experiments

Traveling Salesman Problem-Based Integration of Planning, Scheduling, and Optimal Control for Continuous Processes

Advanced decision making in the process industries requires efficient use of information available at different decision levels. Traditionally, planning, scheduling, and optimal control problems are solved in a decoupled way, neglecting their strong interdependence. Integrated planning, scheduling and optimal control (iPSC) aims to address this issue. Formulating the iPSC, results in a large scale nonconvex mixed integer nonlinear programming problem. In the present work, we propose a new approach for the iPSC of continuous processes aiming to reduce model and computational complexity. For the planning and scheduling, a Traveling Salesman Problem-based formulation is employed, where the planning periods are modeled in discrete time while the scheduling within each week is in continuous time. Another feature of the proposed iPSC framework is that backlog, idle production time, and multiple customers are introduced. The resulting problem is a mixed integer programming problem and different solution strategies are employed and analyzed

Construction-based metaheuristics for personnel scheduling problems

This thesis investigates the idea of balancing different constraints in order to find optimal solutions to two personnel scheduling problems, within the framework of constructive metaheuristic approaches. The two problems considered are a nurse scheduling problem, for which finding feasible solutions is known to be difficult and for which the hard and soft constraints are in direct conflict, and a medical student scheduling problem for which there is little relevant literature this second problem also has conflicting hard and soft constraints, but presents further conflict between the different soft constraints. The methods used to solve these problems are focused on two constructive metaheuristics in particular: Greedy Randomised Adaptive Search Procedures (GRASP) and Ant Colony Optimisation (ACO) and for each approach several construction heuristics are introduced and compared. Using GRASP, a number of local search neighbourhoods are established for each problem, while for ACO the suitability of three trail definitions are compared. In order to further explore the balance which may obtained between the different constraints and objectives for the two problems, hybrid constructions are investigated, incorporating exact methods which take advantage of the underlying structures of each problem with regards to feasibility. For medical student scheduling, this exact method was developed into a new type of construction mechanism providing much improved results over a standard heuristic approach. Further enhancements investigated include the use of problem-specific feedback for nurse scheduling and the use of an intelligent memory procedure for the medical student scheduling problem. For the nurse scheduling problem, the final algorithm developed was able to rival the best in the literature so far and produce optimal solutions for all available datasets. For the medical student scheduling problem, optimal solutions are not known, but the results obtained are very promising and provide a good basis for further study of the problem.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

A Digital Twin Framework for Production Planning Optimization: Applications for Make-To-Order Manufacturers

In this dissertation, we develop a Digital Twin framework for manufacturing systems and apply it to various production planning and scheduling problems faced by Make-To-Order (MTO) firms. While this framework can be used to digitally represent a particular manufacturing environment with high fidelity, our focus is in using it to generate realistic settings to test production planning and scheduling algorithms in practice. These algorithms have traditionally been tested by either translating a practical situation into the necessary modeling constructs, without discussion of the assumptions and inaccuracies underlying this translation, or by generating random instances of the modeling constructs, without assessing the limitations in accurately representing production environments. The consequence has been a serious gap between theory advancement and industry practice. The major goal of this dissertation is to develop a framework that allows for practical testing, evaluation, and implementation of new approaches for seamless industry adoption. We develop this framework as a modular software package and emphasize the practicality and configurability of the framework, such that minimal modelling effort is required to apply the framework to a multitude of optimization problems and manufacturing systems. Throughout this dissertation, we emphasize the importance of the underlying scheduling problems which provide the basis for additional operational decision making. We focus on the computational evaluation and comparisons of various modeling choices within the developed frameworks, with the objective of identifying models which are both effective and computationally efficient. In Part 1 of this dissertation, we consider a class of Production Planning and Execution problems faced by job shop manufacturing systems. In Part 2 of this dissertation, we consider a class of scheduling problems faced by manufacturers whose production system is dominated by a single operation