Intelligent Scheduling Method for Bulk Cargo Terminal Loading Process Based on Deep Reinforcement Learning

Funding Information: Funding: This research was funded by the National Natural Science Foundation of China under Grant U1964201 and Grant U21B6001, the Major Scientific and Technological Special Project of Hei-longjiang Province under Grant 2021ZX05A01, the Heilongjiang Natural Science Foundation under Grant LH2019F020, and the Major Scientific and Technological Research Project of Ningbo under Grant 2021Z040. Publisher Copyright: © 2022 by the authors. Licensee MDPI, Basel, Switzerland.Sea freight is one of the most important ways for the transportation and distribution of coal and other bulk cargo. This paper proposes a method for optimizing the scheduling efficiency of the bulk cargo loading process based on deep reinforcement learning. The process includes a large number of states and possible choices that need to be taken into account, which are currently performed by skillful scheduling engineers on site. In terms of modeling, we extracted important information based on actual working data of the terminal to form the state space of the model. The yard information and the demand information of the ship are also considered. The scheduling output of each convey path from the yard to the cabin is the action of the agent. To avoid conflicts of occupying one machine at same time, certain restrictions are placed on whether the action can be executed. Based on Double DQN, an improved deep reinforcement learning method is proposed with a fully connected network structure and selected action sets according to the value of the network and the occupancy status of environment. To make the network converge more quickly, an improved new epsilon-greedy exploration strategy is also proposed, which uses different exploration rates for completely random selection and feasible random selection of actions. After training, an improved scheduling result is obtained when the tasks arrive randomly and the yard state is random. An important contribution of this paper is to integrate the useful features of the working time of the bulk cargo terminal into a state set, divide the scheduling process into discrete actions, and then reduce the scheduling problem into simple inputs and outputs. Another major contribution of this article is the design of a reinforcement learning algorithm for the bulk cargo terminal scheduling problem, and the training efficiency of the proposed algorithm is improved, which provides a practical example for solving bulk cargo terminal scheduling problems using reinforcement learning.publishersversionpublishe

ANN application in maritime industry : Baltic Dry Index forecasting & optimization of the number of container cranes

This dissertation is a study of dry bulk freight index forecasting and port planning, both based on Artificial Neural network application. First the dry bulk market is reviewed, and the reason for the high fluctuation of freight rates through the demand-supply mechanism is examined. Due to the volatile BDI, the traditional linear regression forecasting method cannot guarantee the performance of forecasting, but ANN overcomes this difficulty and gives better performance especially in a short time. Besides, in order to improve the performance of ANN further, wavelet is introduced to pre-process the BDI data. But when the noise (high frequency parts) is stripped, the hidden useful data may also be eliminated. So the performance of different degrees of de-noising models is evaluated, and the best one (most suitable de-noising model) is chosen to forecast BDI, which avoids over de-noising and keeps a fair ability of forecasting. In the second case study, the collected container terminals and ranked, and the throughput of each combination (different crane number) is estimated by applying a trained BP network. The BP network with DEA output is combined, simulating the efficiency of each combination. And finally, the optimal container crane number is fixed due to the highest efficiency and practical reasons. The Conclusion and Recommendation chapter gives some further advice, and many recommendations are given

The synergistic effect of operational research and big data analytics in greening container terminal operations: a review and future directions

Container Terminals (CTs) are continuously presented with highly interrelated, complex, and uncertain planning tasks. The ever-increasing intensity of operations at CTs in recent years has also resulted in increasing environmental concerns, and they are experiencing an unprecedented pressure to lower their emissions. Operational Research (OR), as a key player in the optimisation of the complex decision problems that arise from the quay and land side operations at CTs, has been therefore presented with new challenges and opportunities to incorporate environmental considerations into decision making and better utilise the ‘big data’ that is continuously generated from the never-stopping operations at CTs. The state-of-the-art literature on OR's incorporation of environmental considerations and its interplay with Big Data Analytics (BDA) is, however, still very much underdeveloped, fragmented, and divergent, and a guiding framework is completely missing. This paper presents a review of the most relevant developments in the field and sheds light on promising research opportunities for the better exploitation of the synergistic effect of the two disciplines in addressing CT operational problems, while incorporating uncertainty and environmental concerns efficiently. The paper finds that while OR has thus far contributed to improving the environmental performance of CTs (rather implicitly), this can be much further stepped up with more explicit incorporation of environmental considerations and better exploitation of BDA predictive modelling capabilities. New interdisciplinary research at the intersection of conventional CT optimisation problems, energy management and sizing, and net-zero technology and energy vectors adoption is also presented as a prominent line of future research

A Data-driven and multi-agent decision support system for time slot management at container terminals: A case study for the Port of Rotterdam

Controlling the departure time of the trucks from a container hub is important to both the traffic and the logistics systems. This, however, requires an intelligent decision support system that can control and manage truck arrival times at terminal gates. This paper introduces an integrated model that can be used to understand, predict, and control logistics and traffic interactions in the port-hinterland ecosystem. This approach is context-aware and makes use of big historical data to predict system states and apply control policies accordingly, on truck inflow and outflow. The control policies ensure multiple stakeholders satisfaction including those of trucking companies, terminal operators, and road traffic agencies. The proposed method consists of five integrated modules orchestrated to systematically steer truckers toward choosing those time slots that are expected to result in lower gate waiting times and more cost-effective schedules. The simulation is supported by real-world data and shows that significant gains can be obtained in the system

An Improved Fuzzy Knowledge-Based Model For Long Stay Container Yards

This paper considers the problem of allocating newly arrived containers to stacks of existing containers in a yard when the departure date/time for containers is unknown. Many factors and constraints need to be considered when modelling this storage allocation problem. These constraints include the size, type and weight of the containers. The factors are the number of containers in a stack and the duration of stay of the topmost container in the stack. This paper aims to develop an improved Fuzzy Knowledge-Based ‘FKB’ model for best allocation practice of long-stay containers in a yard. In this model, the duration of stay factor does not need to be considered in the allocation decision if the duration of stay for the topmost containers in a stack is similar; hence, a new ‘ON/OFF’ strategy is proposed within the Fuzzy Knowledge-Based model to activate/deactivate this factor in the stacking algorithm whenever is required. Discrete Event Simulation and Fuzzy Knowledge-Based techniques are used to develop the proposed model. The model’s behaviour is tested using three real-life scenarios, including allocating containers in busy, moderately busy and quiet yards. The total number of re-handlings, the number of re-handlings per stack, and the number of re-handlings for containers were considered KPIs in each scenario

Sequence-Based Simulation-Optimization Framework With Application to Port Operations at Multimodal Container Terminals

It is evident in previous works that operations research and mathematical algorithms can provide optimal or near-optimal solutions, whereas simulation models can aid in predicting and studying the behavior of systems over time and monitor performance under stochastic and uncertain circumstances. Given the intensive computational effort that simulation optimization methods impose, especially for large and complex systems like container terminals, a favorable approach is to reduce the search space to decrease the amount of computation. A maritime port can consist of multiple terminals with specific functionalities and specialized equipment. A container terminal is one of several facilities in a port that involves numerous resources and entities. It is also where containers are stored and transported, making the container terminal a complex system. Problems such as berth allocation, quay and yard crane scheduling and assignment, storage yard layout configuration, container re-handling, customs and security, and risk analysis become particularly challenging. Discrete-event simulation (DES) models are typically developed for complex and stochastic systems such as container terminals to study their behavior under different scenarios and circumstances. Simulation-optimization methods have emerged as an approach to find optimal values for input variables that maximize certain output metric(s) of the simulation. Various traditional and nontraditional approaches of simulation-optimization continue to be used to aid in decision making. In this dissertation, a novel framework for simulation-optimization is developed, implemented, and validated to study the influence of using a sequence (ordering) of decision variables (resource levels) for simulation-based optimization in resource allocation problems. This approach aims to reduce the computational effort of optimizing large simulations by breaking the simulation-optimization problem into stages. Since container terminals are complex stochastic systems consisting of different areas with detailed and critical functions that may affect the output, a platform that accurately simulates such a system can be of significant analytical benefit. To implement and validate the developed framework, a large-scale complex container terminal discrete-event simulation model was developed and validated based on a real system and then used as a testing platform for various hypothesized algorithms studied in this work

Control of free-ranging automated guided vehicles in container terminals

Container terminal automation has come to the fore during the last 20 years to improve their efficiency. Whereas a high level of automation has already been achieved in vertical handling operations (stacking cranes), horizontal container transport still has disincentives to the adoption of automated guided vehicles (AGVs) due to a high degree of operational complexity of vehicles. This feature has led to the employment of simple AGV control techniques while hindering the vehicles to utilise their maximum operational capability. In AGV dispatching, vehicles cannot amend ongoing delivery assignments although they have yet to receive the corresponding containers. Therefore, better AGV allocation plans would be discarded that can only be achieved by task reassignment. Also, because of the adoption of predetermined guide paths, AGVs are forced to deploy a highly limited range of their movement abilities while increasing required travel distances for handling container delivery jobs. To handle the two main issues, an AGV dispatching model and a fleet trajectory planning algorithm are proposed. The dispatcher achieves job assignment flexibility by allowing AGVs towards to container origins to abandon their current duty and receive new tasks. The trajectory planner advances Dubins curves to suggest diverse optional paths per origin-destination pair. It also amends vehicular acceleration rates for resolving conflicts between AGVs. In both of the models, the framework of simulated annealing was applied to resolve inherent time complexity. To test and evaluate the sophisticated AGV control models for vehicle dispatching and fleet trajectory planning, a bespoke simulation model is also proposed. A series of simulation tests were performed based on a real container terminal with several performance indicators, and it is identified that the presented dispatcher outperforms conventional vehicle dispatching heuristics in AGV arrival delay time and setup travel time, and the fleet trajectory planner can suggest shorter paths than the corresponding Manhattan distances, especially with fewer AGVs.Open Acces