Combining reinforcement learning and conventional control to improve automatic guided vehicles tracking of complex trajectories

With the rapid growth of logistics transportation in the framework of Industry 4.0, automated guided vehicle (AGV) technologies have developed speedily. These systems present two coupled control problems: the control of the longitudinal velocity, essential to ensure the application requirements such as throughput and tag time, and the trajectory tracking control, necessary to ensure the proper accuracy in loading and unloading manoeuvres. When the paths are very short or have abrupt changes, the kinematic constraints play a restrictive role, and the tracking control becomes more challenging. In this case, advanced control strategies such as those based on intelligent techniques, including machine learning (ML) can be useful. Hence, in this work, we present an intelligent hybrid control scheme that combines reinforcement learning-based control (RLC) with conventional PI regulators to face both control problems simultaneously. On the one hand, PIs are used to control the speed of each wheel. On the other hand, the input reference of these regulators is calculated by the RLC in order to reduce the guiding error of the path tracking and to maintain the longitudinal speed. The latter is compared with a PID path following controller. The PID regulators have been tuned by genetic algorithms. The RLC allows the vehicle to learn how to improve the trajectory tracking in an adaptive way and thus, the AGV can face disturbances or unknown physical system parameters that may change due to friction and degradation of AGV mechanical components. Extensive simulation experiments of the proposed intelligent control strategy on a hybrid tricycle and differential AGV model, that considers the kinematics and the dynamics of the vehicle, prove the efficiency of the approach when following different demanding trajectories. The performance of the RL tracking controller in comparison with the optimized PID gives errors around 70% smaller, and the average maximum error is also 48% lower.Open access funding enabled and organized by Projekt DEAL

Supporting the design of automated guided vehicle systems in internal logistics

Applications of automated guided vehicle (AGV) systems are becoming increasingly widespread in internal logistics for performing transports automatically. Recent technological advancements in navigation and intelligence have improved the functionality of vehicles and together with attention to Industry 4.0 have created further interest in AGV systems in industry and academia. Research on AGV systems has mainly focused on technical aspects, but to support AGV system design and, thereby, be able to achieve the full potential from use of AGV systems in internal logistics, more knowledge is needed that takes further into consideration aspects related to humans and the organisation, alongside the technical aspects. The purpose of this thesis is to develop knowledge to support the design of AGV systems and three research questions are formulated. The thesis is based on three papers, two of which are based on multiple case studies and one study based on simulation modelling. The thesis results provide input to the design process for AGV systems in three main ways. First, in developing an understanding for which requirements influence an AGV systems and how the requirements can be met in the AGV system configuration. Second, regarding how the load capacity of AGVs impact the performance of the AGV system, and third by identifying challenges with respect to the work organisation and related to human factors when AGV systems are introduced in internal logistics settings

Parameter tuning and cooperative control for automated guided vehicles

For several practical control engineering applications it is desirable that multiple systems can operate independently as well as in cooperation with each other. Especially when the transition between individual and cooperative behavior and vice versa can be carried out easily, this results in ??exible and scalable systems. A subclass is formed by systems that are physically separated during individual operation, and very tightly coupled during cooperative operation. One particular application of multiple systems that can operate independently as well as in concert with each other is the cooperative transportation of a large object by multiple Automated Guided Vehicles (AGVs). AGVs are used in industry to transport all kinds of goods, ranging from small trays of compact and video discs to pallets and 40-tonne coils of steel. Current applications typically comprise a ??eet of AGVs, and the vehicles transport products on an individual basis. Recently there has been an increasing demand to transport very large objects such as sewer pipes, rotor blades of wind turbines and pieces of scenery for theaters, which may reach lengths of over thirty meters. A realistic option is to let several AGVs operate together to handle these types of loads. This Ph.D. thesis describes the development, implementation, and testing of distributed control algorithms for transporting a load by two or more Automated Guided Vehicles in industrial environments. We focused on the situations where the load is connected to the AGVs by means of (semi-)rigid interconnections. Attention was restricted to control on the velocity level, which we regard as an intermediate step for achieving fully automatic operation. In our setup the motion setpoint is provided by an external host. The load is assumed to be already present on the vehicles. Docking and grasping procedures are not considered. The project is a collaboration between the company FROG Navigation Systems (Utrecht, The Netherlands) and the Control Systems group of the Technische Universiteit Eindhoven. FROG provided testing facilities including two omni-directional AGVs. Industrial AGVs are custom made for the transportation tasks at hand and come in a variety of forms. To reduce development times it is desirable to follow a model-based control design approach as this allows generalization to a broad class of vehicles. We have adopted rigid body modeling techniques from the ??eld of robotic manipulators to derive the equations of motion for the AGVs and load in a systematic way. These models are based on physical considerations such as Newton's second law and the positions and dimensions of the wheels, sensors, and actuators. Special emphasis is put on the modeling of the wheel-??oor interaction, for which we have adopted tire models that stem from the ??eld of vehicle dynamics. The resulting models have a clear physical interpretation and capture a large class of vehicles with arbitrary wheel con??gurations. This ensures us that the controllers, which are based on these models, are applicable to a broad class of vehicles. An important prerequisite for achieving smooth cooperative behavior is that the individual AGVs operate at the required accuracy. The performance of an individual AGV is directly related to the precision of the estimates for the odometric parameters, i.e. the effective wheel diameters and the offsets of the encoders that measure the steering angles of the wheels. Cooperative transportation applications will typically require AGVs that are highly maneuverable, which means that all the wheels of an individual AGV ahould be able to steer. Since there will be more than one steering angle encoder, the identi??cation of the odometric parameters is substantially more dif??cult for these omni-directional AGVs than for the mobile wheeled robots that are commonly seen in literature and laboratory settings. In this thesis we present a novel procedure for simultaneously estimating effective wheel diameters and steering angle encoder offsets by driving several pure circle segments. The validity of the tuning procedure is con??rmed by experiments with the two omni-directional test vehicles with varying loads. An interesting result is that the effective wheel diameters of the rubber wheels of our AGVs increase with increasing load. A crucial aspect in all control designs is the reconstruction of the to-be-controlled variables from measurement data. Our to-be-controlled variables are the planar motion of the load and the motions of the AGVs with respect to the load, which have to be reconstruct from the odometric sensor information. The odometric sensor information consists of the drive encoder and steering encoder readings. We analyzed the observability of an individual AGV and proved that it is theoretically possible to reconstruct its complete motion from the odometric measurements. Due to practical considerations, we pursued a more pragmatic least-squares based observer design. We show that the least-squares based motion estimate is independent of the coordinate system that is being used. The motion estimator was subsequently analyzed in a stochastic setting. The relation between the motion estimator and the estimated velocity of an arbitrary point on the vehicle was explored. We derived how the covariance of the velocity estimate of an arbitrary point on the vehicle is related to the covariance of the motion estimate. We proved that there is one unique point on the vehicle for which the covariance of the estimated velocity is minimal. Next, we investigated how the local motion estimates of the individual AGVs can be combined to yield one global estimate. When the load and AGVs are rigidly interconnected, it suf??ces that each AGVs broadcasts its local motion estimate and receives the estimates of the other AGVs. When the load is semi-rigidly interconnected to the AGVs, e.g. by means of revolute or prismatic joints, then generally each AGV needs to broadcasts the corresponding information matrix as well. We showed that the information matrix remains constant when the load is connected to the AGV with a revolute joint that is mounted at the aforementioned unique point with the smallest velocity estimate covariance. This means that the corresponding AGV does not have to broadcast its information matrix for this special situation. The key issue in the control design for cooperative transportation tasks is that the various AGVs must not counteract each others' actions. The decentralized controller that we derived makes the AGVs track an externally provided planar motion setpoint while minimizing the interconnection forces between the load and the vehicles. Although the control design is applicable to cooperative transportation by multiple AGVs with arbitrary semi-rigid AGV-load interconnections, it is noteworthy that a particularly elegant solution arises when all interconnections are completely rigid. Then the derived local controllers have the same structure as the controllers that are normally used for individual operation. As a result, changing a few parameter settings and providing the AGVs with identical setpoints is all that is required to achieve cooperative behavior on the velocity level for this situation. The observer and controller designs for the case that the AGVs are completely rigidly interconnected to the load were successfully implemented on the two test vehicles. Experi ments were carried out with and without a load that consisted of a pallet with 300 kg pave stones. The results were reproducible and illustrated the practical validity of the observer and controller designs. There were no substantial drawbacks when the local observers used only their local sensor information, which means that our setup can also operate satisfactory when the velocity estimates are not shared with the other vehicles

An obstacle detection system for automated guided vehicles

The objective of this master's thesis is to investigate the utilization of computer vision and object detection as an integral part of an automated guided vehicle's navigation system, which operates within the facilities of the target company. The rationale for conducting this research and developing an application for this purpose arises from the inability of automated guided vehicles to detect smaller or partially obstructed objects, and the lack of differentiation between stationary and moving objects. These limitations pose a safety hazard and negatively impact the overall performance of the system. The anticipated outcome of this thesis is a proof-of-concept computer vision application that would enhance the automated guided vehicle's obstacle detection capacity. The primary aim is to offer practical insights to the target company regarding the practical implementation of computer vision by developing and training a YOLOv7 object detection model, as a proposed resolution to the research problem. A thorough theoretical part of the required technologies and tools for training an object detection model is followed by a plan for the application to define requirements for the object detection model. The training and development are conducted using open-source and standard software tools and libraries. Python is the primary programming language employed throughout the development process and the object detector itself constitutes a YOLOv7 (You Only Look Once) object detection algorithm. The model is trained to identify and classify a predetermined number of objects or obstacles that impede the present automated guided vehicle system. Model optimization follows a fundamental trial-and-error methodology and simulated testing of the best-performing model. The data required for training the object detection model is obtained by attaching a camera to an automated guided vehicle and capturing its movements within the target company's facilities. The gathered data is annotated using Label studio, and all necessary data preparation and processing are carried out using plain Python. The result of this master’s thesis was a proof of concept for a computer vision application that would improve and benefit the target company’s day-to-day operations in their production and storage facilities in Vaasa. The trained model was substantiated to perform up to expectations in terms of both speed and accuracy. This project not only demonstrated the application's benefits but also laid grounds for the business to further utilize machine learning and computer vision in other areas of their business regarding the operational improvement competency of the target company’s employees. The results of this master’s thesis showed that finding an optimal object detection model for a specific dataset within a reasonable timeframe requires both appropriate tools and sufficient research data premises in terms of model configuration. The trained model could be utilized as a foundation for similar projects and thereby reduce the time and costs involved in preliminary research efforts

Robots learn to behave: improving human-robot collaboration in flexible manufacturing applications

L'abstract è presente nell'allegato / the abstract is in the attachmen

Platooning-based control techniques in transportation and logistic

This thesis explores the integration of autonomous vehicle technology with smart manufacturing systems. At first, essential control methods for autonomous vehicles, including Linear Matrix Inequalities (LMIs), Linear Quadratic Regulation (LQR)/Linear Quadratic Tracking (LQT), PID controllers, and dynamic control logic via flowcharts, are examined. These techniques are adapted for platooning to enhance coordination, safety, and efficiency within vehicle fleets, and various scenarios are analyzed to confirm their effectiveness in achieving predetermined performance goals such as inter-vehicle distance and fuel consumption. A first approach on simplified hardware, yet realistic to model the vehicle's behavior, is treated to further prove the theoretical results. Subsequently, performance improvement in smart manufacturing systems (SMS) is treated. The focus is placed on offline and online scheduling techniques exploiting Mixed Integer Linear Programming (MILP) to model the shop floor and Model Predictive Control (MPC) to adapt scheduling to unforeseen events, in order to understand how optimization algorithms and decision-making frameworks can transform resource allocation and production processes, ultimately improving manufacturing efficiency. In the final part of the work, platooning techniques are employed within SMS. Autonomous Guided Vehicles (AGVs) are reimagined as autonomous vehicles, grouping them within platoon formations according to different criteria, and controlled to avoid collisions while carrying out production orders. This strategic integration applies platooning principles to transform AGV logistics within the SMS. The impact of AGV platooning on key performance metrics, such as makespan, is devised, providing insights into optimizing manufacturing processes. Throughout this work, various research fields are examined, with intersecting future technologies from precise control in autonomous vehicles to the coordination of manufacturing resources. This thesis provides a comprehensive view of how optimization and automation can reshape efficiency and productivity not only in the domain of autonomous vehicles but also in manufacturing

3D Perception Based Lifelong Navigation of Service Robots in Dynamic Environments

Lifelong navigation of mobile robots is to ability to reliably operate over extended periods of time in dynamically changing environments. Historically, computational capacity and sensor capability have been the constraining factors to the richness of the internal representation of the environment that a mobile robot could use for navigation tasks. With affordable contemporary sensing technology available that provides rich 3D information of the environment and increased computational power, we can increasingly make use of more semantic environmental information in navigation related tasks.A navigation system has many subsystems that must operate in real time competing for computation resources in such as the perception, localization, and path planning systems. The main thesis proposed in this work is that we can utilize 3D information from the environment in our systems to increase navigational robustness without making trade-offs in any of the real time subsystems. To support these claims, this dissertation presents robust, real world 3D perception based navigation systems in the domains of indoor doorway detection and traversal, sidewalk-level outdoor navigation in urban environments, and global localization in large scale indoor warehouse environments.The discussion of these systems includes methods of 3D point cloud based object detection to find respective objects of semantic interest for the given navigation tasks as well as the use of 3D information in the navigational systems for purposes such as localization and dynamic obstacle avoidance. Experimental results for each of these applications demonstrate the effectiveness of the techniques for robust long term autonomous operation

Multi AGV Communication Failure Tolerant Industrial Supervisory System

Hoje em dia, em muitos ambientes industriais que utilizam vários robots, existe o problema de controlar o tráfego. Para se controlar o tráfego é preciso planear caminhos seguros, evitar os chamados deadlocks e estar imune a falhas de rede. O objetivo deste projeto consiste em implementar um sistema supervisor que controle esse tráfego, ou seja, seja capaz de detetar as falhas de rede, detetar desvios nas rotas dos robôs e replanear se necessário. O sistema de planeamento de trajetórias é o TEA*, um algoritmo A* mas que entra com a noção de tempo.The use of multi AGV implies an optimisation of traffic control. Several approaches focus on a trajectory planning method that guarantees an efficient and safe coordination of multi AGV. However, many fail to detect, treat and prevent the possible failure and delay in the communication between the AGV and the control platform. These faults can result in possible deadlock situations and collisions. In environments where communication faults are common, we might face a decrease of efficiency. Therefore, the aim of this project is to implement a supervisory system that controls the traffic of a fleet of AGV by being able to detect communication faults, delays in the communication, deviations in the routes of the robots and re-plan trajectories if necessary. For this purpose, the algorithm TEA*, an A* based algorithm (a graph search algorithm) with time notion, will be used to keep the efficiency and allow time optimisations