Chaotic exploration and learning of locomotion behaviours

We present a general and fully dynamic neural system, which exploits intrinsic chaotic dynamics, for the real-time goal-directed exploration and learning of the possible locomotion patterns of an articulated robot of an arbitrary morphology in an unknown environment. The controller is modeled as a network of neural oscillators that are initially coupled only through physical embodiment, and goal-directed exploration of coordinated motor patterns is achieved by chaotic search using adaptive bifurcation. The phase space of the indirectly coupled neural-body-environment system contains multiple transient or permanent self-organized dynamics, each of which is a candidate for a locomotion behavior. The adaptive bifurcation enables the system orbit to wander through various phase-coordinated states, using its intrinsic chaotic dynamics as a driving force, and stabilizes on to one of the states matching the given goal criteria. In order to improve the sustainability of useful transient patterns, sensory homeostasis has been introduced, which results in an increased diversity of motor outputs, thus achieving multiscale exploration. A rhythmic pattern discovered by this process is memorized and sustained by changing the wiring between initially disconnected oscillators using an adaptive synchronization method. Our results show that the novel neurorobotic system is able to create and learn multiple locomotion behaviors for a wide range of body configurations and physical environments and can readapt in realtime after sustaining damage

Intelligent Navigation for a Solar Powered Unmanned Underwater Vehicle

In this paper, an intelligent navigation system for an unmanned underwater vehicle powered by renewable energy and designed for shadow water inspection in missions of a long duration is proposed. The system is composed of an underwater vehicle, which tows a surface vehicle. The surface vehicle is a small boat with photovoltaic panels, a methanol fuel cell and communication equipment, which provides energy and communication to the underwater vehicle. The underwater vehicle has sensors to monitor the underwater environment such as sidescan sonar and a video camera in a flexible configuration and sensors to measure the physical and chemical parameters of water quality on predefined paths for long distances. The underwater vehicle implements a biologically inspired neural architecture for autonomous intelligent navigation. Navigation is carried out by integrating a kinematic adaptive neuro‐controller for trajectory tracking and an obstacle avoidance adaptive neuro‐  controller. The autonomous underwater vehicle is capable of operating during long periods of observation and monitoring. This autonomous vehicle is a good tool for observing large areas of sea, since it operates for long periods of time due to the contribution of renewable energy. It correlates all sensor data for time and geodetic position. This vehicle has been used for monitoring the Mar Menor lagoon.Supported by the Coastal Monitoring System for the Mar Menor (CMS‐  463.01.08_CLUSTER) project founded by the Regional Government of Murcia, by the SICUVA project (Control and Navigation System for AUV Oceanographic Monitoring Missions. REF: 15357/PI/10) founded by the Seneca Foundation of Regional Government of Murcia and by the DIVISAMOS project (Design of an Autonomous Underwater Vehicle for Inspections and oceanographic mission‐UPCT: DPI‐ 2009‐14744‐C03‐02) founded by the Spanish Ministry of Science and Innovation from Spain

Control of autonomous multibody vehicles using artificial intelligence

The field of autonomous driving has been evolving rapidly within the last few years and a lot of research has been dedicated towards the control of autonomous vehicles, especially car-like ones. Due to the recent successes of artificial intelligence techniques, even more complex problems can be solved, such as the control of autonomous multibody vehicles. Multibody vehicles can accomplish transportation tasks in a faster and cheaper way compared to multiple individual mobile vehicles or robots. But even for a human, driving a truck-trailer is a challenging task. This is because of the complex structure of the vehicle and the maneuvers that it has to perform, such as reverse parking to a loading dock. In addition, the detailed technical solution for an autonomous truck is challenging and even though many single-domain solutions are available, e.g. for pathplanning, no holistic framework exists. Also, from the control point of view, designing such a controller is a high complexity problem, which makes it a widely used benchmark. In this thesis, a concept for a plurality of tasks is presented. In contrast to most of the existing literature, a holistic approach is developed which combines many stand-alone systems to one entire framework. The framework consists of a plurality of modules, such as modeling, pathplanning, training for neural networks, controlling, jack-knife avoidance, direction switching, simulation, visualization and testing. There are model-based and model-free control approaches and the system comprises various pathplanning methods and target types. It also accounts for noisy sensors and the simulation of whole environments. To achieve superior performance, several modules had to be developed, redesigned and interlinked with each other. A pathplanning module with multiple available methods optimizes the desired position by also providing an efficient implementation for trajectory following. Classical approaches, such as optimal control (LQR) and model predictive control (MPC) can safely control a truck with a given model. Machine learning based approaches, such as deep reinforcement learning, are designed, implemented, trained and tested successfully. Furthermore, the switching of the driving direction is enabled by continuous analysis of a cost function to avoid collisions and improve driving behavior. This thesis introduces a working system of all integrated modules. The system proposed can complete complex scenarios, including situations with buildings and partial trajectories. In thousands of simulations, the system using the LQR controller or the reinforcement learning agent had a success rate of >95 % in steering a truck with one trailer, even with added noise. For the development of autonomous vehicles, the implementation of AI at scale is important. This is why a digital twin of the truck-trailer is used to simulate the full system at a much higher speed than one can collect data in real life.Tesi

Adaptive dynamic programming with eligibility traces and complexity reduction of high-dimensional systems

This dissertation investigates the application of a variety of computational intelligence techniques, particularly clustering and adaptive dynamic programming (ADP) designs especially heuristic dynamic programming (HDP) and dual heuristic programming (DHP). Moreover, a one-step temporal-difference (TD(0)) and n-step TD (TD(λ)) with their gradients are utilized as learning algorithms to train and online-adapt the families of ADP. The dissertation is organized into seven papers. The first paper demonstrates the robustness of model order reduction (MOR) for simulating complex dynamical systems. Agglomerative hierarchical clustering based on performance evaluation is introduced for MOR. This method computes the reduced order denominator of the transfer function by clustering system poles in a hierarchical dendrogram. Several numerical examples of reducing techniques are taken from the literature to compare with our work. In the second paper, a HDP is combined with the Dyna algorithm for path planning. The third paper uses DHP with an eligibility trace parameter (λ) to track a reference trajectory under uncertainties for a nonholonomic mobile robot by using a first-order Sugeno fuzzy neural network structure for the critic and actor networks. In the fourth and fifth papers, a stability analysis for a model-free action-dependent HDP(λ) is demonstrated with batch- and online-implementation learning, respectively. The sixth work combines two different gradient prediction levels of critic networks. In this work, we provide a convergence proofs. The seventh paper develops a two-hybrid recurrent fuzzy neural network structures for both critic and actor networks. They use a novel n-step gradient temporal-difference (gradient of TD(λ)) of an advanced ADP algorithm called value-gradient learning (VGL(λ)), and convergence proofs are given. Furthermore, the seventh paper is the first to combine the single network adaptive critic with VGL(λ). --Abstract, page iv

A Systematic Literature Review of Path-Planning Strategies for Robot Navigation in Unknown Environment

The Many industries, including ports, space, surveillance, military, medicine and agriculture have benefited greatly from mobile robot technology.  An autonomous mobile robot navigates in situations that are both static and dynamic. As a result, robotics experts have proposed a range of strategies. Perception, localization, path planning, and motion control are all required for mobile robot navigation. However, Path planning is a critical component of a quick and secure navigation. Over the previous few decades, many path-planning algorithms have been developed. Despite the fact that the majority of mobile robot applications take place in static environments, there is a scarcity of algorithms capable of guiding robots in dynamic contexts. This review compares qualitatively mobile robot path-planning systems capable of navigating robots in static and dynamic situations. Artificial potential fields, fuzzy logic, genetic algorithms, neural networks, particle swarm optimization, artificial bee colonies, bacterial foraging optimization, and ant-colony are all discussed in the paper. Each method's application domain, navigation technique and validation context are discussed and commonly utilized cutting-edge methods are analyzed. This research will help researchers choose appropriate path-planning approaches for various applications including robotic cranes at the sea ports as well as discover gaps for optimization

Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

A brief review of neural networks based learning and control and their applications for robots

As an imitation of the biological nervous systems, neural networks (NN), which are characterized with powerful learning ability, have been employed in a wide range of applications, such as control of complex nonlinear systems, optimization, system identification and patterns recognition etc. This article aims to bring a brief review of the state-of-art NN for the complex nonlinear systems. Recent progresses of NNs in both theoretical developments and practical applications are investigated and surveyed. Specifically, NN based robot learning and control applications were further reviewed, including NN based robot manipulator control, NN based human robot interaction and NN based behavior recognition and generation

Locomotion training of legged robots using hybrid machine learning techniques

In this study artificial neural networks and fuzzy logic are used to control the jumping behavior of a three-link uniped robot. The biped locomotion control problem is an increment of the uniped locomotion control. Study of legged locomotion dynamics indicates that a hierarchical controller is required to control the behavior of a legged robot. A structured control strategy is suggested which includes navigator, motion planner, biped coordinator and uniped controllers. A three-link uniped robot simulation is developed to be used as the plant. Neurocontrollers were trained both online and offline. In the case of on-line training, a reinforcement learning technique was used to train the neurocontroller to make the robot jump to a specified height. After several hundred iterations of training, the plant output achieved an accuracy of 7.4%. However, when jump distance and body angular momentum were also included in the control objectives, training time became impractically long. In the case of off-line training, a three-layered backpropagation (BP) network was first used with three inputs, three outputs and 15 to 40 hidden nodes. Pre-generated data were presented to the network with a learning rate as low as 0.003 in order to reach convergence. The low learning rate required for convergence resulted in a very slow training process which took weeks to learn 460 examples. After training, performance of the neurocontroller was rather poor. Consequently, the BP network was replaced by a Cerebeller Model Articulation Controller (CMAC) network. Subsequent experiments described in this document show that the CMAC network is more suitable to the solution of uniped locomotion control problems in terms of both learning efficiency and performance. A new approach is introduced in this report, viz., a self-organizing multiagent cerebeller model for fuzzy-neural control of uniped locomotion is suggested to improve training efficiency. This is currently being evaluated for a possible patent by NASA, Johnson Space Center. An alternative modular approach is also developed which uses separate controllers for each stage of the running stride. A self-organizing fuzzy-neural controller controls the height, distance and angular momentum of the stride. A CMAC-based controller controls the movement of the leg from the time the foot leaves the ground to the time of landing. Because the leg joints are controlled at each time step during flight, movement is smooth and obstacles can be avoided. Initial results indicate that this approach can yield fast, accurate results

Navigational Path Analysis of Mobile Robot in Various Environments

This dissertation describes work in the area of an autonomous mobile robot. The objective is navigation of mobile robot in a real world dynamic environment avoiding structured and unstructured obstacles either they are static or dynamic. The shapes and position of obstacles are not known to robot prior to navigation. The mobile robot has sensory recognition of specific objects in the environments. This sensory-information provides local information of robots immediate surroundings to its controllers. The information is dealt intelligently by the robot to reach the global objective (the target). Navigational paths as well as time taken during navigation by the mobile robot can be expressed as an optimisation problem and thus can be analyzed and solved using AI techniques. The optimisation of path as well as time taken is based on the kinematic stability and the intelligence of the robot controller. A successful way of structuring the navigation task deals with the issues of individual behaviour design and action coordination of the behaviours. The navigation objective is addressed using fuzzy logic, neural network, adaptive neuro-fuzzy inference system and different other AI technique.The research also addresses distributed autonomous systems using multiple robot