Extended QDSEGA for Controlling Real Robot : Acquisition of Locomotion Patterns for Snake : like Robot

Reinforcement learning is very effective for robot learning. It is because it does not need prior knowledge and has higher capability of reactive and adaptive behaviors. In our previous works, we proposed new reinforce learning algorithm: &quot;Q-learning with dynamic structuring of exploration space based on genetic algorithm (QDSEGA)&quot;. It is designed for complicated systems with large action-state space like a robot with many redundant degrees of freedom. However the application of QDSEGA is restricted to static systems. A snake-like robot has many redundant degrees of freedom and the dynamics of the system are very important to complete the locomotion task. So application of usual reinforcement learning is very difficult. In this paper, we extend layered structure of QDSEGA so that it becomes possible to apply it to real robots that have complexities and dynamics. We apply it to acquisition of locomotion pattern of the snake-like robot and demonstrate the effectiveness and the validity of QDSEGA with the extended layered structure by simulation and experiment. </p

Emergence of adaptive behaviors by redundant robots : Robustness to changes environment and failures

Acquiring adaptive behaviors of robots automatically is one of the most interesting topics of the evolutionary systems. In previous works, we have developed an adaptive autonomous control method for redundant robots. The QDSEGA is one of the methods that we have proposed for them. The QDSEGA is realized by combining Q-learning and GA, and it can acquire suitable behaviors by adapting a movement of a robot for a task. In this paper, we focus on the adaptability of the QDSEGA and discuss the robustness of the autonomous redundant robot that is controlled by the QDSEGA. To demonstrate the effectiveness of the QDSEGA, simulations of obstacle avoidance by a 10-link manipulator in the changeable environment and locomotion by a 12-legged robot with failures have been carried out, and as a result, adaptive behaviors for each environment and each broken body have emerged. </p

Control of hyper-redundant robot using QDSEGA

We consider a flexible autonomous system. To realize the system, we employ a hyper-redundant system (a flexible hardware system) and reinforcement learning controller &#34;QDSEGA&#34; (Q-learning with structuring exploration space based on genetic algorithm) which is a flexible software system. In this paper we apply QDSEGA for controlling of the hyper-redundant robot. To demonstrate the effectiveness, a task of acquisition of locomotion patterns is applied to a multi-legged formation and a snake-like formation, from which an effective locomotion is obtained.</p

From Rolling Over to Walking: Enabling Humanoid Robots to Develop Complex Motor Skills

This paper presents an innovative method for humanoid robots to acquire a comprehensive set of motor skills through reinforcement learning. The approach utilizes an achievement-triggered multi-path reward function rooted in developmental robotics principles, facilitating the robot to learn gross motor skills typically mastered by human infants within a single training phase. The proposed method outperforms standard reinforcement learning techniques in success rates and learning speed within a simulation environment. By leveraging the principles of self-discovery and exploration integral to infant learning, this method holds the potential to significantly advance humanoid robot motor skill acquisition.Comment: 8 pages, 9 figures. Submitted to IEEE Robotics and Automation Letters. Video available at https://youtu.be/d0RqrW1Ezj

A study of reinforcement learning with knowledge sharing

In this paper, we consider multi-agent system in which every agents have own tasks that differs each other. We propose a method that decreases learning time of reinforcement learning by using the model of environment. In the proposed algorithm, the model is created by sharing the experiences of agents each other. To demonstrate the effectiveness of the proposed method, simulations of a puddle world and experiments of a maze world have been carried out. As a result effective behaviors have been obtained quickly.</p

MOTION CONTROL SIMULATION OF A HEXAPOD ROBOT

This thesis addresses hexapod robot motion control. Insect morphology and locomotion patterns inform the design of a robotic model, and motion control is achieved via trajectory planning and bio-inspired principles. Additionally, deep learning and multi-agent reinforcement learning are employed to train the robot motion control strategy with leg coordination achieves using a multi-agent deep reinforcement learning framework. The thesis makes the following contributions: First, research on legged robots is synthesized, with a focus on hexapod robot motion control. Insect anatomy analysis informs the hexagonal robot body and three-joint single robotic leg design, which is assembled using SolidWorks. Different gaits are studied and compared, and robot leg kinematics are derived and experimentally verified, culminating in a three-legged gait for motion control. Second, an animal-inspired approach employs a central pattern generator (CPG) control unit based on the Hopf oscillator, facilitating robot motion control in complex environments such as stable walking and climbing. The robot\u27s motion process is quantitatively evaluated in terms of displacement change and body pitch angle. Third, a value function decomposition algorithm, QPLEX, is applied to hexapod robot motion control. The QPLEX architecture treats each leg as a separate agent with local control modules, that are trained using reinforcement learning. QPLEX outperforms decentralized approaches, achieving coordinated rhythmic gaits and increased robustness on uneven terrain. The significant of terrain curriculum learning is assessed, with QPLEX demonstrating superior stability and faster consequence. The foot-end trajectory planning method enables robot motion control through inverse kinematic solutions but has limited generalization capabilities for diverse terrains. The animal-inspired CPG-based method offers a versatile control strategy but is constrained to core aspects. In contrast, the multi-agent deep reinforcement learning-based approach affords adaptable motion strategy adjustments, rendering it a superior control policy. These methods can be combined to develop a customized robot motion control policy for specific scenarios

A study of reinforcement learning with knowledge sharing for distributed autonomous system

Reinforcement learning is one of effective controller for autonomous robots. Because it does not need priori knowledge and behaviors to complete given tasks are obtained automatically be repeating trial and error. However a large number of trials are required to realize complex tasks. So the task that can be obtained using the real robot is restricted to simple ones. Considering these points, various methods that prove the learning cost of reinforcement learning have been proposed. In the method that uses priori knowledge, the methods lose the autonomy that is most important feature of reinforcement learning in applying it to the robots. In the Dyna-Q, that is one of simple and effective reinforcement learning architecture integrating online planning, a model of environment is learned from real experience and by utilizing the model to learn, the learning time is decreased. In this architecture, the autonomy is held, however the model depends on the task, so acquired knowledge of environment cannot be reused to other tasks. In the real world, human beings can learn various behaviors to complete complex tasks without priori knowledge of the tasks. We can try to realize the task in our image without moving our body. After the training in the image, by trying to the real environment, we save time to learn. It means that we have model of environment and we utilize the model to learn. We consider that the key ability that makes the learning process faster is construction of environment model and utilization of it. In this paper, we have proposed a method to obtain an environment model that is independent of the task. And by utilizing the model we have decreased learning time. We consider distributed autonomous agents, and we show that the environment model is constructed quickly by sharing the experience of each agent, even when each agent has own independent task. To demonstrate the effectiveness of the proposed method, we have applied the method to the Q-learning and simulations of a puddle world are carried out. As a result effective behaviors have been obtained quickly. </p