Cooperation of Multiple Fish-like Microrobots Based on Reinforcement Learning

Abstract-This paper is concerned with cooperative control of a kind of multiple fish-like microrobots. Most of previous work on multi-robot cooperation is focused on the terrestrial robots and seldom deals with underwater applications. In fact, the tasks in hydro-environment is more challenging than those in ground circumstances and need the cooperation of robots much more. In this paper, we investigate this problem in the framework of an adversarial game with several underwater microrobots. A fuzzy reinforcement learning approach is adopted to acquire cooperative behavior and a behavioral hierarchical architecture is proposed. We conduct extensive experiments to verify the effectiveness of the proposed algorithms

Design, Actuation, and Functionalization of Untethered Soft Magnetic Robots with Life-Like Motions: A Review

Soft robots have demonstrated superior flexibility and functionality than conventional rigid robots. These versatile devices can respond to a wide range of external stimuli (including light, magnetic field, heat, electric field, etc.), and can perform sophisticated tasks. Notably, soft magnetic robots exhibit unparalleled advantages among numerous soft robots (such as untethered control, rapid response, and high safety), and have made remarkable progress in small-scale manipulation tasks and biomedical applications. Despite the promising potential, soft magnetic robots are still in their infancy and require significant advancements in terms of fabrication, design principles, and functional development to be viable for real-world applications. Recent progress shows that bionics can serve as an effective tool for developing soft robots. In light of this, the review is presented with two main goals: (i) exploring how innovative bioinspired strategies can revolutionize the design and actuation of soft magnetic robots to realize various life-like motions; (ii) examining how these bionic systems could benefit practical applications in small-scale solid/liquid manipulation and therapeutic/diagnostic-related biomedical fields

Challenges and attempts to make intelligent microswimmers

The study of microswimmers’ behavior, including their self-propulsion, interactions with the environment, and collective phenomena, has received significant attention over the past few decades due to its importance for various biological and medical applications. Microswimmers can easily access micro-fluidic channels and manipulate microscopic entities, enabling them to perform sophisticated tasks as untethered mobile microrobots inside the human body or microsize devices. Thanks to the advancements in micro/nano-technologies, a variety of synthetic and biohybrid microrobots have been designed and fabricated. Nevertheless, a key challenge arises: how to guide the microrobots to navigate through complex fluid environments and perform specific tasks. The model-free reinforcement learning (RL) technique appears to be a promising approach to address this problem. In this review article, we will first illustrate the complexities that microswimmers may face in realistic biological fluid environments. Subsequently, we will present recent experimental advancements in fabricating intelligent microswimmers using physical intelligence and biohybrid techniques. We then introduce several popular RL algorithms and summarize the recent progress for RL-powered microswimmers. Finally, the limitations and perspectives of the current studies in this field will be discussed

Locomotion Optimization of Photoresponsive Small-scale Robot: A Deep Reinforcement Learning Approach

Soft robots comprise of elastic and flexible structures, and actuatable soft materials are often used to provide stimuli-responses, remotely controlled with different kinds of external stimuli, which is beneficial for designing small-scale devices. Among different stimuli-responsive materials, liquid crystal networks (LCNs) have gained a significant amount of attention for soft small-scale robots in the past decade being stimulated and actuated by light, which is clean energy, able to transduce energy remotely, easily available and accessible to sophisticated control. One of the persistent challenges in photoresponsive robotics is to produce controllable autonomous locomotion behavior. In this Thesis, different types of photoresponsive soft robots were used to realize light-powered locomotion, and an artificial intelligence-based approach was developed for controlling the movement. A robot tracking system, including an automatic laser steering function, was built for efficient robotic feature detection and steering the laser beam automatically to desired locations. Another robot prototype, a swimmer robot, driven by the automatically steered laser beam, showed directional movements including some degree of uncertainty and randomness in their locomotion behavior. A novel approach is developed to deal with the challenges related to the locomotion of photoresponsive swimmer robots. Machine learning, particularly deep reinforcement learning method, was applied to develop a control policy for autonomous locomotion behavior. This method can learn from its experiences by interacting with the robot and its environment without explicit knowledge of the robot structure, constituent material, and robotic mechanics. Due to the requirement of a large number of experiences to correlate the goodness of behavior control, a simulator was developed, which mimicked the uncertain and random movement behavior of the swimmer robots. This approach effectively adapted the random movement behaviors and developed an optimal control policy to reach different destination points autonomously within a simulated environment. This work has successfully taken a step towards the autonomous locomotion control of soft photoresponsive robots

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems like the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges, and give perspectives on future research directions, looking to inspire interdisciplinary approaches.Comment: 46 pages, 9 figure

Study of artificial intelligence and computer vision methods for tracking transmission lines with the AID of UAVs

Currently, Unmanned Aerial Vehicles (UAVs) have been used in the most diverse applications in both the civil and military sectors. In the civil sector, aerial inspection services have been gaining a lot of attention, especially in the case of inspections of high voltage electrical systems transmission lines. This type of inspection involves a helicopter carrying three or more people (technicians, pilot, etc.) flying over the transmission line along its entire length which is a dangerous service especially due to the proximity of the transmission line and possible environmental conditions (wind gusts, for example). In this context, the use of UAVs has shown considerable interest due to their low cost and safety for transmission line inspection technicians. This work presents research results related to the application of UAVs for transmission lines inspection, autonomously, allowing the identification of invasions of the transmission line area as well as possible defects in components (cables, insulators, connection, etc.) through the use of Convolutional Neural Networks (CNN) for fault detection and identification. This thesis proposes the development of an autonomous system to track power transmission lines using UAVs efficiently and with low implementation and operation costs, based exclusively on rea-time image processing that identifies the structure of the towers and transmission lines durin the flight and controls the aircraft´s movements, guiding it along the closest possible path. A sumary of the work developed will be presented in the next sections.Atualmente, os Veículos Aéreos Não Tripulados – VANTs têm sido utilizados nas mais diversas aplicações tanto no setor civil quanto militar. No setor civil, os serviços de inspeção aérea vêm ganhando bastante atenção, principalmente no caso de inspeções de linhas de transmissão de sistemas elétricos de alta tensão. Este tipo de inspeção envolve um helicóptero transportando três ou mais pessoas (técnicos, pilotos, etc.) sobrevoando a linha de transmissão em toda a sua extensão, o que constitui um serviço perigoso principalmente pela proximidade da linha de transmissão e possíveis condições ambientais (rajadas de vento, por exemplo). Neste contexto, a utilização de VANTs tem demonstrado considerável interesse devido ao seu baixo custo e segurança para técnicos de inspeção de linhas de transmissão. Este trabalho apresenta resultados de pesquisas relacionadas à aplicação de VANTs para inspeção de linhas de transmissão, de forma autônoma, permitindo a identificação de invasões da área da linha de transmissão bem como possíveis defeitos em componentes (cabos, isoladores, conexões, etc.) através do uso de Convolucional. Redes Neurais - CNN para detecção e identificação de falhas. Esta tese propõe o desenvolvimento de um sistema autônomo para rastreamento de linhas de transmissão de energia utilizando VANTs de forma eficiente e com baixos custos de implantação e operação, baseado exclusivamente no processamento de imagens em tempo real que identifica a estrutura das torres e linhas de transmissão durante o voo e controla a velocidade da aeronave. movimentos, guiando-o pelo caminho mais próximo possível. Um resumo do trabalho desenvolvido será apresentado nas próximas seções

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches

Behaviour design in microrobots:hierarchical reinforcement learning under resource constraints

In order to verify models of collective behaviors of animals, robots could be manipulated to implement the model and interact with real animals in a mixed-society. This thesis describes design of the behavioral hierarchy of a miniature robot, that is able to interact with cockroaches, and participates in their collective decision makings. The robots are controlled via a hierarchical behavior-based controller in which, more complex behaviors are built by combining simpler behaviors through fusion and arbitration mechanisms. The experiments in the mixed-society confirms the similarity between the collective patterns of the mixed-society and those of the real society. Moreover, the robots are able to induce new collective patterns by modulation of some behavioral parameters. Difficulties in the manual extraction of the behavioral hierarchy and inability to revise it, direct us to benefit from machine learning techniques, in order to devise the composition hierarchy and coordination in an automated way. We derive a Compact Q-Learning method for micro-robots with processing and memory constraints, and try to learn behavior coordination through it. The behavior composition part is still done manually. However, the problem of the curse of dimensionality makes incorporation of this kind of flat-learning techniques unsuitable. Even though optimizing them could temporarily speed up the learning process and widen their range of applications, their scalability to real world applications remains under question. In the next steps, we apply hierarchical learning techniques to automate both behavior coordination and composition parts. In some situations, many features of the state space might be irrelevant to what the robot currently learns. Abstracting these features and discovering the hierarchy among them can help the robot learn the behavioral hierarchy faster. We formalize the automatic state abstraction problem with different heuristics, and derive three new splitting criteria that adapt decision tree learning techniques to state abstraction. Proof of performance is supported by strong evidences from simulation results in deterministic and non-deterministic environments. Simulation results show encouraging enhancements in the required number of learning trials, robot's performance, size of the learned abstraction trees, and computation time of the algorithms. In the other hand, learning in a group provides free sources of knowledge that, if communicated, can broaden the scales of learning, both temporally and spatially. We present two approaches to combine output or structure of abstraction trees. The trees are stored in different RL robots in a multi-robot system, or in the trees learned by the same robot but using different methods. Simulation results in a non-deterministic football learning task provide strong evidences for enhancement in convergence rate and policy performance, specially in heterogeneous cooperations