Edge Robotics: are we ready? An experimental evaluation of current vision and future directions

Cloud-based robotics systems leverage a wide range of Information Technologies (IT) to offer tangible benefits like cost reduction, powerful computational capabilities, data offloading, etc. However, the centralized nature of cloud computing is not well-suited for a multitude of Operational Technologies (OT) nowadays used in robotics systems that require strict real-time guarantees and security. Edge computing and fog computing are complementary approaches that aim at mitigating some of these challenges by providing computing capabilities closer to the users. The goal of this work is hence threefold: i) to analyze the current edge computing and fog computing landscape in the context of robotics systems, ii) to experimentally evaluate an end-to-end robotics system based on solutions proposed in the literature, and iii) to experimentally identify current benefits and open challenges of edge computing and fog computing. Results show that, in the case of an exemplary delivery application comprising two mobile robots, the robot coordination and range can be improved by consuming real-time radio information available at the edge. However, our evaluation highlights that the existing software, wireless and virtualization technologies still require substantial evolution to fully support edge-based robotics systems.This work has been partially funded by European Union’s Horizon 2020 research and innovation programme under grant agreement No 101015956, and the Spanish Ministry of Economic Affairs and Digital Transformation and the European Union- NextGenerationEU through the UNICO 5G I+ D 6G-EDGEDT and 6G-DATADRIVE

Computer numerical control machine tool information reusability within virtual machining systems

Virtual Machining (VM) allows simulation of the machining process by realistically representing kinematic, static and dynamic behaviour of the intended machine tools. Using this method, manufacturing related issues can be brought to light and corrected before the product is physically manufactured. Machining systems utilised in the manufacturing processes are represented in the VM environment and there is a plethora of commercial VM software used in the industry. Each software system has a different focus and approach towards virtual machining; more than one system may be needed to complete machining verification. Thus, the significant increase in the use of virtual machining systems in industry has increased the need for information reusability. Substantial time and money has been put into the research of virtual machining systems. However, very little of this research has been deployed within industrial best practice and its acceptance by the end user remains unclear. This paper reviews current research trends in the domain of VM and also discusses how much of this research has been taken on board by software venders in order to facilitate machine tool information reusability. The authors present a use cases which utilises the novel concept of Machining Capability Profile (MCP) and the emerging STEP-NC compliant process planning framework for resource allocation. The use cases clearly demonstrate the benefits of using a neutral file format for representing MCPs, as opposed to remodelling and or reconfiguring of this information multiple times for different scenarios. The paper has shown through the use cases that MCPs are critical for representing recourse information from a kinematic, static and dynamic perspective that commercial software vendors can subsequently use. The impact of this on mainstream manufacturing industry is potentially significant as it will enable a true realisation of interoperability

Towards edge robotics: the progress from cloud-based robotic systems to intelligent and context-aware robotic services

Current robotic systems handle a different range of applications such as video surveillance, delivery of goods, cleaning, material handling, assembly, painting, or pick and place services. These systems have been embraced not only by the general population but also by the vertical industries to help them in performing daily activities. Traditionally, the robotic systems have been deployed in standalone robots that were exclusively dedicated to performing a specific task such as cleaning the floor in indoor environments. In recent years, cloud providers started to offer their infrastructures to robotic systems for offloading some of the robot’s functions. This ultimate form of the distributed robotic system was first introduced 10 years ago as cloud robotics and nowadays a lot of robotic solutions are appearing in this form. As a result, standalone robots became software-enhanced objects with increased reconfigurability as well as decreased complexity and cost. Moreover, by offloading the heavy processing from the robot to the cloud, it is easier to share services and information from various robots or agents to achieve better cooperation and coordination. Cloud robotics is suitable for human-scale responsive and delay-tolerant robotic functionalities (e.g., monitoring, predictive maintenance). However, there is a whole set of real-time robotic applications (e.g., remote control, motion planning, autonomous navigation) that can not be executed with cloud robotics solutions, mainly because cloud facilities traditionally reside far away from the robots. While the cloud providers can ensure certain performance in their infrastructure, very little can be ensured in the network between the robots and the cloud, especially in the last hop where wireless radio access networks are involved. Over the last years advances in edge computing, fog computing, 5G NR, network slicing, Network Function Virtualization (NFV), and network orchestration are stimulating the interest of the industrial sector to satisfy the stringent and real-time requirements of their applications. Robotic systems are a key piece in the industrial digital transformation and their benefits are very well studied in the literature. However, designing and implementing a robotic system that integrates all the emerging technologies and meets the connectivity requirements (e.g., latency, reliability) is an ambitious task. This thesis studies the integration of modern Information andCommunication Technologies (ICTs) in robotic systems and proposes some robotic enhancements that tackle the real-time constraints of robotic services. To evaluate the performance of the proposed enhancements, this thesis departs from the design and prototype implementation of an edge native robotic system that embodies the concepts of edge computing, fog computing, orchestration, and virtualization. The proposed edge robotics system serves to represent two exemplary robotic applications. In particular, autonomous navigation of mobile robots and remote-control of robot manipulator where the end-to-end robotic system is distributed between the robots and the edge server. The open-source prototype implementation of the designed edge native robotic system resulted in the creation of two real-world testbeds that are used in this thesis as a baseline scenario for the evaluation of new innovative solutions in robotic systems. After detailing the design and prototype implementation of the end-to-end edge native robotic system, this thesis proposes several enhancements that can be offered to robotic systems by adapting the concept of edge computing via the Multi-Access Edge Computing (MEC) framework. First, it proposes exemplary network context-aware enhancements in which the real-time information about robot connectivity and location can be used to dynamically adapt the end-to-end system behavior to the actual status of the communication (e.g., radio channel). Three different exemplary context-aware enhancements are proposed that aim to optimize the end-to-end edge native robotic system. Later, the thesis studies the capability of the edge native robotic system to offer potential savings by means of computation offloading for robot manipulators in different deployment configurations. Further, the impact of different wireless channels (e.g., 5G, 4G andWi-Fi) to support the data exchange between a robot manipulator and its remote controller are assessed. In the following part of the thesis, the focus is set on how orchestration solutions can support mobile robot systems to make high quality decisions. The application of OKpi as an orchestration algorithm and DLT-based federation are studied to meet the KPIs that autonomously controlledmobile robots have in order to provide uninterrupted connectivity over the radio access network. The elaborated solutions present high compatibility with the designed edge robotics system where the robot driving range is extended without any interruption of the end-to-end edge robotics service. While the DLT-based federation extends the robot driving range by deploying access point extension on top of external domain infrastructure, OKpi selects the most suitable access point and computing resource in the cloud-to-thing continuum in order to fulfill the latency requirements of autonomously controlled mobile robots. To conclude the thesis the focus is set on how robotic systems can improve their performance by leveraging Artificial Intelligence (AI) and Machine Learning (ML) algorithms to generate smart decisions. To do so, the edge native robotic system is presented as a true embodiment of a Cyber-Physical System (CPS) in Industry 4.0, showing the mission of AI in such concept. It presents the key enabling technologies of the edge robotic system such as edge, fog, and 5G, where the physical processes are integrated with computing and network domains. The role of AI in each technology domain is identified by analyzing a set of AI agents at the application and infrastructure level. In the last part of the thesis, the movement prediction is selected to study the feasibility of applying a forecast-based recovery mechanism for real-time remote control of robotic manipulators (FoReCo) that uses ML to infer lost commands caused by interference in the wireless channel. The obtained results are showcasing the its potential in simulation and real-world experimentation.Programa de Doctorado en Ingeniería Telemática por la Universidad Carlos III de MadridPresidente: Karl Holger.- Secretario: Joerg Widmer.- Vocal: Claudio Cicconett

Constrained Motion Planning System for MRI-Guided, Needle-Based, Robotic Interventions

In needle-based surgical interventions, accurate alignment and insertion of the tool is paramount for providing proper treatment at a target site while minimizing healthy tissue damage. While manually-aligned interventions are well-established, robotics platforms promise to reduce procedure time, increase precision, and improve patient comfort and survival rates. Conducting interventions in an MRI scanner can provide real-time, closed-loop feedback for a robotics platform, improving its accuracy, yet the tight environment potentially impairs motion, and perceiving this limitation when planning a procedure can be challenging. This project developed a surgical workflow and software system for evaluating the workspace and planning the motions of a robotics platform within the confines of an MRI scanner. 3D Slicer, a medical imaging visualization and processing platform, provided a familiar and intuitive interface for operators to quickly plan procedures with the robotics platform over OpenIGTLink. Robotics tools such as ROS and MoveIt! were utilized to analyze the workspace of the robot within the patient and formulate the motion planning solution for positioning of the robot during surgical procedures. For this study, a 7 DOF robot arm designed for ultrasonic ablation of brain tumors was the targeted platform. The realized system successfully yielded prototype capabilities on the neurobot for conducting workspace analysis and motion planning, integrated systems using OpenIGTLink, provided an opportunity to evaluate current software packages, and informed future work towards production-grade medical software for MRI-guided, needle-based robotic interventions

Automated 3D Printer Farm

Automation can be applied to many different disciplines including 3D printing. This project applies automation to a 3D printer farm with the goal of removing the need for human interactions. The robot incorporated into the system, the Fanuc 200iB, is used for removing the plates from the printer, placing completed prints onto a cooling rack and placing new plates onto the printer. A management system consisting of an Arduino, a Raspberry Pi and a Repetier server is used for coordinating the movements of system elements

Real-Time LiDAR-based Power Lines Detection for Unmanned Aerial Vehicles

The growing dependence of modern-day societies on electricity leads to the increasing importance of effective monitoring and maintenance of power lines. Due to the population’s renouncement to the installation of new electric power lines, the existing ones are constantly operating at maximum capacity. This leaves no room for breakdowns, as it leads to major economic losses for the electrical companies and blackouts for the consumers. Endowing Unmanned Aerial Vehicles (UAVs) with the appropriate sensors for inspection the power lines, the costs and risks associated with the traditional foot patrol and helicopter-based inspections can be reduced. However, this implies the development of algorithms to make the inspection process reliable and autonomous. Visual detection methods are usually applied to locate the power lines and their components. Although, they are generally too sensitive to atmospheric conditions and noisy background. Poor light conditions or a background rich in edges may compromise their results. In order to overcome those limitations, this dissertation addresses the problem of power line detection and modeling based on the use of a Light Detection And Ranging (LiDAR) sensor. A novel approach to the power line detection was developed, the Power Line LiDARbased Detection and Modeling (PL2DM). It is based in a scan-by-scan adaptive neighbor minimalist comparison for all the points in a point cloud. In the segmentation, the breaking cluster points are detected by an analysis of their planar properties. Exporting the potential power line points to a further step, it performs a scan based straight line detection. The final model of the power line is obtained by matching and grouping the several line segments detected using their collinearity properties. Horizontally, the power lines are modeled as a straight line, while vertically are approximated to a catenary curve. The algorithm was tested with a real dataset, showing promising results both in terms of outputs and processing time. From there, it was demonstrated that the proposed algorithm can be applied to real-time operations of the UAV, adding object-based perception capabilities for other layers of processing.A crescente dependência das sociedades modernas no uso de eletricidade conduz a uma crescente importância da eficiência da monitorização e manutenção das linhas elétricas. A renitência das populações `a instalação de novas linhas elétricas faz com que as existentes estejam constantemente a operar na sua máxima capacidade. Isto faz com que não possam existir falhas, uma vez que resultariam em grandes perdas económicas para as companhias elétricas e em falhas energéticas para os consumidores. Equipando um Unmanned Aerial Vehicle (UAV) com os sensores adequados `a inspeção de linhas elétricas, podem ser reduzidos os custos e riscos de operação associados `as inspeções tradicionais, baseadas em patrulhas pedonais e no uso de um helicóptero. No entanto, isto implica o desenvolvimento de algoritmos para que o processo de inspeção seja fiável e autónomo. As linhas elétricas e os componentes associados são geralmente localizados através de métodos de deteção visual. Estes m´métodos são, geralmente, muito sensíveis `as condições atmosféricas e a fundos ruidosos. Condições de luz deficientes ou fundos ricos em contrastes são alguns dos fatores que podem comprometer os seus resultados. De forma a ultrapassar essas limitações, esta dissertação endereça o problema da deteção e modelação de linhas elétricas, tendo por base o uso de um sensor Light Detection And Ranging (LiDAR). Foi desenvolvida uma nova abordagem aos métodos de deteção de linhas elétricas, o Power Line LiDAR-based Detection and Modeling (PL2DM). Esta abordagem ´e baseada numa análise individual de varrimentos, em que ´e feita uma comparação minimalista de todos os pontos, presentes numa dada nuvem de pontos, com uma vizinhança adaptativa. Na segmentação, os pontos de quebra dos grupos criados são detetados tendo em conta as suas propriedades planares. Passando os pontos passíveis de pertencerem a linhas elétricas para o processamento seguinte, é realizada, em cada varrimento, uma deteção de linhas retas. O modelo final das linhas elétricas é obtido a partir da associação e agrupamento dos diversos segmentos de reta detetados, tendo por base a sua colinearidade. Na sua projeção horizontal, as linhas elétricas são modeladas como linhas retas. Verticalmente, são aproximadas ao modelo de uma curva catenária. O algoritmo foi testado com um conjunto de dados reais, tendo mostrado resultados promissores, tanto em termos de dados gerados como de tempo de processamento. Com isso, ficou demonstrado que o algoritmo proposto pode ser aplicado nas operações do UAV em tempo real, adicionando capacidades de perceção baseada em objetos para outras camadas de processamento

Software Infrastructure and Course Design of a Robotics Course

Käesolev töö käsitleb robootika kursuse disaini ja arenduse protsessi. Käsitletav kursus on Tartu Ülikoolis üheks võimalikuks lähtepunktiks robootika valdkonda sisenemiseks. Disaini protsessis on kasutatud elemente tegevusuuringu ülesehitusest ning DevOps nime kandvast süsteemiarenduse kultuurist. Mõlemale praktikale omane tsükliline paranduste tegemine harmoniseerub hästi tänapäevase robootika kursuse jaoks vajaliku kiire arenguga. Taristu kui kood (infrastructure as code), versioonihaldussüsteemid ja püüd rakendada pideva integratsiooni (continuous integration) põhimõtteid - on kõik osa kursuse arengust. Robootika kursus on teemaks oleva kahe toimumiskorra jooksul aidanud arendada enam kui saja tudengi teadmisi mõningatest tänapäevases robootikas kasutatavatest töövahenditest ja tehnoloogiatest. Töös kajastatud kursuse arendusprotsess on toetanud robootikaalaste teadmistega tudengite ning nende juhendajate järelkasvu, mis loob aluse keerukamate kursuste arendamiseks ja kasvatab tehnoloogiainstituudi - ning ülikooli üldisemalt - võimekust olla koostööpartner robootika valdkonnas. Autor kandis vastutava õppejõu rolli, juhtides kursuse arendust, uute tehnoloogiate kasutamist ning juurutamist. Töö eesmärk on olla läbitud arengu, tehtud valikute, ning nende põhjenduste süstematiseeritud dokumentatsioon.This thesis covers the design and development process of a robotics course. The course in question serves as an entry point into the field of robotics in the University of Tartu. The design of the thesis draws on the ideas of educational action research and the infrastructure of the course on the software development and information technology operations practices (DevOps). Cycles of improvements, which are core to both of these practices, are in tune with the rapid evolution required of a modern robotics course.Infrastructure as code, version control systems, and incentive towards continuous integration are all used in the development process. The two runs of the course covered by the thesis have advanced the knowledge of more than a hundred students in some of the modern tools and technologies used in robotics. Altogether, the development of the course has helped to reinforce the line-up of students and instructors available for projects andmore advanced courses in the Institute of Technology and the university as a whole. The author, being the lecturer in charge leading the development the course and also pushing for the use of new technology and maintaining chosen solutions. The thesis serves as an organised documentation on the evolution process and reasoning for the choices made

Programming tools for intelligent systems

Les outils de programmation sont des programmes informatiques qui aident les humains à programmer des ordinateurs. Les outils sont de toutes formes et tailles, par exemple les éditeurs, les compilateurs, les débogueurs et les profileurs. Chacun de ces outils facilite une tâche principale dans le flux de travail de programmation qui consomme des ressources cognitives lorsqu’il est effectué manuellement. Dans cette thèse, nous explorons plusieurs outils qui facilitent le processus de construction de systèmes intelligents et qui réduisent l’effort cognitif requis pour concevoir, développer, tester et déployer des systèmes logiciels intelligents. Tout d’abord, nous introduisons un environnement de développement intégré (EDI) pour la programmation d’applications Robot Operating System (ROS), appelé Hatchery (Chapter 2). Deuxièmement, nous décrivons Kotlin∇, un système de langage et de type pour la programmation différenciable, un paradigme émergent dans l’apprentissage automatique (Chapter 3). Troisièmement, nous proposons un nouvel algorithme pour tester automatiquement les programmes différenciables, en nous inspirant des techniques de tests contradictoires et métamorphiques (Chapter 4), et démontrons son efficacité empirique dans le cadre de la régression. Quatrièmement, nous explorons une infrastructure de conteneurs basée sur Docker, qui permet un déploiement reproductible des applications ROS sur la plateforme Duckietown (Chapter 5). Enfin, nous réfléchissons à l’état actuel des outils de programmation pour ces applications et spéculons à quoi pourrait ressembler la programmation de systèmes intelligents à l’avenir (Chapter 6).Programming tools are computer programs which help humans program computers. Tools come in all shapes and forms, from editors and compilers to debuggers and profilers. Each of these tools facilitates a core task in the programming workflow which consumes cognitive resources when performed manually. In this thesis, we explore several tools that facilitate the process of building intelligent systems, and which reduce the cognitive effort required to design, develop, test and deploy intelligent software systems. First, we introduce an integrated development environment (IDE) for programming Robot Operating System (ROS) applications, called Hatchery (Chapter 2). Second, we describe Kotlin∇, a language and type system for differentiable programming, an emerging paradigm in machine learning (Chapter 3). Third, we propose a new algorithm for automatically testing differentiable programs, drawing inspiration from techniques in adversarial and metamorphic testing (Chapter 4), and demonstrate its empirical efficiency in the regression setting. Fourth, we explore a container infrastructure based on Docker, which enables reproducible deployment of ROS applications on the Duckietown platform (Chapter 5). Finally, we reflect on the current state of programming tools for these applications and speculate what intelligent systems programming might look like in the future (Chapter 6)