Mixture of experts on Riemannian manifolds for visual-servoing fixtures

Adaptive Virtual Fixtures (VFs) for teleoperation often rely on visual inputs for online adaptation. State estimation from visual detections is never perfect, and thus affects the quality and robustness of adaptation. It is therefore important to be able to quantify how uncertain an estimation from vision is. This can, for example, inform on how to modulate a fixture's stiffness to decrease the physical force a human operator has to apply. Furthermore, the target of a manipulation operation might not be known from the beginning of the task, which creates the need for a principled way to add and remove fixtures when possible targets appear in the robot workspace. In this paper we propose an on-manifold Mixture of Experts (MoE) model that synthesizes visual-servoing fixtures while elegantly handling full pose detection uncertainties and 6D teleoperation goals in a unified framework. An arbitration function allocating the authority between multiple vision-based fixtures arises naturally from the MoE formulation. We show that this approach allows a teleoperator to insert multiple printed circuit boards (PCBs) with high precision without requiring the manual design of VFs to guide the robot motion. An exemplary video visualizing the probability distribution resulting from our model is available at: https://youtu.be/GKMQvbJ5Oz

Multi-Phase Multi-Modal Haptic Teleoperation

Virtual Fixtures facilitate teleoperation, for instance by guiding the human operator. Developing these Virtual Fixtures in tasks with tight tolerances remains challenging. Fixtures with a high stiffness allow for more precise guidance, whereas a lower stiffness is required to allow for corrections. We observed that many assembly operations can be split into different phases - approaching, positioning, in-contact manipulation - each with different accuracy requirements. Therefore, we propose to use multi-modal fixtures, satisfying the different requirements of these phases: i.e. a position-based Trajectory Fixture for approaching and a more accurate Visual Servoing Fixture for the positioning phase. A state estimation and arbitration component ensures smooth transitions between the fixtures to provide optimal support for the operator and to achieve global availability paired with local precision at the same time. It also allows a high stiffness to be used throughout, thus achieving good guidance for all phases. The approach is validated in an application from a space scenario, consisting of the assembly of a CubeSat subsystem. The empirical results from a pilot study on this task show that our approach is faster and requires less interaction force from the operator than the baseline method

Transforming scholarship in the archives through handwritten text recognition:Transkribus as a case study

Purpose: An overview of the current use of handwritten text recognition (HTR) on archival manuscript material, as provided by the EU H2020 funded Transkribus platform. It explains HTR, demonstrates Transkribus, gives examples of use cases, highlights the affect HTR may have on scholarship, and evidences this turning point of the advanced use of digitised heritage content. The paper aims to discuss these issues. - Design/methodology/approach: This paper adopts a case study approach, using the development and delivery of the one openly available HTR platform for manuscript material. - Findings: Transkribus has demonstrated that HTR is now a useable technology that can be employed in conjunction with mass digitisation to generate accurate transcripts of archival material. Use cases are demonstrated, and a cooperative model is suggested as a way to ensure sustainability and scaling of the platform. However, funding and resourcing issues are identified. - Research limitations/implications: The paper presents results from projects: further user studies could be undertaken involving interviews, surveys, etc. - Practical implications: Only HTR provided via Transkribus is covered: however, this is the only publicly available platform for HTR on individual collections of historical documents at time of writing and it represents the current state-of-the-art in this field. - Social implications: The increased access to information contained within historical texts has the potential to be transformational for both institutions and individuals. - Originality/value: This is the first published overview of how HTR is used by a wide archival studies community, reporting and showcasing current application of handwriting technology in the cultural heritage sector

Visual-Inertial RGB-D Mapping for Quadruped Locomotion

Quadruped Locomotion can profit from a local map by determining optimal foot placement locations and planning the foot motion knowing the ground height at foot position. Rapid motion of the robot necessites a robust yet accurate method to estimate odometry able to deal with high amounts of motion blur. In this work, the Realsense D435i RGB-D camera equipped with an IMU (both accelerometer and gyroscope) is used in a dense visual-inertial odometry algorithm to infer the motion. The dense odometry algorithm is based on DVO, it uses depth and intensity information for odometry estimation. It is fused in a tightly-coupled manner in a Gauss-Newton optimization scheme with the preintegrated IMU data. The vision sensor and IMU are calibrated using the algorithm of Rehder et al. Dense visual descriptor learning is being evaluated to relax the brightness constancy assumption and to improve the robustness to motion blur. Different network architectures are trained to output features at different scales. Two approaches for training are being evaluated, one using pixel-wise loss functions and the other using two RGB-D images as input which are then aligned using a differentiable implementation of DVO. The effectiveness of the full algorithm is evaluated on the ETH3D benchmark which contains accurately calibrated and synchronized RGB-D and IMU data. It is shown that the algorithm is in principal suitable for the problem of quadruped locomotion. Its accuracy however highly depends on the environment and the depth sensor quality. Thus, several possible future research directions are proposed to better deal with these deficiencies

ACOR - AI-enabled Cyber-Physical In-Orbit Self-Recovering Factory

Durch Verwendung von modularen und standardisierten Komponenten können individuell konfigurierte CubeSats schnell, effizient und angepasst auf ihre jeweiligen Missionsanforderungen gefertigt werden. Mit dem Ansatz einer In-Orbit Factory ist es möglich, den Zeitraum von Bestellung zur Inbetriebnahme im Orbit noch weiter zu verringern. Dies erlaubt die Vision eines Betriebsstarts nur Stunden nach der Spezifikation. Jedoch benötigt das Konzept einer hochgradig automatisierten und fehlertoleranten Fabrik im Orbit adaptive und extrem zuverlässige Prozesse für den Zusammenbau und anschließenden Test der CubeSats. Ziel des Projekts AI-enabled Cyber-Physical In-Orbit Self-Recovering Factory (ACOR) ist es, aufbauend auf den Ergebnisse der Vorgängerprojekte Space Factory 4.0 und AI-In-Orbit-Factory, die dafür notwendigen Methoden basierend auf Ansätzen aus Industrie 4.0 und künstlicher Intelligenz (KI) zu entwickeln. Da nach dem Start der In-Orbit Factory kein direkter menschlicher Eingriff in den Produktionsprozess mehr möglich ist, wurden speziell Methoden zur automatisierten Fehlerdetektion und -behebung (FDIR) als wesentliches Merkmal in der Methodik einer Fabrik im Orbit identifiziert. Zunächst müssen deshalb existierende Methoden für den automatisierten robotischen Zusammenbau und Test (AIT) um solche FDIR-Ansätze erweitert werden, um bei der automatisierten Produktion von Miniatursatelliten auftretende typische Fehler zuverlässig zu detektieren, identifizieren und sofern möglich korrigieren zu können. Dabei sollen sowohl Defekte in Satellitenkomponenten als auch Fehler im AIT-Prozess selbst behandelt werden. Weiterhin muss aufgrund der Komplexität der In-Orbit Factory jedoch immer mit Fehlerfällen gerechnet werden, die nicht automatisch korrigiert werden können. Auch kann ein individueller Eingriff in den Bauprozess notwendig werden, der die hohen kognitiven Fähigkeiten eines Menschen erfordert. Für diese Fälle ist im Projekt eine Teleoperationsschnittstelle basierend auf adaptiven Virtual Fixtures (VFs) vorgesehen, die einem menschlichen Operator Eingriffsmöglichkeiten auf der Erde mit Kraftfeedback bereitstellt. Um diesen adaptiven VF-Ansatz basierend auf positions- und kamerabasierten Fixtures robuster zu gestalten, werden probabilistische Methoden genutzt, welche die Unsicherheit der aktuellen Fixture basierend auf gelernten und auf Kamerabild-basierten Schätzungen modulieren. Dieses Vorgehen erlaubt es, entsprechende Gewichtungen zwischen verschiedenen Fixtures und dem Teleoperator vorzunehmen. Darüber hinaus repräsentiert ein Digitaler Prozesszwilling (DPT) die Prozessdaten sowohl der automatisierten als auch der teleoperierten Montage und orchestriert den Produktionsprozess. Seine Fehlerdetektions- und -recoveryfähigkeit werden standardisiert und die Interaktion mit den digitalen Zwillingen der Produktionsressourcen und des Produkts untersucht. Weiterhin wird die autonome Planung der Prozessschritte flexibilisiert und optimiert. Auf dem zugehörigen Poster werden diese Ansätze im Detail präsentiert

AI-In-Orbit-Factory - AI approaches for adaptive robotic in-orbit manufacturing of modular satellites

Ongoing advances in modular satellite architectures, coupled with improvements in adaptive manufacturing processes are paving the way for innovations in manufacturing in space and, beyond that, even on-orbit servicing. Current challenges for in-orbit manufacturing of satellites include, in particular, highly reliable, precise and adaptive manufacturing and inspection processes, teleoperation methods to resolve unexpected problems from Earth, and means for a digital representation of all relevant activities and conditions to maintain full control. Each challenge is addressed in the project AI-In-Orbit-Factory with various of AI methods. For the necessary digital representation of the in-orbit factory and all ongoing processes a knowledge-based approach and digital-twin methodology is used, which enables adaptive, flexible and understandable manufacturing processes. Especially the complex information flow between different manufacturing machines, digital process twins that orchestrate the production process and digital twins of satellites in production can be described. Furthermore, conflicts and possible error sources can be identified through inference. Utilizing the aforementioned knowledge base and standardized modular components the composition of a mission specific satellite is automatically planned based on the desired mission requirements. With the help of a robotic manipulator each module is optically inspected for production errors using a high-resolution camera and reference images, before they are integrated into the satellite structure. Once integrated, the submodules undergo optimized testing and anomaly detection routines with learned nominal subsystem behaviour models as input. Additionally, each manipulation step is supervised using force-feedback and vision-based anomaly detectors. For cases where automated assembly fails, a bilateral teleoperation system with force feedback is developed. In order to increase precision during teleoperated assembly and reduce mental and physical load, the human operator is assisted by adaptive virtual fixtures (haptic constraints). Adaptive fixtures are learned from both demonstration and simulation and parametrized depending on the manipulation phase, providing coarse to fine-grained support throughout approaching, positioning and haptic manipulation phases. An arbitration component detects the current manipulation phase to select the appropriate supporting fixture and ensure smooth transitions. This paper outlines the AI methods and our approach to reliable and adaptive in-orbit manufacturing and presents first results

AI-enabled Cyber-Physical In-Orbit Factory - AI approaches based on digital twin technology for robotic small satellite production

With the ever increasing number of active satellites in space, the rising demand for larger formations of small satellites and the commercialization of the space industry (so-called New Space), the realization of manufacturing processes in orbit comes closer to reality. Reducing launch costs and risks, allowing for faster on-demand deployment of individually configured satellites as well as the prospect for possible on-orbit servicing for satellites makes the idea of realizing an in-orbit factory promising. In this paper, we present a novel approach to an in-orbit factory of small satellites covering a digital process twin, AI-based fault detection, and teleoperated robot-control, which are being researched as part of the "AI-enabled Cyber-Physical In-Orbit Factory" project. In addition to the integration of modern automation and Industry 4.0 production approaches, the question of how artificial intelligence (AI) and learning approaches can be used to make the production process more robust, fault-tolerant and autonomous is addressed. This lays the foundation for a later realization of satellite production in space in the form of an in-orbit factory. Central aspect is the development of a robotic AIT (Assembly, Integration and Testing) system where a small satellite could be assembled by a manipulator robot from modular subsystems. Approaches developed to improving this production process with AI include employing neural networks for optical and electrical fault detection of components. Force sensitive measuring and motion training helps to deal with uncertainties and tolerances during assembly. An AI-guided teleoperated control of the robot arm allows for human intervention while a Digital Process Twin represents process data and provides supervision during the whole production process. Approaches and results towards automated satellite production are presented in detail