Improving the Visual Perception of Heavy Duty Manipulators in Challenging Scenarios

Robotic vision is a subfield of computer vision intended to provide robots with the capability to visually perceive the surrounding environment. For example, a robotic manipulator leverages its visual perception system to gather visual data through cameras and other sensors, then uses that input to recognize different objects in order to safely perform an autonomous operation. However, in many robotics applications, robots have to face a cluttered and dynamic scene, where classic computer vision algorithms show the limitation of tackling the environmental uncertainty. Such scene understanding requires a fusion of traditional and modern approaches involving classic computer vision, machine learning and deep learning methods. This thesis examines visual perception challenges in remote handling and the mining industry. It begins with two research questions: Can the robustness of targetobject pose estimation be improved in challenging real-world, heavy-duty robotic scenarios? Can fast detection and localization for objects be obtained without prior known geometry in a scenario with piles of overlapping objects? Six publications cover the methods from algorithm design to system-level integration used to solve real-world problems. In the ITER fusion reactor, the operator teleoperates a robotic manipulator to perform maintenance tasks amidst a high level of noise and erosion. The operator cannot fully rely on the virtual reality (VR) system, which may not reflect the current scene accurately, as physical conditions may have changed in the harsh environment. Meanwhile, every operation inside the reactor requires robust, millimeterlevel accuracy. This thesis analyzes research questions and presents a novel edgepoint iterative closest point (ICP) method as a solution for target-object detection, tracking and pose estimation. Using the knuckle of a divertor cassette as an example, the overall accuracy of the developed visual system meets ITER requirements, and the conducted experiments with the manipulator demonstrated the efficiency of the method. Smartbooms2 is a project in the mining industry that requires a heavy manipulator with a hydraulic hammer to autonomously break rocks in a cluttered outdoor environment. Based on the output data of the three-dimensional (3D) sensors, several solutions are proposed. Examining a popular time-of-flight (TOF) sensor, this thesis explores state-of-the-art unsupervised machine learning methods and proposes a novel clustering method. Using an industrial stereo camera, this thesis proposes a novel 3D rock detection and localization pipeline. The results and system accuracy are detailed in published research papers

Integration of multi-camera vision system for automatic robotic assembly

An automated assembly cell does not come with the luxury of intelligence. A non-adaptive robot manipulator must have more accurate parts and assembly fixtures to a greater accuracy than their assembly tolerances. Accurate parts presentation and the fixturing of assemblies can be a complex and costly portion of an automated cell. Machine vision can be integrated with the assembly process to effectively and efficiently guide the assembly process. This paper presents a framework for an automatic assembly system that consists of an industrial robot and a three-camera vision system, where the tasks involve object detection, pose estimation during pick and place operations, and assembly verification. Given a pocket calculator as an application, the proposed framework can successfully perform autonomous assembly of a pocket calculator. The experiment results verified the efficacy of the proposed method.publishedVersionPeer reviewe

Efficient 3D visual perception for robotic rock breaking

In recent years, underground mining automation (e.g., the heavy-duty robots carrying rock breaker tools for secondary breaking) has drawn substantial interest. This breaking process is needed only when over-sized rocks threaten to jam the mine material flow. In the worst case, a pile of overlapped rocks can get stuck on top of a crusher's grate plate. For a human operator, it is relatively easy to make the decisions about the rock locations in the pile and the order of rocks to be crushed. In an autonomous operation, a robust and fast visual perception system is needed for executing robot motion commands. In this paper, we propose a pipeline for fast detection and pose estimation of individual rocks in cluttered scenes. We employ the state-of-art YOLOv3 as a 2D detector to perform 3D reconstruction from point cloud for detected rocks in 2D regions using our proposed novel method, and finally estimating the rock centroid positions and the normal-to-surface vectors based on the predicted point cloud. The detected centroids in the scene are ordered according to the depth of rock surface to the camera, which provides the breaking sequence of the rocks. During the system evaluation in the real rock breaking experiments, we have collected a new dataset with 4780 images having from 1 to 12 rocks on a grate plate. The proposed pipeline achieves 97.47% precision on overall detection with a real-time speed around 15Hz.acceptedVersionPeer reviewe

A stereoscopic eye-in-hand vision system for remote handling in ITER

acceptedVersionPeer reviewe

Autonomous robotic rock breaking using a real‐time 3D visual perception system

Crushing of blasted ore is an essential phase in extraction of valuable minerals in mining industry. It is typically performed in multiple stages with each stage producing finer fragmentation. Performance and throughput of the first stage of crushing is highly dependent on the size distribution of the blasted ore. In the crushing plant, a metal grate prevents oversized boulders from getting into the crusher jaws, and a human-controlled hydraulic manipulator equipped with a rock hammer is required to break oversized boulders and ensure continuous material flow. This secondary breaking task is event-based in the sense that ore trucks deliver boulders at irregular intervals, thus requiring constant human supervision to ensure continuous material flow and prevent blockages. To automatize such breaking tasks, an intelligent robotic control system along with a visual perception system (VPS) is essential. In this manuscript, we propose an autonomous breaker system that includes a VPS capable of detecting multiple irregularly shaped rocks, a robotic control system featuring a decision-making mechanism for determining the breaking order when dealing with multiple rocks, and a comprehensive manipulator control system. We present a proof of concept for an autonomous robotic boulder breaking system, which consists of a stereo-camera-based VPS and an industrial rock-breaking manipulator robotized with our retrofitted system design. The experiments in this study were conducted in a real-world setup, and the results were evaluated based on the success rates of breaking. The experiments yielded an average success rate of 34% and a break pace of 3.3 attempts per minute.publishedVersionPeer reviewe

Eye-in-Hand Manipulation for Remote Handling: Experimental Setup

A prototype for eye-in-hand manipulation in the context of remote handling in the fusion power plant ITER is presented in this paper. The setup consists of an industrial robot manipulator with a modified open control architecture and equipped with a pair of stereoscopic cameras, a force/torque sensor, and pneumatic tools. It is controlled through a haptic device in a mock-up environment. The industrial robot controller has been replaced by a single industrial PC running Xenomai that has a real-time connection to both the robot controller and another Linux PC running as the controller for the haptic device. The new remote handling control environment enables further development of advanced control schemes for autonomous and semi-autonomous manipulation tasks. This setup benefits from a stereovision system for accurate tracking of the target objects with irregular shapes. The overall environmental setup successfully demonstrates the required robustness and precision that remote handling tasks need.publishedVersionPeer reviewe