Recognizing object surface material from impact sounds for robot manipulation

© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We investigated the use of impact sounds generated during exploratory behaviors in a robotic manipulation setup as cues for predicting object surface material and for recognizing individual objects. We collected and make available the YCB-impact sounds dataset which includes over 3,000 impact sounds for the YCB set of everyday objects lying on a table. Impact sounds were generated in three modes: (i) human holding a gripper and hitting, scratching, or dropping the object; (ii) gripper attached to a teleoperated robot hitting the object from the top; (iii) autonomously operated robot hitting the objects from the side with two different speeds. A convolutional neural network is trained from scratch to recognize the object material (steel, aluminium, hard plastic, soft plastic, other plastic, ceramic, wood, paper/cardboard, foam, glass, rubber) from a single impact sound. On the manually collected dataset with more variability in the speed of the action, nearly 60% accuracy for the test set (not presented objects) was achieved. On a robot setup and a stereotypical poking action from top, accuracy of 85% was achieved. This performance drops to 79% if multiple exploratory actions are combined. Individual objects from the set of 75 objects can be recognized with a 79% accuracy. This work demonstrates promising results regarding the possibility of using impact sound for recognition in tasks like single-stream recycling where objects have to be sorted based on their material composition.This work was supported by the project Interactive Perception-Action-Learning for Modelling Objects (IPALM) (H2020 – FET – ERA-NET Cofund – CHIST-ERA III / Technology Agency of the Czech Republic, EPSILON, no. TH05020001) and partially supported by the project MDM2016-0656 funded by MCIN/ AEI /10.13039/501100011033. M.D. was supported by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033 and by “ESF Investing in your future”. S.P. and M.H. were additionally supported by OP VVV MEYS funded project CZ.02.1.01/0.0/0.0/16 019/0000765 “Research Center for Informatics”. We thank Bedrich Himmel for assistance with sound setup, Antonio Miranda and Andrej Kruzliak for data collection, and Lukas Rustler for video preparation.This work was supported by the project Interactive Perception-Action-Learning for Modelling Objects (IPALM) (H2020 – FET – ERA-NET Cofund – CHIST-ERA III / Technology Agency of the Czech Republic, EPSILON, no. TH05020001) and partially supported by the project MDM2016-0656 funded by MCIN/ AEI /10.13039/501100011033. M.D. was supported by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033 and by “ESF Investing in your future”. S.P. and M.H. were additionally supported by OP VVV MEYS funded project CZ.02.1.01/0.0/0.0/16 019/0000765 “Research Center for Informatics”. We thank Bedrich Himmel for assistance with sound setup, Antonio Miranda and Andrej Kruzliak for data collection, and Lukas Rustler for video preparation.Peer ReviewedPostprint (author's final draft

EDAMS: an Encoder-Decoder Architecture for Multi-grasp Soft sensing object recognition

The use of tactile sensing exhibits benefits over visual detection as it can be deployed in occluded environments and can provide deeper information about an object's material properties. Soft hands have increasingly been used for tactile object identification, providing a high degree of adaptability without requiring complex control schemes. In this work, we propose a framework for identifying a range of objects in any pose by exploiting the compliance of a soft hand equipped with distributed tactile sensing. We propose EDAMS, an Encoder-Decoder Architecture for Multi-grasp Soft sensing and an ad-hoc data structure capable of encoding information on multiple grasps, while decoupling the dependency on the pose order. We train the model to map the high-dimensional multi-grasp tactile sensor data into a lower-dimensional latent space capable of achieving the geometrical separation of each object class, and enabling accurate object classification. We provide an empirical analysis of the benefit of multi-grasp perception for object identification, and show its impact on the separation of the objects in sensor space. Notably, we find the classification accuracy to change widely across the number of grasps, ranging from 47.0% for a single grasp, to 99.9% for 10 grasps

Scalable Tactile Sensing for an Omni-adaptive Soft Robot Finger

Robotic fingers made of soft material and compliant structures usually lead to superior adaptation when interacting with the unstructured physical environment. In this paper, we present an embedded sensing solution using optical fibers for an omni-adaptive soft robotic finger with exceptional adaptation in all directions. In particular, we managed to insert a pair of optical fibers inside the finger's structural cavity without interfering with its adaptive performance. The resultant integration is scalable as a versatile, low-cost, and moisture-proof solution for physically safe human-robot interaction. In addition, we experimented with our finger design for an object sorting task and identified sectional diameters of 94\% objects within the $\pm$6mm error and measured 80\% of the structural strains within $\pm$0.1mm/mm error. The proposed sensor design opens many doors in future applications of soft robotics for scalable and adaptive physical interactions in the unstructured environment.Comment: 8 pages, 6 figures, full-length version of a submission to IEEE RoboSoft 202

Design of a 3D-printed soft robotic hand with distributed tactile sensing for multi-grasp object identification

Tactile object identification is essential in environments where vision is occluded or when intrinsic object properties such as weight or stiffness need to be discriminated between. The robotic approach to this task has traditionally been to use rigid-bodied robots equipped with complex control schemes to explore different objects. However, whilst varying degrees of success have been demonstrated, these approaches are limited in their generalisability due to the complexity of the control schemes required to facilitate safe interactions with diverse objects. In this regard, Soft Robotics has garnered increased attention in the past decade due to the ability to exploit Morphological Computation through the agent's body to simplify the task by conforming naturally to the geometry of objects being explored. This exists as a paradigm shift in the design of robots since Soft Robotics seeks to take inspiration from biological solutions and embody adaptability in order to interact with the environment rather than relying on centralised computation. In this thesis, we formulate, simplify, and solve an object identification task using Soft Robotic principles. We design an anthropomorphic hand that has human-like range of motion and compliance in the actuation and sensing. The range of motion is validated through the Feix GRASP taxonomy and the Kapandji Thumb Opposition test. The hand is monolithically fabricated using multi-material 3D printing to enable the exploitation of different material properties within the same body and limit variability between samples. The hand's compliance facilitates adaptable grasping of a wide range of objects and features integrated distributed tactile sensing. We emulate the human approach of integrating information from multiple contacts and grasps of objects to discriminate between them. Two bespoke neural networks are designed to extract patterns from both the tactile data and the relationships between grasps to facilitate high classification accuracy

Neuromorphic vision based contact-level classification in robotic grasping applications

In recent years, robotic sorting is widely used in the industry, which is driven by necessity and opportunity. In this paper, a novel neuromorphic vision-based tactile sensing approach for robotic sorting application is proposed. This approach has low latency and low power consumption when compared to conventional vision-based tactile sensing techniques. Two Machine Learning (ML) methods, namely, Support Vector Machine (SVM) and Dynamic Time Warping-K Nearest Neighbor (DTW-KNN), are developed to classify material hardness, object size, and grasping force. An Event-Based Object Grasping (EBOG) experimental setup is developed to acquire datasets, where 243 experiments are produced to train the proposed classifiers. Based on predictions of the classifiers, objects can be automatically sorted. If the prediction accuracy is below a certain threshold, the gripper re-adjusts and re-grasps until reaching a proper grasp. The proposed ML method achieves good prediction accuracy, which shows the effectiveness and the applicability of the proposed approach. The experimental results show that the developed SVM model outperforms the DTW-KNN model in term of accuracy and efficiency for real time contact-level classification

Factories of the Future

Engineering; Industrial engineering; Production engineerin

Robotic disassembly of electronic components to support end‐of‐life recycling of electric vehicles

This thesis reports on the research undertaken to analyse the factors affecting End-of-Life (EoL) recycling of future Electric Vehicles (EVs). The principle objective of the research is to generate an understanding of challenges and opportunities for the development and implementation of an automated robotic disassembly approach to aid with EoL management of electrical and electronic components within EVs. The research contributions are considered in three main parts. The first part contains a review of advancement in the development of automotive technology, and in particular the alternative fuel vehicles. A review of existing industrial recycling technologies and processes has been conducted which highlighted a number of key challenges in the adoption of current recycling technologies for EVs. The review concludes that there is a need to develop novel recycling technologies and processes to deal with the increased part complexity and material mixture in such vehicles. In this context, the second part of the research details a framework for EoL management of EV components. This framework presents a comprehensive automated robotic disassembly approach in which three specific steps are defined, namely manual disassembly to develop an understanding of product design, initial automated disassembly to test process capability, and optimisation and validation to improve repeatability and efficiency of the robotic disassembly operations. The framework also includes the development of a multi-criteria decision-making tool that assesses the environmental, technological and economic benefits of such robotic disassembly approach. The applicability of the research concepts has been demonstrated via three case studies. The results have highlighted the applicability of the automated robotic disassembly approach in a variety of scenarios of different design complexity and recovery rate. The results indicate that the adoption of this robotic disassembly enhances the pre-concentration of Strategically Important Materials (SIMs) and leads to minimisation of environmental impacts and increased material recovery value

Modular robots for sorting

Current industrial sorting systems allow for low error, high throughput sorts with tightly constrained properties. These sorters, however, are often hardware limited to certain items and criteria. There is a need for more adaptive sorting systems for processes that involve a high throughput of heterogeneous items such as import testing by port authorities, warehouse sorting for online retailers, and sorting recycling. The variety of goods that need to be sorted in these applications mean that existing sorting systems are unsuitable, and the objects often need to be sorted by hand. In this work I detail my design and control of a modular, robotic sorting system using linear actuating robots to create both low-frequency vibrations and peristaltic waves for sorting. The adaptability of the system allows for multimodal sorting and can improve heterogeneous sorting systems. I have designed a novel modular robot called the Linbot. These Linbots are based on an actuator design utilising a voice coil for linear motion. I designed this actuator to be part of a modular robot by adding on-board computation, sensing and communication. I demonstrate the individual characteristics of these robots through a series of experiments in order to give a comprehensive overview of their abilities. By combining multiple Linbots in a collective I demonstrate their ability to move and sort objects using cooperative peristaltic motion. In order to allow for rapid optimisation of control schemes for Linbot collectives I required a peristaltic table simulator. I designed and implemented a peristaltic table simulator, called PeriSim, due to a lack of existing solutions. Controllers optimised in simulation often suffer from a reduction in performance when moved to a real-world system due to the inaccuracies in the simulation, this effect is called the reality gap. I used a method for reducing the reality gap called the radical envelope of noise hypothesis, whereby I only modelled the key aspects of peristalsis in PeriSim and then varied the underlying physics of the simulation between runs. I used PeriSim to optimise controllers in simulation that worked on a real-world system. I demonstrate the how the Linbots and PeriSim can be used to build and control an adaptive sorter. I built an adaptive sorter made of a 5x5 grid of Linbots with a soft sheet covering them. I demonstrate that the sorter can grade produce and move objects of varying shapes and sizes. My work can guide the future design of industrial adaptive sorting systems