Manipulating Highly Deformable Materials Using a Visual Feedback Dictionary

The complex physical properties of highly deformable materials such as clothes pose significant challenges fanipulation systems. We present a novel visual feedback dictionary-based method for manipulating defoor autonomous robotic mrmable objects towards a desired configuration. Our approach is based on visual servoing and we use an efficient technique to extract key features from the RGB sensor stream in the form of a histogram of deformable model features. These histogram features serve as high-level representations of the state of the deformable material. Next, we collect manipulation data and use a visual feedback dictionary that maps the velocity in the high-dimensional feature space to the velocity of the robotic end-effectors for manipulation. We have evaluated our approach on a set of complex manipulation tasks and human-robot manipulation tasks on different cloth pieces with varying material characteristics.Comment: The video is available at goo.gl/mDSC4

3D Spectral Domain Registration-Based Visual Servoing

This paper presents a spectral domain registration-based visual servoing scheme that works on 3D point clouds. Specifically, we propose a 3D model/point cloud alignment method, which works by finding a global transformation between reference and target point clouds using spectral analysis. A 3D Fast Fourier Transform (FFT) in R3 is used for the translation estimation, and the real spherical harmonics in SO(3) are used for the rotations estimation. Such an approach allows us to derive a decoupled 6 degrees of freedom (DoF) controller, where we use gradient ascent optimisation to minimise translation and rotational costs. We then show how this methodology can be used to regulate a robot arm to perform a positioning task. In contrast to the existing state-of-the-art depth-based visual servoing methods that either require dense depth maps or dense point clouds, our method works well with partial point clouds and can effectively handle larger transformations between the reference and the target positions. Furthermore, the use of spectral data (instead of spatial data) for transformation estimation makes our method robust to sensor-induced noise and partial occlusions. We validate our approach by performing experiments using point clouds acquired by a robot-mounted depth camera. Obtained results demonstrate the effectiveness of our visual servoing approach.Comment: Accepted to 2023 IEEE International Conference on Robotics and Automation (ICRA'23

Direct Visual Servoing Based on Discrete Orthogonal Moments

This paper proposes a new approach to achieve direct visual servoing (DVS) based on discrete orthogonal moments (DOM). DVS is conducted whereby the extraction of geometric primitives, matching and tracking steps in the conventional feature-based visual servoing pipeline can be bypassed. Although DVS enables highly precise positioning, and suffers from a small convergence domain and poor robustness, due to the high non-linearity of the cost function to be minimized and the presence of redundant data between visual features. To tackle these issues, we propose a generic and augmented framework to take DOM as visual features into consideration. Through taking Tchebichef, Krawtchouk and Hahn moments as examples, we not only present the strategies for adaptive adjusting the parameters and orders of the visual features, but also exhibit the analytical formulation of the associated interaction matrix. Simulations demonstrate the robustness and accuracy of our method, as well as the advantages over the state of the art. The real experiments have also been performed to validate the effectiveness of our approach

Adaptive Neuro-Filtering Based Visual Servo Control of a Robotic Manipulator

This paper focuses on the solutions to flexibly regulate robotic by vision. A new visual servoing technique based on the Kalman filtering (KF) combined neural network (NN) is developed, which need not have any calibration parameters of robotic system. The statistic knowledge of the system noise and observation noise are first given by Gaussian white noise sequences, the nonlinear mapping between robotic vision and motor spaces are then on-line identified using standard Kalman recursive equations. In real robotic workshops, the perfect statistic knowledge of the noise is not easy to be derived, thus an adaptive neuro-filtering approach based on KF is also studied for mapping on-line estimation in this paper. The Kalman recursive equations are improved by a feedforward NN, in which the neural estimator dynamic adjusts its weights to minimize estimation error of robotic vision-motor mapping, without the knowledge of noise variances. Finally, the proposed visual servoing based on adaptive neuro-filtering has been successfully implemented in robotic pose regulation, and the experimental results demonstrate its validity and practicality for a six-degree-of-freedom (DOF) robotic system which the hand-eye without calibrated

Methods for visual servoing of robotic systems: A state of the art survey

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.This paper surveys the methods used for visual servoing of robotic systems, where the main focus is on mobile robot systems. The three main areas of research include the Direct Visual Servoing, stereo vision systems, and artificial intelligence in visual servoing. The standard methods such as Image-Based Visual Servoing (IBVS) and Position-Based Visual Servoing (PBVS) are analyzed and compared with the new method named Direct Visual Servoing (DVS). DVS methods have better accuracy, compared to IBVS and PBVS, but have limited convergence area. Because of their high accuracy, DVS methods are suitable for integration into hybrid systems. Furthermore, the use of the stereo systems for visual servoing is comprehensively analyzed. The main contribution of the stereo system is the accurate depth estimation, which is critical for many visual servoing tasks. The use of artificial intelligence (AI) in visual servoing purposes has also gained popularity over the years. AI techniques give visual servoing controllers the ability to learn by using predefined examples or empirical knowledge. The learning ability is crucial for the implementation of robotic systems in a real-world dynamic manufacturing environment. Also, we analyzed the use of visual odometry in combination with a visual servoing controller for creating more robust and reliable positioning system

Inteligentno stereo-vizuelno upravljanje mobilnih robota i optimalno terminiranje tehnoloških procesa - pregled rezultata istraživanja u okviru projekta MISSION4.0/Intelligent stereo-visual mobile robot control and optimal process planning and scheduling – overview of research results within the project MISSION4.0

Projekat MISSION4.0 podrazumevao je, u okviru nekoliko radnih paketa, razvoj inteligentnog stereo-vizuelnog upravljanja mobilnih robota, kao i optimalno planiranje i terminiranje tehnoloških procesa, i to baziranim na tehnikama veštačke inteligencije, posebno na konvolucionim veštačkim neuronskim mrežama i biološki inspirisanim algoritmima optimizacije. Tokom dvogodišnjih intenzivnih naučnih istraživanja razvijena je nova metodologija za autonomnu navigaciju i inteligentno upravljanje mobilnih robota sopstvenog razvoja, nazvanih RAICO i DOMINO. Generisanje optimalnog plana terminiranja tehnoloških procesa, u okviru koga se izvršava i inteligentni unutrašnji transport korišćenjem mobilnih robota, takođe je bio jedan od važnih ciljeva ovih naprednih istraživanja. U ovom radu, dat je pregled nekih od ključnih rezultata projekta MISSION4.0, poput publikovanih u vodećim međunarodnim i nacionalnim naučnim časopisima, objavljenih poglavlja u naučnim monografijama, saopštenih i odštampanih naučnih radova u zbornicima prestižnih konferencija održanih u inostranstvu i regionu, zatim u okviru verifikovanih tehničkih rešenja, kao i preko skupova podataka sa otvorenim pristupom

Inteligentno stereo-vizuelno upravljanje mobilnih robota i optimalno terminiranje tehnoloških procesa - pregled rezultata istraživanja u okviru projekta MISSION4.0/Intelligent stereo-visual mobile robot control and optimal process planning and scheduling – overview of research results within the project MISSION4.0

Projekat MISSION4.0 podrazumevao je, u okviru nekoliko radnih paketa, razvoj inteligentnog stereo-vizuelnog upravljanja mobilnih robota, kao i optimalno planiranje i terminiranje tehnoloških procesa, i to baziranim na tehnikama veštačke inteligencije, posebno na konvolucionim veštačkim neuronskim mrežama i biološki inspirisanim algoritmima optimizacije. Tokom dvogodišnjih intenzivnih naučnih istraživanja razvijena je nova metodologija za autonomnu navigaciju i inteligentno upravljanje mobilnih robota sopstvenog razvoja, nazvanih RAICO i DOMINO. Generisanje optimalnog plana terminiranja tehnoloških procesa, u okviru koga se izvršava i inteligentni unutrašnji transport korišćenjem mobilnih robota, takođe je bio jedan od važnih ciljeva ovih naprednih istraživanja. U ovom radu, dat je pregled nekih od ključnih rezultata projekta MISSION4.0, poput publikovanih u vodećim međunarodnim i nacionalnim naučnim časopisima, objavljenih poglavlja u naučnim monografijama, saopštenih i odštampanih naučnih radova u zbornicima prestižnih konferencija održanih u inostranstvu i regionu, zatim u okviru verifikovanih tehničkih rešenja, kao i preko skupova podataka sa otvorenim pristupom