Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach

We present a robot eye-hand coordination learning method that can directly learn visual task specification by watching human demonstrations. Task specification is represented as a task function, which is learned using inverse reinforcement learning(IRL) by inferring differential rewards between state changes. The learned task function is then used as continuous feedbacks in an uncalibrated visual servoing(UVS) controller designed for the execution phase. Our proposed method can directly learn from raw videos, which removes the need for hand-engineered task specification. It can also provide task interpretability by directly approximating the task function. Besides, benefiting from the use of a traditional UVS controller, our training process is efficient and the learned policy is independent from a particular robot platform. Various experiments were designed to show that, for a certain DOF task, our method can adapt to task/environment variances in target positions, backgrounds, illuminations, and occlusions without prior retraining.Comment: Accepted in ICRA 201

Image Based Visual Servoing: Estimated Image Jacobian by Using Fundamental Matrix VS Analytic Jacobian

This paper describes a comparative study of performance between the estimated image Jacobian that come from taking into account the geometry epipolar of a system of two cameras, and the well known analytic image Jacobian that is utilized for most applications in visual servoing. Image Based Visual Servoing architecture is used for controlling a 3 d.o.f. articular system using two cameras in eye to hand configuration. Tests in static and dynamic cases were carried out, and showed that the performance of estimated Jacobian by using the properties of the epipolar geometry is such as good and robust against noise as the analytic Jacobian. This fact is considered as an advantage because the estimated Jacobian does not need laborious previous work prior the control task in contrast to the analytic Jacobian does

Image based autodocking without calibration

The calibration requirements for visual servoing can make it difficult to apply in many real-world situations. One approach to image-based visual servoing without calibration is to dynamically estimate the image Jacobian and use it as the basis for control. However, with the normal motion of a robot toward the goal, the estimation of the image Jacobian deteriorates over time. The authors propose the use of additional exploratory motion to considerably improve the estimation of the image Jacobian. They study the role of such exploratory motion in a visual servoing task. Simulations and experiments with a 6-DOF robot are used to verify the practical feasibility of the approach

A Comparative Study between Analytic and Estimated Image Jacobian by Using a Stereoscopic System of Cameras

This paper describes a comparative study of performance between the estimated image Jacobian that come from taking into account the epipolar geometry in a system of two cameras, and the well known analytic image Jacobian that is utilized for most applications in visual servoing. Image Based Visual Servoing architecture is used for controlling a 3 DOF articular system using two cameras in eye to hand configuration. Tests in static and dynamic cases were carried out, and showed that the performance of estimated Jacobian by using the properties of the epipolar geometry is such as good and robust against noise as the analytic Jacobian. This fact is considered as an advantage because the estimated Jacobian does not need laborious previous work prior to control task in contrast to the analytic Jacobian does

Bridging Low-level Geometry to High-level Concepts in Visual Servoing of Robot Manipulation Task Using Event Knowledge Graphs and Vision-Language Models

In this paper, we propose a framework of building knowledgeable robot control in the scope of smart human-robot interaction, by empowering a basic uncalibrated visual servoing controller with contextual knowledge through the joint usage of event knowledge graphs (EKGs) and large-scale pretrained vision-language models (VLMs). The framework is expanded in twofold: first, we interpret low-level image geometry as high-level concepts, allowing us to prompt VLMs and to select geometric features of points and lines for motor control skills; then, we create an event knowledge graph (EKG) to conceptualize a robot manipulation task of interest, where the main body of the EKG is characterized by an executable behavior tree, and the leaves by semantic concepts relevant to the manipulation context. We demonstrate, in an uncalibrated environment with real robot trials, that our method lowers the reliance of human annotation during task interfacing, allows the robot to perform activities of daily living more easily by treating low-level geometric-based motor control skills as high-level concepts, and is beneficial in building cognitive thinking for smart robot applications

Voronoi Features for Tactile Sensing: Direct Inference of Pressure, Shear, and Contact Locations

There are a wide range of features that tactile contact provides, each with different aspects of information that can be used for object grasping, manipulation, and perception. In this paper inference of some key tactile features, tip displacement, contact location, shear direction and magnitude, is demonstrated by introducing a novel method of transducing a third dimension to the sensor data via Voronoi tessellation. The inferred features are displayed throughout the work in a new visualisation mode derived from the Voronoi tessellation; these visualisations create easier interpretation of data from an optical tactile sensor that measures local shear from displacement of internal pins (the TacTip). The output values of tip displacement and shear magnitude are calibrated to appropriate mechanical units and validate the direction of shear inferred from the sensor. We show that these methods can infer the direction of shear to $\sim$2.3$^{\circ}$ without the need for training a classifier or regressor. The approach demonstrated here will increase the versatility and generality of the sensors and thus allow sensor to be used in more unstructured and unknown environments, as well as improve the use of these tactile sensors in more complex systems such as robot hands.Comment: Presented at ICRA 201

A review on model-based and model-free approaches to control soft actuators and their potentials in colonoscopy

Colorectal cancer (CRC) is the third most common cancer worldwide and responsible for approximately 1 million deaths annually. Early screening is essential to increase the chances of survival, and it can also reduce the cost of treatments for healthcare centres. Colonoscopy is the gold standard for CRC screening and treatment, but it has several drawbacks, including difficulty in manoeuvring the device, patient discomfort, and high cost. Soft endorobots, small and compliant devices thatcan reduce the force exerted on the colonic wall, offer a potential solution to these issues. However, controlling these soft robots is challenging due to their deformable materials and the limitations of mathematical models. In this Review, we discuss model-free and model-based approaches for controlling soft robots that can potentially be applied to endorobots for colonoscopy. We highlight the importance of selecting appropriate control methods based on various parameters, such as sensor and actuator solutions. This review aims to contribute to the development of smart control strategies for soft endorobots that can enhance the effectiveness and safety of robotics in colonoscopy. These strategies can be defined based on the available information about the robot and surrounding environment, control demands, mechanical design impact and characterization data based on calibration.<br/

Jacobiano de la Imagen para un Par Estereoscópico de Cámaras: Comparativa entre el Analítico y el Estimado Incorporando la Geometría Epipolar

El presente artículo describe un estudio comparativo entre el comportamiento del Jacobiano calculado de forma analítica y el estimado, incorporando la geometría epipolar, cuando se utiliza un sistema con dos cámaras. La arquitectura de control utilizada es Control Visual Basado en Imagen controlándose un sistema articular de tres g.d.l. utilizando dos cámaras en configuración cámara fija. Se realizaron pruebas tanto en caso estático como en caso dinámico, las cuales mostraron que el método de estimación del Jacobiano que utiliza las propiedades de la geometría epipolar es tan bueno y robusto frente al ruido como el Jacobiano analítico. Esto es considerado como una ventaja, puesto que el Jacobiano estimado no necesita un laborioso trabajo previo como el analítico