LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

This paper presents a novel approach to enhance autonomous robotic manipulation using the Large Language Model (LLM) for logical inference, converting high-level language commands into sequences of executable motion functions. The proposed system combines the advantage of LLM with YOLO-based environmental perception to enable robots to autonomously make reasonable decisions and task planning based on the given commands. Additionally, to address the potential inaccuracies or illogical actions arising from LLM, a combination of teleoperation and Dynamic Movement Primitives (DMP) is employed for action correction. This integration aims to improve the practicality and generalizability of the LLM-based human-robot collaboration system.Comment: IEEE MHS 202

Securing Reliability: A Brief Overview on Enhancing In-Context Learning for Foundation Models

As foundation models (FMs) continue to shape the landscape of AI, the in-context learning (ICL) paradigm thrives but also encounters issues such as toxicity, hallucination, disparity, adversarial vulnerability, and inconsistency. Ensuring the reliability and responsibility of FMs is crucial for the sustainable development of the AI ecosystem. In this concise overview, we investigate recent advancements in enhancing the reliability and trustworthiness of FMs within ICL frameworks, focusing on four key methodologies, each with its corresponding subgoals. We sincerely hope this paper can provide valuable insights for researchers and practitioners endeavoring to build safe and dependable FMs and foster a stable and consistent ICL environment, thereby unlocking their vast potential.Comment: 18 pages, 15 figure

Visual Tactile Sensor Based Force Estimation for Position-Force Teleoperation

Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and applying a regression algorithm on estimated depth data and ground truth force data to get the depth-force relationship. The experiment is performed by constructing a grasping force feedback system with a haptic device as a leader robot and a parallel robot gripper as a follower robot, where the DIGIT sensor is attached to the tip of the robot gripper to estimate the contact force. The preliminary results show the capability of using the low-cost vision-based sensor for force feedback applications.Comment: IEEE CBS 202

Embodiment Based Sensory Feedback for Human Control Over Robotic Systems

doctoral thesi

Intelligent Detection of Parcels Based on Improved Faster R-CNN

Parcel detection is crucial to achieving automatic sorting in intelligent logistics systems. Most parcels in logistics centers are currently detected manually, imposing low efficiency and high error rate, severely limiting logistics transportation efficiency. Therefore, there is an urgent need for automated parcel detection. However, parcels in logistics centers have challenges such as dense stacking, occlusion and background interference, making it difficult for existing methods to detect parcels accurately. To address the above problem, we developed an improved Faster R-CNN-based parcel detection model spurred by current deep-learning-based object detection trends. The proposed method first solves the false detection problem due to parcel mutual occlusion by augmenting Faster R-CNN with an edge detection branch and adding object edge loss to the loss function. Furthermore, the self-attention ROI Align module is proposed to address the problem of feature misalignment caused by the quantization rounding operation in the ROI Pooling module. The module uses an attention mechanism to filter and enhance the features and uses bilinear interpolation to calculate the feature pixel values, improving detection accuracy. The implementation of the proposed method was validated using parcel images collected in the field and the public dataset SKU110K and compared with four existing parcel detection methods. The results show that our method’s Recall, Precision, [email protected] and Fps are 96.89%, 98.76%, 98.42% and 22.83%, respectively, which significantly improves the parcel detection accuracy.</jats:p