Towards Assistive Feeding with a General-Purpose Mobile Manipulator

General-purpose mobile manipulators have the potential to serve as a versatile form of assistive technology. However, their complexity creates challenges, including the risk of being too difficult to use. We present a proof-of-concept robotic system for assistive feeding that consists of a Willow Garage PR2, a high-level web-based interface, and specialized autonomous behaviors for scooping and feeding yogurt. As a step towards use by people with disabilities, we evaluated our system with 5 able-bodied participants. All 5 successfully ate yogurt using the system and reported high rates of success for the system's autonomous behaviors. Also, Henry Evans, a person with severe quadriplegia, operated the system remotely to feed an able-bodied person. In general, people who operated the system reported that it was easy to use, including Henry. The feeding system also incorporates corrective actions designed to be triggered either autonomously or by the user. In an offline evaluation using data collected with the feeding system, a new version of our multimodal anomaly detection system outperformed prior versions.Comment: This short 4-page paper was accepted and presented as a poster on May. 16, 2016 in ICRA 2016 workshop on 'Human-Robot Interfaces for Enhanced Physical Interactions' organized by Arash Ajoudani, Barkan Ugurlu, Panagiotis Artemiadis, Jun Morimoto. It was peer reviewed by one reviewe

Investigating pre-touch sensing to predict grip success in compliant grippers using machine learning techniques

This work explores the application of pre-touch sensing to a compliant gripper in order to navigate the last few centimeters while grasping fruit in an occluded, cluttered environment. Machine learning was used in conjunction with pre-touch sensors to provide qualitative feedback about the success of the gripper in picking the target fruit prior to contact. Three compliant grippers were each designed to pick a specific fruit (miracle berries, cherry tomatoes and small figs) without damaging them. These grippers were designed to be mounted on the hybrid soft-rigid arm of a mobile field robot. An IR reflectance, time of flight and color sensor were used as pre-touch sensors and arranged on the gripper in various combinations to explore the contribution of each sensor. The gripper-sensor system was trained by positioning it relative to a dummy fruit using a 6 DOF arm and gripping the target. Using the training data, five machine learning methods were explored: nearest neighbor, decision trees, support vector machines, multi-layer perceptrons and a naive Bayes classifier. The various sensor configuration-machine learning combinations were tested and evaluated based on their ability to predict grip success. Additional training was conducted to demonstrate the ability to differentiate fruit from foreign matter (e.g. leaves) that are in the gripper opening. Time of flight sensors using nearest neighbor and support vector machines along with the set of all three sensors using support vector machines and multi-layer perceptrons showed the highest prediction precision (= 90%) with the color sensor playing a key role in detecting foreign objects. The machine learning methods were similar in their ability to predict grip success with nearest neighbor showing the best overall results, while sensor ‘richness’ play an important role in differentiating the sensors with the three sensor combination showing the best results

지도 및 비지도 학습을 이용한 로봇 머니퓰레이터 충돌 감지

학위논문(박사) -- 서울대학교대학원 : 공과대학 기계항공공학부, 2022.2. 박종우.사람과 공유된 구조화되지 않은 동적 환경에서 작동하는 협업 로봇 머니퓰레이터는 날카로운 충돌(경성 충돌)에서 더 긴 지속 시간의 밀고 당기는 동작(연성 충돌)에 이르기까지의 다양한 충돌을 빠르고 정확하게 감지해야 한다. 모터 전류 측정값을 이용해 외부 조인트 토크를 추정하는 동역학 모델 기반 감지 방법을 사용할 경우, 정확한 마찰 파라미터 모델링 및 식별과 같은 모터 마찰에 대한 적절한 처리가 필요하다. 올바르게 적용하면 매우 효과적이지만, 동역학과 마찰 파라미터를 모델링 및 식별하고 여러 개의 감지 임계값을 수동으로 설정하는 데에는 상당한 노력이 필요하기 때문에 대량 생산되는 산업용 로봇에 이를 적용하기는 어렵다. 또한 적절한 식별 후에도 동역학에 백래시, 탄성 등 모델링되지 않은 효과나 불확실성이 여전히 존재할 수 있다. 본 논문에서는 순수 모델 기반 방법의 구현 어려움을 피하고 불확실한 동역학적 효과를 보상하는 수단으로 로봇 머니퓰레이터를 위한 총 네 가지의 학습 기반 충돌 감지 방법을 제안한다. 두 개의 방법은 학습을 위해 충돌 및 비충돌 동작 데이터가 모두 필요한 지도 학습 알고리즘(서포트 벡터 머신 회귀, 일차원 합성곱 신경망 기반)을 사용하며 나머지 두 개의 방법은 학습을 위해 비충돌 동작 데이터만을 필요로 하는 비지도 이상치 탐지 알고리즘(단일 클래스 서포트 벡터 머신, 오토인코더 기반)에 기반한다. 로봇 동역학 모델과 모터 전류 측정값만을 필요로 하며 추가적인 외부 센서나 마찰 모델링, 여러 개의 감지 임계값에 대한 수동 조정은 필요하지 않다. 먼저 지도 및 비지도 감지 방법을 학습시키고 검증하는 데 사용되는, 6자유도 협업 로봇 머니퓰레이터를 이용해 수집된 로봇 충돌 데이터를 설명한다. 우리가 고려하는 충돌 시나리오는 경성 충돌, 연성 충돌, 비충돌 동작으로, 경성 및 연성 충돌은 모두 동일하게 충돌로 간주한다. 감지 성능 검증을 위한 테스트 데이터는 총 787건의 충돌과 62.4분의 비충돌 동작으로 이루어져 있으며, 이는 로봇이 랜덤 점대점 6관절 동작을 수행하는 동안 수집된다. 데이터 수집 중 로봇의 끝단에는 미부착, 3.3 kg, 5.0 kg의 세 가지 유형의 페이로드를 부착한다. 다음으로, 수집된 테스트 데이터를 이용해 지도 감지 방법의 감지 성능을 실험적으로 검증한다. 실험 결과는 지도 감지 방법이 가벼운 네트워크를 이용해 광범위한 경성 및 연성 충돌을 실시간으로 정확하게 감지할 수 있음을 보여주며, 이를 통해 모델 파라미터의 불확실성과 측정 노이즈, 백래시, 변형 등 모델링되지 않은 효과까지 보상됨을 알 수 있다. 또한 서포트 벡터 머신 회귀 기반 방법은 하나의 감지 임계값에 대한 조정만 필요하며 일차원 합성곱 신경망 기반 방법은 하나의 아웃풋 필터 파라미터에 대한 조정만 필요한데, 두 방법 모두 직관적인 감도 조정이 가능하다. 나아가 일련의 시뮬레이션 실험을 통해 지도 감지 방법의 일반화 성능을 실험적으로 검증한다. 마지막으로, 동일한 테스트 데이터에 대해 비지도 감지 방법의 감지 성능과 일반화 성능 또한 검증한다. 실험 결과는 비지도 감지 방법 또한 가벼운 계산과 하나의 감지 임계값에 대한 조정만으로 다양한 경성 및 연성 충돌을 실시간으로 강인하게 감지할 수 있음을 보여주며, 이를 통해 모델링되지 않은 마찰을 포함한 불확실한 동역학적 효과를 비지도 학습으로도 보상할 수 있음을 알 수 있다. 지도 감지 방법이 더 나은 감지 성능을 보이지만, 비지도 감지 방법은 학습을 위해 비충돌 동작 데이터만을 필요로 하며 발생할 수 있는 모든 유형의 충돌에 대한 정보를 필요로 하지 않기 때문에 대량 생산되는 산업용 로봇에 더 적합하다.Collaborative robot manipulators operating in dynamic and unstructured environments shared with humans require fast and accurate detection of collisions, which can range from sharp impacts (hard collisions) to pulling and pushing motions of longer duration (soft collisions). When using dynamics model-based detection methods that estimate the external joint torque with motor current measurements, proper treatment for friction in the motors is required, such as accurate modeling and identification of friction parameters. Although highly effective when done correctly, modeling and identifying the dynamics and friction parameters, and manually setting multiple detection thresholds require considerable effort, making them difficult to be replicated for mass-produced industrial robots. There may also still exist unmodeled effects or uncertainties in the dynamics even after proper identification, e.g., backlash, elasticity. This dissertation presents a total of four learning-based collision detection methods for robot manipulators as a means of sidestepping some of the implementation difficulties of pure model-based methods and compensating for uncertain dynamic effects. Two methods use supervised learning algorithms – support vector machine regression and a one-dimensional convolutional neural network-based – that require both the collision and collision-free motion data for training. The other two methods are based on unsupervised anomaly detection algorithms – a one-class support vector machine and an autoencoder-based – that require only the collision-free motion data for training. Only the motor current measurements together with a robot dynamics model are required while no additional external sensors, friction modeling, or manual tuning of multiple detection thresholds are needed. We first describe the robot collision dataset collected with a six-dof collaborative robot manipulator, which is used for training and validating our supervised and unsupervised detection methods. The collision scenarios we consider are hard collisions, soft collisions, and collision-free, where both hard and soft collisions are treated in the same manner as just collisions. The test dataset for detection performance verification includes a total of 787 collisions and 62.4 minutes of collision-free motions, all collected while the robot is executing random point-to-point six-joint motions. During data collection, three types of payloads are attached to the end-effector: no payload, 3.3 kg payload, and 5.0 kg payload. Then the detection performance of our supervised detection methods is experimentally verified with the collected test dataset. Results demonstrate that our supervised detection methods can accurately detect a wide range of hard and soft collisions in real-time using a light network, compensating for uncertainties in the model parameters as well as unmodeled effects like friction, measurement noise, backlash, and deformations. Moreover, the SVMR-based method requires only one constant detection threshold to be tuned while the 1-D CNN-based method requires only one output filter parameter to be tuned, both of which allow intuitive sensitivity tuning. Furthermore, the generalization capability of our supervised detection methods is experimentally verified with a set of simulation experiments. Finally, our unsupervised detection methods are also validated for the same test dataset; the detection performance and the generalization capability are verified. The experimental results show that our unsupervised detection methods are also able to robustly detect a variety of hard and soft collisions in real-time with very light computation and with only one constant detection threshold required to be tuned, validating that uncertain dynamic effects including the unmodeled friction can be successfully compensated also with unsupervised learning. Although our supervised detection methods show better detection performance, our unsupervised detection methods are more practical for mass-produced industrial robots since they require only the data for collision-free motions for training, and the knowledge of every possible type of collision that can occur is not required.1 Introduction 1 1.1 Model-Free Methods 2 1.2 Model-Based Methods 2 1.3 Learning-Based Methods 4 1.3.1 Using Supervised Learning Algorithms 5 1.3.2 Using Unsupervised Learning Algorithms 6 1.4 Contributions of This Dissertation 7 1.4.1 Supervised Learning-Based Model-Compensating Detection 7 1.4.2 Unsupervised Learning-Based Model-Compensating Detection 8 1.4.3 Comparison with Existing Detection Methods 9 1.5 Organization of This Dissertation 14 2 Preliminaries 17 2.1 Introduction 17 2.2 Robot Dynamics 17 2.3 Momentum Observer-Based Collision Detection 19 2.4 Supervised Learning Algorithms 21 2.4.1 Support Vector Machine Regression 21 2.4.2 One-Dimensional Convolutional Neural Network 23 2.5 Unsupervised Anomaly Detection 25 2.6 One-Class Support Vector Machine 26 2.7 Autoencoder-Based Anomaly Detection 28 2.7.1 Autoencoder Network Architecture and Training 28 2.7.2 Anomaly Detection Using Autoencoders 29 3 Robot Collision Data 31 3.1 Introduction 31 3.2 True Collision Index Labeling 31 3.3 Collision Scenarios 35 3.4 Monitoring Signal 36 3.5 Signal Normalization and Sampling 37 3.6 Test Data for Detection Performance Verification 39 4 Supervised Learning-Based Model-Compensating Detection 43 4.1 Introduction 43 4.2 SVMR-Based Collision Detection 44 4.2.1 Input Feature Vector Design 44 4.2.2 SVMR Training 45 4.2.3 Collision Detection Sensitivity Adjustment 46 4.3 1-D CNN-Based Collision Detection 50 4.3.1 Network Input Design 50 4.3.2 Network Architecture and Training 50 4.3.3 An Output Filtering Method to Reduce False Alarms 53 4.4 Collision Detection Performance Criteria 54 4.4.1 Area Under the Precision-Recall Curve (PRAUC) 54 4.4.2 Detection Delay and Number of Detection Failures 54 4.5 Collision Detection Performance Analysis 56 4.5.1 Global Performance with Varying Thresholds 56 4.5.2 Detection Delay and Number of Detection Failures 57 4.5.3 Real-Time Inference 60 4.6 Generalization Capability Analysis 60 4.6.1 Generalization to Small Perturbations 60 4.6.2 Generalization to an Unseen Payload 62 5 Unsupervised Learning-Based Model-Compensating Detection 67 5.1 Introduction 67 5.2 OC-SVM-Based Collision Detection 68 5.2.1 Input Feature Vector 68 5.2.2 OC-SVM Training 70 5.2.3 Collision Detection with the Trained OC-SVM 70 5.3 Autoencoder-Based Collision Detection 70 5.3.1 Network Input and Output 71 5.3.2 Network Architecture and Training 71 5.3.3 Collision Detection with the Trained Autoencoder 72 5.4 Collision Detection Performance Analysis 74 5.4.1 Global Performance with Varying Thresholds 75 5.4.2 Detection Delay and Number of Detection Failures 75 5.4.3 Comparison with Supervised Learning-Based Methods 80 5.4.4 Real-Time Inference 83 5.5 Generalization Capability Analysis 83 5.5.1 Generalization to Small Perturbations 84 5.5.2 Generalization to an Unseen Payload 85 6 Conclusion 89 6.1 Summary and Discussion 89 6.2 Future Work 93 A Appendix 95 A.1 SVM-Based Classification of Detected Collisions 95 A.2 Direct Estimation-Based Detection Methods 97 A.3 Model-Independent Supervised Detection Methods 101 A.4 Generalization to Large Changes in the Dynamics Model 102 Bibliography 106 Abstract 112박

Intelligent strategies for mobile robotics in laboratory automation

In this thesis a new intelligent framework is presented for the mobile robots in laboratory automation, which includes: a new multi-floor indoor navigation method is presented and an intelligent multi-floor path planning is proposed; a new signal filtering method is presented for the robots to forecast their indoor coordinates; a new human feature based strategy is proposed for the robot-human smart collision avoidance; a new robot power forecasting method is proposed to decide a distributed transportation task; a new blind approach is presented for the arm manipulations for the robots

automatic process modeling with time delay neural network based on low level data

n/

FBG-Based Triaxial Force Sensor Integrated with an Eccentrically Configured Imaging Probe for Endoluminal Optical Biopsy

Accurate force sensing is important for endoluminal intervention in terms of both safety and lesion targeting. This paper develops an FBG-based force sensor for robotic bronchoscopy by configuring three FBG sensors at the lateral side of a conical substrate. It allows a large and eccentric inner lumen for the interventional instrument, enabling a flexible imaging probe inside to perform optical biopsy. The force sensor is embodied with a laser-profiled continuum robot and thermo drift is fully compensated by three temperature sensors integrated on the circumference surface of the sensor substrate. Different decoupling approaches are investigated, and nonlinear decoupling is adopted based on the cross-validation SVM and a Gaussian kernel function, achieving an accuracy of 10.58 mN, 14.57 mN and 26.32 mN along X, Y and Z axis, respectively. The tissue test is also investigated to further demonstrate the feasibility of the developed triaxial force senso