Recognising the Clothing Categories from Free-Configuration Using Gaussian-Process-Based Interactive Perception

In this paper, we propose a Gaussian Process- based interactive perception approach for recognising highly- wrinkled clothes. We have integrated this recognition method within a clothes sorting pipeline for the pre-washing stage of an autonomous laundering process. Our approach differs from reported clothing manipulation approaches by allowing the robot to update its perception confidence via numerous interactions with the garments. The classifiers predominantly reported in clothing perception (e.g. SVM, Random Forest) studies do not provide true classification probabilities, due to their inherent structure. In contrast, probabilistic classifiers (of which the Gaussian Process is a popular example) are able to provide predictive probabilities. In our approach, we employ a multi-class Gaussian Process classification using the Laplace approximation for posterior inference and optimising hyper-parameters via marginal likelihood maximisation. Our experimental results show that our approach is able to recognise unknown garments from highly-occluded and wrinkled con- figurations and demonstrates a substantial improvement over non-interactive perception approaches

双腕アームロボットによる布被覆作業に関する研究

本研究の目的は，物体を布で包む作業（被覆作業）をモデル化し，ロボットによる被覆作業を実現させることである．本論文では「目標線」の概念に基づいて物体を布で包む作業（被覆作業）をモデル化することを提案した．これにより，まず人間が大まかな包み方を教示し，次に布と物体の形状から被覆作業を計画し，最終的にロボットの動作を生成し，ロボットによる被覆作業を実現した．近年，工場のロボット化が行われているが，ロボット化できない作業はまだまだ存在している．それらは，人間にしか行えないような巧みで複雑な作業，あるいは，ロボットより人間の方が効率的にできてしまうような作業である．そのような作業の１つとして，布を扱う作業が挙げられる．布を扱う作業の中には，布単体だけでなく，物体も一緒に取り扱っていく被覆作業が多く存在している．しかし，この被覆作業をロボットに指示するための有効な作業モデルは確立されていない．先行研究では，ロボットによる布操作の記述方法として，点，折り線や手先経路が用いられている．また，コンピュータグラフィクス分野では目標線という記述方法があり，これは被覆を表現するために用いられている．被覆作業をロボット化する上では，まず，実世界のロボットのために，汎用的な被覆モデルとして必要となる物体と布の関係や作業手順を，どのように記述すればいいのかという問題に直面する．このような点を考慮し，被覆作業に適した記述モデルを導入しなければならない．次に，そのような被覆のための作業記述を，実際のロボットにどのように入力すればいいのかという問題がある．煩雑な指示方法ではなく，実空間上で人間が考えている被覆作業を，直感的にロボットに指示できるのが望ましい．最後に，その作業記述から実際のロボットの動きをどのように生成すればよいのかという問題が現れてくる．ロボットが被覆作業を達成するためには，実際の手先軌道や干渉を回避するための動作を，状況に合わせて生成しなければならない．以上を踏まえて，本研究ではロボットによる被覆作業の課題に取り組んだ．具体的には以下の課題について取り組んだ．・布と物体の関係を適切に表す記述方法・直感的な被覆手順の指示方法・ロボットの動作軌道の生成方法まず，布と物体の関係を適切に表す記述方法について検討した．本研究では，コンピュータグラフィクス分野で用いられた目標線という記述方法を，実空間のロボットに導入することを提案した．この目標線は平面だけでなく曲面形状への指示が行いやすい．そして，物体のどこを布で包んでいくかという被覆の本質的な情報を自然に表せる利点を持つ．その中では，凹凸が存在するような物体に対しても被覆を行う場合があり，その凹凸を適切に処理して，作業を記述する必要がある．そこで，物体の埋めるべき凹部と埋めるべきでない凹部分を考慮し，凹凸へ適切な目標線指示を行うための局所凸という概念，及び局所凸生成方法を提案した．次に，直感的な被覆手順の指示方法について検討した．本研究では，人間の大まかな包む指示と被覆の関係を考え，物体と布のどこを重ね合わせるかという人間の被覆の意図を目標線として入力する方法を提案した．本研究は，作業指示を行う手の正確な３次元的な軌跡ではなく，手の軌跡とその軌跡が通過していく物体表面の関係に注目した．そして，デプスセンサとモーションキャプチャセンサを組合せた教示デバイスを用いて，人間の被覆の意図を抽出した．その中では，指示中の手振れの影響を小さくするための目標線逆走防止処理手法とスムージングと間引き処理を合わせた補正処理手法を提案した．最後に，ロボットの動作軌道の生成方法について検討した．本研究では，目標線と把持点から布の動きを表す手先経路を生成する方法と，その手先経路を実行するためのロボット動作の生成方法を提案した．実際のロボットを動かすためには，目標線だけでなく，手先経路や動作指令が必要であり，可動域や物体との干渉を考慮し，右手と左手を用いた布の持ち替えや持ち直しを行わなければならない．これらの情報を生成する上で，目標線が被覆の本質的な情報を保持している．そのため，手先経路・動作指令は自動的に生成可能である．動作生成手法の中では，各操作の布への重力の影響，動作ステップ数やロボットと布の位置関係を考慮した確実性を求め，それを基に生成された動作遷移グラフを用いて，最適な持ち替えや持ち直し操作の組み合わせを計画する方法を提案した．以上，本研究では，物体を布で包むという被覆作業について，ロボット化のための枠組みを提案した．さらに，各課題に対する提案方法を統合し，一連の被覆作業システムとして実装した．これにより，実際に人間の大まかな指示から，目標線を用いて布と物体の関係を記述し，そこから布の動きを表す手先経路，状況に合わせた最適なロボット動作を生成できるようになりロボットによる被覆作業が実現した．電気通信大学201

Visual grasp point localization, classification and state recognition in robotic manipulation of cloth: an overview

© . This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/Cloth manipulation by robots is gaining popularity among researchers because of its relevance, mainly (but not only) in domestic and assistive robotics. The required science and technologies begin to be ripe for the challenges posed by the manipulation of soft materials, and many contributions have appeared in the last years. This survey provides a systematic review of existing techniques for the basic perceptual tasks of grasp point localization, state estimation and classification of cloth items, from the perspective of their manipulation by robots. This choice is grounded on the fact that any manipulative action requires to instruct the robot where to grasp, and most garment handling activities depend on the correct recognition of the type to which the particular cloth item belongs and its state. The high inter- and intraclass variability of garments, the continuous nature of the possible deformations of cloth and the evident difficulties in predicting their localization and extension on the garment piece are challenges that have encouraged the researchers to provide a plethora of methods to confront such problems, with some promising results. The present review constitutes for the first time an effort in furnishing a structured framework of these works, with the aim of helping future contributors to gain both insight and perspective on the subjectPeer ReviewedPostprint (author's final draft

Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems

The goal of this research is to enable a robotic system to manipulate clothing and other highly non-rigid objects using an RGBD sensor. The focus of this thesis is to define and test various algorithms / models that are used to solve parts of the laundry process (i.e. handling, classifying, sorting, unfolding, and folding). First, a system is presented for automatically extracting and classifying items in a pile of laundry. Using only visual sensors, the robot identifies and extracts items sequentially from the pile. When an item is removed and isolated, a model is captured of the shape and appearance of the object, which is then compared against a dataset of known items. The contributions of this part of the laundry process are a novel method for extracting articles of clothing from a pile of laundry, a novel method of classifying clothing using interactive perception, and a multi-layer approach termed L-M-H, more specifically L-C-S-H for clothing classification. This thesis describes two different approaches to classify clothing into categories. The first approach relies upon silhouettes, edges, and other low-level image measurements of the articles of clothing. Experiments from the first approach demonstrate the ability of the system to efficiently classify and label into one of six categories (pants, shorts, short-sleeve shirt, long-sleeve shirt, socks, or underwear). These results show that, on average, classification rates using robot interaction are 59% higher than those that do not use interaction. The second approach relies upon color, texture, shape, and edge information from 2D and 3D data within a local and global perspective. The multi-layer approach compartmentalizes the problem into a high (H) layer, multiple mid-level (characteristics(C), selection masks(S)) layers, and a low (L) layer. This approach produces \u27local\u27 solutions to solve the global classification problem. Experiments demonstrate the ability of the system to efficiently classify each article of clothing into one of seven categories (pants, shorts, shirts, socks, dresses, cloths, or jackets). The results presented in this paper show that, on average, the classification rates improve by +27.47% for three categories, +17.90% for four categories, and +10.35% for seven categories over the baseline system, using support vector machines. Second, an algorithm is presented for automatically unfolding a piece of clothing. A piece of cloth is pulled in different directions at various points of the cloth in order to flatten the cloth. The features of the cloth are extracted and calculated to determine a valid location and orientation in which to interact with it. The features include the peak region, corner locations, and continuity / discontinuity of the cloth. In this thesis, a two-stage algorithm is presented, introducing a novel solution to the unfolding / flattening problem using interactive perception. Simulations using 3D simulation software, and experiments with robot hardware demonstrate the ability of the algorithm to flatten pieces of laundry using different starting configurations. These results show that, at most, the algorithm flattens out a piece of cloth from 11.1% to 95.6% of the canonical configuration. Third, an energy minimization algorithm is presented that is designed to estimate the configuration of a deformable object. This approach utilizes an RGBD image to calculate feature correspondence (using SURF features), depth values, and boundary locations. Input from a Kinect sensor is used to segment the deformable surface from the background using an alpha-beta swap algorithm. Using this segmentation, the system creates an initial mesh model without prior information of the surface geometry, and it reinitializes the configuration of the mesh model after a loss of input data. This approach is able to handle in-plane rotation, out-of-plane rotation, and varying changes in translation and scale. Results display the proposed algorithm over a dataset consisting of seven shirts, two pairs of shorts, two posters, and a pair of pants. The current approach is compared using a simulated shirt model in order to calculate the mean square error of the distance from the vertices on the mesh model to the ground truth, provided by the simulation model

Flexible Object Manipulation

Flexible objects are a challenge to manipulate. Their motions are hard to predict, and the high number of degrees of freedom makes sensing, control, and planning difficult. Additionally, they have more complex friction and contact issues than rigid bodies, and they may stretch and compress. In this thesis, I explore two major types of flexible materials: cloth and string. For rigid bodies, one of the most basic problems in manipulation is the development of immobilizing grasps. The same problem exists for flexible objects. I have shown that a simple polygonal piece of cloth can be fully immobilized by grasping all convex vertices and no more than one third of the concave vertices. I also explored simple manipulation methods that make use of gravity to reduce the number of fingers necessary for grasping. I have built a system for folding a T-shirt using a 4 DOF arm and a fixed-length iron bar which simulates two fingers. The main goal with string manipulation has been to tie knots without the use of any sensing. I have developed single-piece fixtures capable of tying knots in fishing line, solder, and wire, along with a more complex track-based system for autonomously tying a knot in steel wire. I have also developed a series of different fixtures that use compressed air to tie knots in string. Additionally, I have designed four-piece fixtures, which demonstrate a way to fully enclose a knot during the insertion process, while guaranteeing that extraction will always succeed