A systems engineering approach to robotic bin picking

In recent times the presence of vision and robotic systems in industry has become common place, but in spite of many achievements a large range of industrial tasks still remain unsolved due to the lack of flexibility of the vision systems when dealing with highly adaptive manufacturing environments. An important task found across a broad range of modern flexible manufacturing environments is the need to present parts to automated machinery from a supply bin. In order to carry out grasping and manipulation operations safely and efficiently we need to know the identity, location and spatial orientation of the objects that lie in an unstructured heap in a bin. Historically, the bin picking problem was tackled using mechanical vibratory feeders where the vision feedback was unavailable. This solution has certain problems with parts jamming and more important they are highly dedicated. In this regard if a change in the manufacturing process is required, the changeover may include an extensive re-tooling and a total revision of the system control strategy (Kelley et al., 1982). Due to these disadvantages modern bin picking systems perform grasping and manipulation operations using vision feedback (Yoshimi & Allen, 1994). Vision based robotic bin picking has been the subject of research since the introduction of the automated vision controlled processes in industry and a review of existing systems indicates that none of the proposed solutions were able to solve this classic vision problem in its generality. One of the main challenges facing such a bin picking system is its ability to deal with overlapping objects. The object recognition in cluttered scenes is the main objective of these systems and early approaches attempted to perform bin picking operations for similar objects that are jumbled together in an unstructured heap using no knowledge about the pose or geometry of the parts (Birk et al., 1981). While these assumptions may be acceptable for a restricted number of applications, in most practical cases a flexible system must deal with more than one type of object with a wide scale of shapes. A flexible bin picking system has to address three difficult problems: scene interpretation, object recognition and pose estimation. Initial approaches to these tasks were based on modeling parts using the 2D surface representations. Typical 2D representations include invariant shape descriptors (Zisserman et al., 1994), algebraic curves (Tarel & Cooper, 2000), 2 Name of the book (Header position 1,5) conics (Bolles & Horaud, 1986; Forsyth et al., 1991) and appearance based models (Murase & Nayar, 1995; Ohba & Ikeuchi, 1997). These systems are generally better suited to planar object recognition and they are not able to deal with severe viewpoint distortions or objects with complex shapes/textures. Also the spatial orientation cannot be robustly estimated for objects with free-form contours. To address this limitation most bin picking systems attempt to recognize the scene objects and estimate their spatial orientation using the 3D information (Fan et al., 1989; Faugeras & Hebert, 1986). Notable approaches include the use of 3D local descriptors (Ansar & Daniilidis, 2003; Campbell & Flynn, 2001; Kim & Kak, 1991), polyhedra (Rothwell & Stern, 1996), generalized cylinders (Ponce et al., 1989; Zerroug & Nevatia, 1996), super-quadrics (Blane et al., 2000) and visual learning methods (Johnson & Hebert, 1999; Mittrapiyanuruk et al., 2004). The most difficult problem for 3D bin picking systems that are based on a structural description of the objects (local descriptors or 3D primitives) is the complex procedure required to perform the scene to model feature matching. This procedure is usually based on complex graph-searching techniques and is increasingly more difficult when dealing with object occlusions, a situation when the structural description of the scene objects is incomplete. Visual learning methods based on eigenimage analysis have been proposed as an alternative solution to address the object recognition and pose estimation for objects with complex appearances. In this regard, Johnson and Hebert (Johnson & Hebert, 1999) developed an object recognition scheme that is able to identify multiple 3D objects in scenes affected by clutter and occlusion. They proposed an eigenimage analysis approach that is applied to match surface points using the spin image representation. The main attraction of this approach resides in the use of spin images that are local surface descriptors; hence they can be easily identified in real scenes that contain clutter and occlusions. This approach returns accurate results but the pose estimation cannot be inferred, as the spin images are local descriptors and they are not robust to capture the object orientation. In general the pose sampling for visual learning methods is a problem difficult to solve as the numbers of views required to sample the full 6 degree of freedom for object pose is prohibitive. This issue was addressed in the paper by Edwards (Edwards, 1996) when he applied eigenimage analysis to a one-object scene and his approach was able to estimate the pose only in cases where the tilt angle was limited to 30 degrees with respect to the optical axis of the sensor. In this chapter we describe the implementation of a vision sensor for robotic bin picking where we attempt to eliminate the main problem faced by the visual learning methods, namely the pose sampling problem. This paper is organized as follows. Section 2 outlines the overall system. Section 3 describes the implementation of the range sensor while Section 4 details the edge-based segmentation algorithm. Section 5 presents the viewpoint correction algorithm that is applied to align the detected object surfaces perpendicular on the optical axis of the sensor. Section 6 describes the object recognition algorithm. This is followed in Section 7 by an outline of the pose estimation algorithm. Section 8 presents a number of experimental results illustrating the benefits of the approach outlined in this chapter

Image morphological processing

Mathematical Morphology with applications in image processing and analysis has been becoming increasingly important in today\u27s technology. Mathematical Morphological operations, which are based on set theory, can extract object features by suitably shaped structuring elements. Mathematical Morphological filters are combinations of morphological operations that transform an image into a quantitative description of its geometrical structure based on structuring elements. Important applications of morphological operations are shape description, shape recognition, nonlinear filtering, industrial parts inspection, and medical image processing. In this dissertation, basic morphological operations, properties and fuzzy morphology are reviewed. Existing techniques for solving corner and edge detection are presented. A new approach to solve corner detection using regulated mathematical morphology is presented and is shown that it is more efficient in binary images than the existing mathematical morphology based asymmetric closing for corner detection. A new class of morphological operations called sweep mathematical morphological operations is developed. The theoretical framework for representation, computation and analysis of sweep morphology is presented. The basic sweep morphological operations, sweep dilation and sweep erosion, are defined and their properties are studied. It is shown that considering only the boundaries and performing operations on the boundaries can substantially reduce the computation. Various applications of this new class of morphological operations are discussed, including the blending of swept surfaces with deformations, image enhancement, edge linking and shortest path planning for rotating objects. Sweep mathematical morphology is an efficient tool for geometric modeling and representation. The sweep dilation/erosion provides a natural representation of sweep motion in the manufacturing processes. A set of grammatical rules that govern the generation of objects belonging to the same group are defined. Earley\u27s parser serves in the screening process to determine whether a pattern is a part of the language. Finally, summary and future research of this dissertation are provided

A Stochastic Modeling Approach to Region-and Edge-Based Image Segmentation

The purpose of image segmentation is to isolate objects in a scene from the background. This is a very important step in any computer vision system since various tasks, such as shape analysis and object recognition, require accurate image segmentation. Image segmentation can also produce tremendous data reduction. Edge-based and region-based segmentation have been examined and two new algorithms based on recent results in random field theory have been developed. The edge-based segmentation algorithm uses the pixel gray level intensity information to allocate object boundaries in two stages: edge enhancement, followed by edge linking. Edge enhancement is accomplished by maximum energy filters used in one-dimensional bandlimited signal analysis. The issue of optimum filter spatial support is analyzed for ideal edge models. Edge linking is performed by quantitative sequential search using the Stack algorithm. Two probabilistic search metrics are introduced and their optimality is proven and demonstrated on test as well as real scenes. Compared to other methods, this algorithm is shown to produce more accurate allocation of object boundaries. Region-based segmentation was modeled as a MAP estimation problem in which the actual (unknown) objects were estimated from the observed (known) image by a recursive classification algorithms. The observed image was modeled by an Autoregressive (AR) model whose parameters were estimated locally, and a Gibbs-Markov random field (GMRF) model was used to model the unknown scene. A computational study was conducted on images having various types of texture images. The issues of parameter estimation, neighborhood selection, and model orders were examined. It is concluded that the MAP approach for region segmentation generally works well on images having a large content of microtextures which can be properly modeled by both AR and GMRF models. On these texture images, second order AR and GMRF models were shown to be adequate

Detecção e agrupamento de contornos

A detecção de contornos a partir de imagens digitais é um procedimento do qual resulta informação essencial para muitos algoritmos de visão por computador. A natureza das imagens digitais bidimensionais: a sua relativamente baixa resolução; a amostragem espacial e em amplitude; a presença de ruído; a falta de informação em profundidade; as oclusões, etc., e a importância dos contornos como informação básica para muitos outros algoritmos a montante, fazem com que a detecção de contornos seja um problema apenas parcialmente resolvido, com múltiplas abordagens e dando origem desde há algumas décadas a larga quantidade de publicações. Continua a ser um tema actual de investigação como se comprova pela quantidade e qualidade das publicações científicas mais actuais nesta área. A tese discute a detecção de contornos nas suas fases clássicas: a estimação da amplitude do sinal que aponta a presença de um ponto de contorno; a pré-classificação dos pontos da imagem com base nos sinais estimados e o posterior agrupamento dos pontos de contorno individuais em segmentos de curvas de contorno. Propõe-se, nesta tese: um método de projecto de estimadores de presença de pontos de contorno baseado na utilização de equações integrais de Fredholm; um classificador não-linear que utiliza informação de pontos vizinhos para a tomada de decisão, e uma metodologia de agrupamento de pontos de contorno com crescimento iterativo com uma função de custo com suporte local. A metodologia de extracção das propriedades baseada na equação integral de Fredholm de primeira ordem permite uma análise unificadora de vários métodos previamente propostos na literatura sobre o assunto. O procedimento de classificação dos pontos de contorno baseia-se na análise das sequências ordenadas das amplitudes do gradiente na vizinhança do ponto de contorno. O procedimento é estudado com base nas funções densidade de distribuição das estatísticas ordenadas dos pontos de contorno vizinhos e na assunção de que os pontos de um mesmo contorno possuem distribuições ordenadas similares. A fase final da detecção de contornos é realizada com um procedimento de agrupamento de contornos em que se constrói uma hipótese de vizinhança para eventual crescimento do contorno e em que se estima o melhor ponto para agregação ao contorno. Os resultados experimentais para os métodos propostos são apresentados e analisados com imagens reais e sintéticas

Fahrspurschätzung aus monokularen Bildfolgen für innerstädtische Fahrerassistenzanwendungen

Die Lage der Fahrspur vor dem eigenen Fahrzeug ist eine elementare Basisinformation für viele Assistenzfunktionen. Diese Arbeit behandelt die Frage, wie diese Information aus monokularen Bildfolgen innerstädtischer Verkehrsszenarien gewonnen werden kann. Aus grundlegenden Abbildungseigenschaften der Fahrspur wird ein modellbasierter Schätzer hergeleitet und anschließend im Versuchsfahrzeug realisiert. Anhand von Testfahrten wird die Leistungsfähigkeit des Verfahrens gezeigt und bewertet

Feature detection in grayscale aerial images

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Civil and Environmental Engineering, 1999.Includes bibliographical references (p. 95-96).by Katherine Treash.S.M