We present visual servoing strategies based on bitangents for aligning planar shapes. In order to acquire bitangents we use convex-hull of a curve. Bitangent points are employed in the construction of a feature vector to be used in visual control. Experimental results obtained on a 7 DOF Mitsubishi PA10 robot, verifies the proposed method

Ozgur, Erol

Unel, Mustafa

Özgür, Erol

Ünel, Mustafa

English

Sabanci University Research Database

IMAGE BASED VISUAL SERVOING USING BITANGENT POINTS APPLIEDTO PLANAR SHAPE ALIGNMENTErol Ozgur and Mustafa UnelFaculty of Engineering and Natural SciencesSabanci UniversityOrhanli Tuzla 34956 Istanbul, Turkeyemail: erol@su.sabanciuniv.edu, munel@sabanciuniv.eduABSTRACTWe present visual servoing strategies based on bitangentsfor aligning planar shapes. In order to acquire bitangentswe use convex-hull of a curve. Bitangent points are em-ployed in the construction of a feature vector to be used invisual control. Experimental results obtained on a 7 DOFMitsubishi PA10 robot, verifies the proposed method.KEY WORDSVisual servoing, alignment, convex hull, bitangents1 IntroductionCurve alignment is a central problem in current research ar-eas and has played a key role in many particular domain ofapplications such as object recognition [1], [2] and tracking[3]. In the domain of visual servoing, most of the currentalignment systems are based on known geometrical shapedobjects such as industrial parts or those have good featureslike corners, straight edges which are feasible to extract andtrack in real time [4]. The alignment of smooth free-formplanar objects in unknown environments presents a chal-lenge in visually guided assembly tasks.In this paper we propose to use bitangent points inaligning planar curves by employing both calibrated [5]and uncalibrated image based visual servoing [6] schemes.In literature the use of bitangents in recognizing planar ob-jects by affine invariant alignment was first considered in[7] and for servoing purposes bitangent lines (lines joiningcorresponding features on the superposition of two viewsof a scene) were utilized to align the orientation betweentwo cameras at different locations in space by [8]. In [9]similar principles were applied on landing and surveillanceof aerial vehicles using vanishing points and lines. In orderto acquire bitangent points, we used convex-hull of a curve[10]. Bitangent points are then used in the construction ofa feature vector.The remainder of this paper is organized as follows:Section 2 presents bitangents of curves and how to ac-quire them. Section 3 reviews both model based and modelfree image based visual servoing, for calibrated and uncali-brated approaches. Section 4 is on experimental results forcurve alignment and discussions. Finally, Section 5 con-cludes the paper with some remarks.2 Bitangents of CurvesA line that is tangent to a curve at two points is called abitangent and the points of tangency are called bitangentpoints. See Fig.1. It is well known [11] that these bitangentFigure 1. Some curves and their bitangents.points directly map to one another under projective trans-formations. They are also called contact points.2.1 Computation of Bitangent PointsComputation of bitangent points of a curve is presented asa block diagram in Fig.2. Block-I receives a sequence ofimages from a camera and tracks a region in a specifiedwindow using a tracking algorithm such as ESM algorithm[12]. Block-II applies Canny edge detection algorithm tothe specified region and extracts the curve boundary data.Finally, Block-III employs a Convex-hull algorithm [10] tofind convex hull of the curve. Fig.3 depicts a curve with 3concavities. The convex-hull algorithm yields convex por-tion of the original data. Initial and final points of eachconvex portion are bitangent points.RegionTrackingCurveDetectionConvexHullBitangentPointsI II IIIImageSequenceFigure 2. Block diagram representation of the algorithmfor extracting bitangent points.(a) (b)Figure 3. (a) a curve and its convex-hull, (b) convex dataportions and the bitangent points3 Visual Servoing3.1 BackgroundLet θ ∈ℜn, s ∈ℜm and r ∈ℜ6 denote the vectors of jointvariables of a robot, image features obtained from visualsensors and the pose of the end-effector of the robot, re-spectively. The differential relation between θ and r withrespect to time impliesr˙ = JR(θ) ˙θ (1)where JR(θ) = ∂ r/∂θ ∈ℜ6×n is the robot Jacobian whichdescribes the relation between the robot joint velocities andthe velocities of its end-effector in Cartesian space. The re-lation between s and r is given as s = s(r) and its differen-tiation with respect to time yieldss˙ = JI(r)r˙ (2)where JI(r) = ∂ s/∂ r ∈ℜm×6 is the image Jacobian whichdescribes the differential relation of the image features, andpose of the robot end-effector, and the r˙ is the camera ve-locity screw (Vc). The composite Jacobian is defined asJ = JIJR (3)where J ∈ ℜm×n is a matrix which is the product of im-age and robot Jacobian. Thus, the relation between jointcoordinates and image features is given bys˙ = J ˙θ (4)3.2 Calibrated Visual ServoingLet s∗ ∈ℜm be the constant desired feature vector and de-fine the error e ∈ ℜm on image plane as e = s− s∗. Thenthe control problem can be formulated as follows: designan end-effector velocity screw u in such a way that the errordisappears, i.e. e→ 0.The image Jacobian of a single point feature vectors = [x,y]T for a fixed-camera system is given by:(x˙y˙)=( 1Z 0−xZ −xy (1+ x2) −y0 1Z−yZ −(1+ y2) xy x)︸ ︷︷ ︸JxyVc(5)wherex =xp− xcfx , y =yp− ycfy (6)and (xp,yp) are pixel coordinates of the image point and (xc,yc)are the coordinates of the principle point, and ( fx, fy) are effectivefocal lengths of the vision sensor, respectively.By rearranging and differentiating (6), and writing in matrixform, we get the following expression(x˙py˙p)=( fx 00 fy)(x˙y˙)(7)and substituting (5) into (7), the following equation appears(x˙py˙p)=( fx 00 fy)Jxy︸ ︷︷ ︸JIVc (8)s˙ = JIVc (9)where JI is the pixel-image Jacobian. In eye-to-hand case, the im-age jacobian has to consider the mapping from the camera frameonto the robot control frame. This relationship is given by therobot-to-camera transformation, denoted by:Vc = TVR (10)where VR is the end-effector velocity screw in robot control frame.The robot-to-camera velocity transformation T ∈ℜ6×6 is definedas belowT =(R [t]xR03 R)(11)where [R, t] are being rotational matrix and the translation vectorthat map camera frame onto robot control frame and [t]x is theskew symmetric matrix associated with vector t.Substituting (10) into (9), an expression that relates the im-age motion to the end-effector velocity is acquired:s˙ = JIT︸︷︷︸, ¯JIVR = ¯JIVR (12)where ¯JI is the new image Jacobian which directly relates thechanges of the image features to the end-effector velocity in robotcontrol frame. Note that if k feature points are taken into account,e.g. s = [x1,y1 . . .xk,yk]T, ¯JI is given by the following stackedimage Jacobian¯JI =¯J1I...¯JkI (13)By imposing e˙ = −Λe an exponential decrease of the errorfunction is realized. Solving (12), the control for end-effectormotion is obtained as follow:VR =− ¯J†I Λ(s− s∗) (14)where Λ ∈ ℜ6×6 is a positive constant gain matrix, ¯J†Iis the pseudo-inverse of the image Jacobian and VR =(Vx Vy Vz Ωx Ωy Ωz)T.3.3 Uncalibrated Visual ServoingHere the composite Jacobian is unknown and it has to be esti-mated dynamically. The error function in the image plane fora moving target at position s∗(t) and an end-effector at positions(θ) is given ase(θ , t) = s(θ)− s∗(t) (15)where s∗(t) represents desired image features at time t. The con-trol problem can be formulated as follows: design a controller thatcomputes the velocity of joint variables u in such a way that theerror disappears, i.e. e→ 0.3.3.1 Dynamic Jacobian EstimationSince the system model is assumed to be unknown, a recursiveleast-squares (RLS) algorithm [6] is used to estimate the compos-ite Jacobian J. This is accomplished by minimizing the followingcost function, which is a weighted sum of the changes in the affinemodel of error over time,εk =k−1∑i=0λ k−i−1‖∆mki‖2 (16)where∆mki = mk(θi, ti)−mi(θi, ti) (17)with mk(θ , t) being an expansion of m(θ , t), which is the affinemodel of the error function e(θ , t), about the kth data point asfollows:mk(θ , t) = e(θk, tk)+ ˆJk(θ −θk)+∂ek∂ t (t− tk) (18)In light of (18), (17) becomes∆mki = e(θk, tk)− e(θi, ti)−∂ek∂ t (tk− ti)−ˆJkhki, (19)where hki = θk − θi, the weighting factor λ satisfies 0 < λ < 1,and the unknown variables are the elements of ˆJk.Solution of the minimization problem yields the followingrecursive update rule for the composite Jacobian:ˆJk = ˆJk−1 +(∆e− ˆJk−1hθ −∂ek∂ t ht)(λ +hTθ Pk−1hθ )−1hTθ Pk−1(20)wherePk =1λ (Pk−1−Pk−1hθ (λ +hTθ Pk−1hθ )−1hTθ Pk−1) (21)and hθ = θk − θk−1, ht = tk − tk−1, ∆e = ek − ek−1, and ek =sk− s∗k , which is the difference between the end-effector positionand the target position at kth iteration. The term ∂ek∂ t predicts thechange in the error function for the next iteration, and in the caseof a static camera it can directly be estimated from the target im-age feature vector with a first-order difference:∂ek∂ t∼=− s∗k − s∗k−1ht(22)The weighting factor is 0 < λ ≤ 1 and when close to 1 re-sults in a filter with a longer memory. The Jacobian estimate isused in the visual controllers to determine the joint variables θkthat track the target.3.3.2 Dynamic Gauss-Newton ControllerThe dynamic Gauss-Newton method [6] minimizes the followingtime varying objective functionE(θ , t) = 12eT (θ , t)e(θ , t) (23)By minimizing above objective function it computes the jointvariables iteratively as follows:θk+1 = θk− ( ˆJTk ˆJk)−1 ˆJTk (ek +∂ek∂ t ht) (24)Control is defined asuk+1 = ˙θk+1 =−Kp ˆJ†k (ek +∂ek∂ t ht) (25)where Kp and ˆJ†k are some positive proportional gain and thepseudo-inverse of the estimated Jacobian at kth iteration, respec-tively.4 ExperimentsIn this section, experimental results are presented both for cali-brated and uncalibrated visual servoing to demonstrate the valid-ity of the proposed scheme.Experiments were conducted with a 7 DOF MitsubishiPA10 robot arm and a Unibrain Fire-i400 digital camera. Thecamera was mounted on a tripod in eye-to-hand configuration inorder to observe the motion of the end-effctor. The images weredigitized at 320× 240 resolution.The system setup is shown inFig.4. The visual control and image processing modules wereimplemented in VC++ 6.0 using OpenCV library and run on P42.26GHz with 1GB ram personal computer.XRZRYRXCZCYCFigure 4. System setup.Fig. 5 shows a test shape, which is on a plane and rigidlyattached to the end-effector. Bitangent points of the shape areacquired using the proposed algorithm in this paper. For visualservoing purposes, either bitangent points or their midpoints, seepoints denoted by 1, 2 and 3 in Fig. 5, can be used. Unlikebitangent points, which are projective invariant, midpoints areaffine invariant. If the scene’s depth is much less than its dis-tance from the camera, a weak-perspective projection can be as-sumed. Throughout the experiments weak-perspective assump-tion is made and the visual feature vector s is constructed fromthe midpoints as follows:s = [x1,y1,x2,y2,x3,y3]TFigure 5. Test shape and midpoints of bitangent pointsIn the case of perspective projection, i.e. if the weak-perspective assumption does not hold, one can use bitangentpoints to construct the visual feature vector.For alignment task, desired pose of the curve is obtainedduring an off-line stage by moving the robot in xz-plane of therobot control frame with some Vx, Vz and Ωy for a certain timeinterval. Consequently, the desired feature vector s∗ is constructedfrom this reference pose.4.1 Calibrated Viual Servoing ResultsThe parameters fx = 1000, fy = 1000, xc = 160, yc = 120 areobtained by a coarse calibration of the camera and Z = 2000 mm.The robot base frame is positioned at z = 2000 mm in z-axis andy = 1000 mm in y-axis away from the camera frame. Thus, wehaveR = −1 0 00 0 −10 −1 0 , t = 010002000where R is the rotational matrix and t is the translational vec-tor that are used for the construction of robot-to-camera trans-formation matrix T . The gain matrix Λ is tuned as Λi = 0.3 fori = 1,2, ..,6. The control input is defined asu =(Vx Vz Ωy)Twhere u consists of the 1st and 3rd components of VR for the mo-tion in xz-plane and 5th component of VR for the rotation aroundy-axis in robot control frame, respectively. Fig.6 depicts the initialand the desired images. Fig.7 shows feature trajectories. Align-ment errors and control signals are plotted in Figs.8-9. The normof the resulting alignment error is found to be less than 1 pixel.4.2 Uncalibrated Visual Servoing ResultsHere we do not need the calibration parameters since the compos-ite Jacobian J ∈ℜ6×3 is estimated in a recursive manner. Only 3joints, namely the 2nd , the 4th and the 6th joints of PA10 robot areused to steer the end-effector by locking the remaining 4 joints.The control parameters are set as λ = 0.96 and Kp = 0.6. Thecontrol input is defined asFigure 6. Initial and desired imagesFigure 7. Feature trajectories on the image plane0 5 10 15 20 25 30 35 40 45−1000100200t (s)error (pixels)Midpoint 10 5 10 15 20 25 30 35 40 45−1000100200t (s)error (pixels)Midpoint 20 5 10 15 20 25 30 35 40 45−1000100200t (s)error (pixels)Midpoint 3Figure 8. Alignment errorsu =(Ω2 Ω4 Ω6)Twhere Ω2, Ω4 and Ω6 are the joint velocities. Figs.10-11 de-pict the initial and the desired images, and the feature trajecto-ries on the image plane. Alignment errors and the control signals0 5 10 15 20 25 30 35 40 45−300−200−1000100t (s)V x, V z (mm/s)Translational Control Signals0 5 10 15 20 25 30 35 40 45−0.100.10.20.3t (s)Ωy (rad/s)Rotational Control SignalFigure 9. Control signals Vx, Vz and Ωyare plotted in Figs.12-13, respectively. The norm of the resultingalignment error is found to be less than 1.5 pixel.Figure 10. Initial and desired imagesFigure 11. Feature trajectories on the image plane4.3 DiscussionsIn both visual servoing approaches we observed that alignmenttask errors are less than 1.5 pixels, which corresponds to 5 mm inrobot workspace. It can be seen that calibrated approach draws0 5 10 15 20 25 30−200−1000100t (s)error (pixels)Midpoint 10 5 10 15 20 25 30−150−100−50050t (s)error (pixels)Midpoint 20 5 10 15 20 25 30−150−100−50050t (s)error (pixels)Midpoint 3Figure 12. Alignment errors0 5 10 15 20 25 30−4−202t (s)Ω2 (rad/s)Joint Control Signals0 5 10 15 20 25 30−4−2024t (s)Ω4 (rad/s)0 5 10 15 20 25 30−1012t (s)Ω6 (rad/s)Figure 13. Control signals Ω2, Ω4 and Ω6more smoother trajectories while the uncalibrated one shows am-biguous behaviour until the Jacobian converges and the end-effector moves towards the desired pose. Computation times ofregion tracking, curve detection and bitangent extraction modulesare approximately 13 ms, 5 ms and 4 ms, respectively.5 ConclusionIn this paper, bitangents are used to design image based visualservoing schemes, both calibrated and uncalibrated, for aligningplanar shapes with a fixed camera. The assumption is that thecurve has at least one concavity on its boundary shape. Exper-imental results validates the proposed method. Alignment tasksare performed with approximately 5 mm accuracy.References[1] N.J. Ayache and O.D. Faugeras, HYPER: A new approachfor the recognition and positioning of two-dimensional ob-jects, IEEE Trans. Pattern Analysis and Machine Intelli-gence, 8(1), 1986, 4454.[2] C. A. Rothwell, A. Zisserman, D. A. Forsyth and J. L.Mundy, Planar Object Recognition using Projective ShapeRepresentation, International J. of Computer Vision, 16,1995, 57-59.[3] Hemant D. Tagare, Shape-based non-rigid correspondencewith application to heart motion analysis, IEEE Trans. Med-ical Imaging, 18(7), 1999, 570578.[4] G. D. Hager. A modular system for robust positioning usingfeedback from stereo vision. IEEE Transactions on Roboticsand Automation, 13(4), 1997, 582-595.[5] S. Hutchinson, G. D. Hager, and P. I. Corke, A tutorial on vi-sual servo control, IEEETrans. on Robotics and Automation,12(5), 1996, 651-670.[6] J. A. Piepmeier, H. Lipkin, Uncalibrated Eye-in-Hand Vi-sual Servoing, The International Journal of Robotics Re-search, 2003.[7] Y.Lamdan, J.T.Schwartz, and H.J. Wolfson. Object recog-nition by affine invariant matching. In Proc. CVPR, pages1988, 335-344.[8] Jacopo Piazzi, Domenico Prattichizzo, Noah J. Cowan,Auto-epipolar Visual Servoing, International Conference onIntelligent Robots and Systems, 2004.[9] Patrick Rives, Jose R. Azinheira, Linear Structures Follow-ing by an Airship using Vanishing Point and Horizon Linein a Visual Servoing Scheme, Internatlonal Conference onRobotics & Automation, 2004.[10] J.Sklansky.Measuring concavity on a rectangular mosaic.IEEE Trans Comput. 21, 1972, 1355-1364.[11] J. L. Mundy, Andrew Zisserman, Geometric invariance incomputer vision, The MIT Press, 1992.[12] S. Benhimane and E. Malis, Real-time image-based track-ing of planes using efficient second-order minimization,IEEE/RSJInternational Conference on Intelligent RobotsSystems, Sendai, Japan, October 2004.

Image based visual servoing using bitangent points applied to planar shape alignment

http://research.sabanciuniv.edu/5638/1/563-125.pdf

Image based visual servoing using bitangent points applied to planar shape alignment

Abstract

Similar works

Full text

Available Versions

Sabanci University Research Database