In this paper, model-based and model-free image based visual servoing (VS) approaches are implemented on a microassembly workstation, and their regulation and tracking performances are evaluated. A precise image based VS relies on computation of the image jacobian. In the model-based visual servoing, the image Jacobian is computed via calibrating the optical system. Precisely calibrated model based VS promises better positioning and tracking performance than the model-free approach. However, in the model-free approach, optical system calibration is not required due to the dynamic Jacobian estimation, thus it has the advantage of adapting to the different operating modes

Bilen, Hakan

Hocaoglu, Muhammet Ali

Hocaoğlu, Muhammet Ali

Ozgur, Erol

Unel, Mustafa

Özgür, Erol

Ünel, Mustafa

English

Sabanci University Research Database

MODEL-BASED VS. MODEL-FREE VISUAL SERVOING: A PERFORMANCEEVALUATION IN MICROSYSTEMSMuhammet A. Hocaoglu, Hakan Bilen, Erol Ozgur, Mustafa UnelFaculty of Engineering and Natural SciencesSabanci UniversityOrhanli Tuzla 34956 Istanbul, Turkeyemail: {muhammet, hakanbil, erol}@su.sabanciuniv.edu, munel@sabanciuniv.eduABSTRACTIn this paper, model-based and model-free image based vi-sual servoing (VS) approaches are implemented on a mi-croassembly workstation, and their regulation and trackingperformances are evaluated. A precise image based VS re-lies on computation of the image jacobian. In the model-based visual servoing, the image Jacobian is computed viacalibrating the optical system. Precisely calibrated modelbased VS promises better positioning and tracking per-formance than the model-free approach. However, in themodel-free approach, optical system calibration is not re-quired due to the dynamic Jacobian estimation, thus it hasthe advantage of adapting to the different operating modes.KEY WORDSVisual servoing, Visual tracking, Micropositioning1 IntroductionVisual servoing is one of the effective methods to compen-sate the uncertainties in the calibration of systems, manip-ulators and workspaces. Over the past years, intense re-search effort in this area has resulted in a number of suc-cessful applications. Two major approaches are presentedin the visual servoing (VS) literature, position-based andimage-based VS [1]-[5]. The first approach is based onreconstruction of 3D model of the object and a calibratedcamera to provide feedback in the cartesian space. In thesecond one, control values are defined in terms of image co-ordinates and no estimation of robot pose is required. Thecomplex geometry of the observed micro-objects and highnumerical apertures of optical microscope which results insmall depth of field lead to a challenging 3D constructionand pose estimation problem. Therefore, an image basedapproach is preferred in our micro visual servoing experi-ments since it does not require an inverse perspective pro-jection.In this paper, model-based and model-free visual ser-voing approaches are experimentally tested in point-to-point positioning and trajectory following tasks. Since theaccuracy of image based VS depends on the computation ofthe image Jacobian matrix, which relates the changes in thecartesian pose to the corresponding changes in the visualfeatures, includes the intrinsic and extrinsic parameters ofthe microscope-camera system. Thus, the calibration infor-mation is vital for computation of the image Jacobian ma-trix and thus the control design. On the other hand, model-free visual servoing does not require a priori information ofthe (robot + optical) system since the composite Jacobian,i.e. product of robot and image Jacobians, is estimated dy-namically [6]. Thus, model-free visual servoing approacheliminates the dependence to the system parameters.The paper is organized as follows: Section 2 definesimage based model-free and model-based visual servoingalong with controller synthesis. Section 3 introduces hard-ware setup and real-time tracking algorithm, and presentsexperimental results and discussions. Finally, Section 4concludes the paper with some remarks.2 Image Based Visual ServoingImage based visual servoing approaches employ the fol-lowing differential relations˙ = Jr˙ (1)where s is a vector of visual features, J is the image Jaco-bian matrix which is a function of the visual features andintrinsic/extrinsic parameters of the visual sensor, and r˙ isa velocity screw in the task space.Depending on the computation of the Jacobian ma-trix, one can talk about model-based or model-free visualservoing strategies. In the sequel, we will review these ap-proaches.2.1 Model Based Visual ServoingModel based visual servoing implies analytical computa-tion of the Jacobian matrix through the calibration of theoptical system.To develop an analytical model of the Jacobian forcalibration purposes, let the objective frame coordinates ofan observed feature point be Po = (Xo,Yo,Zo). Locatingthe image coordinate frame at the center of the CCD arrayand assuming weak perspective projection, the undistortedimage coordinates (x′s,y′s) in objective frame are given asx′s = MXo, y′s = MYo (2)Figure 1. Ray Diagram of the Optical Modelwhere M = Top+ ff+d is the total magnification of the opti-cal system, f is the objective focal length, Top is the tubelength, and d is the working distance, as shown in Fig.1. Since the lens radial distortion parameter (κ1) is verysmall, the distorted image coordinates (xs,ys) in pixels canbe written asxs ≈ x′s =MsxXo, ys ≈ y′s =MsyYo (3)where sx and sy are the effective pixel sizes.The optical flow equations can be obtained by differ-entiating (3) with respect to timex˙s =Msx˙Xo, y˙s =Msy˙Yo (4)Assume that the point P is rigidly attached to theend effector of the manipulator and moves with an angu-lar velocity Ωo = (ωx,ωy,ωz) and a translational velocityVo = (Vx,Vy,Vz). The motion in the objective frame is givenby ˙Xo˙Yo˙Zo= VxVyVz+ 0 −ωz ωyωz 0 −ωx−ωy ωx 0 XoYoZo(5)Substituting (5) into (4) and using (3) impliesx˙s =MsxVx +MsxZoωy− sysxysωz (6)andy˙s =MsyVy− MsyZoωx +sxsyxsωz (7)In light of (6) and (7), the Jacobian matrix is obtained asJ =(Msx0 0 0 MsxZo − sysx ys0 Msy0 −MsyZo 0 sxsy xs)(8)2.2 Model-Free Visual ServoingLet θ denote the vector of joint variables of the robot. Theerror function in the image plane is defined ase(θ , t) = s(θ)− s∗(t)where s∗(t) and s(θ) denote the positions of a moving tar-get and the end-effector at time t, respectively.Since the system (robot+optical microscope) modelis assumed to be unknown, a recursive least-squares (RLS)algorithm [6], main steps of which are briefly summarizedbelow, is used to estimate the composite Jacobian J = JIJR,where JI and JR are the image and the robot Jacobians.Jacobian estimation is accomplished by minimizing thefollowing cost function, which is a weighted sum of thechanges in the affine model over time,εk =k−1∑i=0λ k−i−1‖∆mki‖2 (9)where∆mki = mk(θi, ti)−mi(θi, ti) (10)where mk(θ , t) is an expansion of m(θ , t), which is theaffine model of the error function e(θ , t), about the kth datapoint as follows:mk(θ , t) = e(θk, tk)+ ˆJk(θ −θk)+∂ek∂ t (t− tk) (11)In light of (11), (10) becomes∆mki = e(θk, tk)− e(θi, ti)−∂ek∂ t (tk− ti)−ˆJkhki, (12)where hki = θk − θi, the weighting factor λ satisfies 0 <λ < 1, and the unknown variables are the elements of ˆJk.Solution of the minimization problem yields the fol-lowing recursive update rule for the composite Jacobian:ˆJk = ˆJk−1+(∆e− ˆJk−1hθ−∂ek∂ t ht)(λ +hTθ Pk−1hθ )−1hTθ Pk−1(13)wherePk =1λ (Pk−1−Pk−1hθ (λ +hTθ Pk−1hθ )−1hTθ Pk−1) (14)and hθ = θk−θk−1, ht = tk−tk−1, ∆e= ek−ek−1, and ek =sk − s∗k , which is the difference between the end-effectorposition and the target position at kth iteration. The term∂ek∂ t predicts the change in the error function for the nextiteration, and in the case of a static camera it can directlybe estimated from the target image feature vector with afirst-order difference.2.3 Visual Controller DesignDiscrete-time equivalent of equation (1) can be written ass(k+1) = s(k)+T J(k)u(k) (15)where s∈R2N is the vector of image features being tracked,N is the number of the features, T is the sampling time ofthe vision sensor, and u(k) is the velocity vector of the endeffector.Controller synthesis in this paper is done by optimiz-ing the following cost functionE(k+1) = (s(k+1)− s∗(k+1))T Q( f (k+1)− s∗(k+1))+uT (k)Lu(k) (16)whose solution yields the following control inputu(k) =−(T JT (k)QT J(k)+L)−1T JT (k)Q(s(k)− s∗(k+1))(17)where Q and L are adjustable weighting matrices.3 Experimental Results and DiscussionThe Microassembly Workstation is shown in Fig. 2. Itconsists of PI M-111.1 high-resolution micro-translationstages with 50 nm incremental motion in x, y and z posi-tioning axes, and is controlled by a dSpace ds1005 motioncontrol board. A Zyvex microgripper, with a 100 µm open-ing gap is rigidly attached to the translational stage to graspand pick objects.Nikon SMZ 1500 stereomicroscope coupled witha Basler A602fc camera, orthogonal to XY plane with9.9 µm× 9.9 µm cell sizes was utilized to provide vi-sual feedback. The microscope has 1.6X objective andadditional zoom. Zoom levels can be varied between0.75X −11.25X , implying 15 : 1 zoom ratio.  Stereoscopic    Optical MicroscopeManipulation Tool Holder  Autofocus     Device    Sample    Stages Manipulation      StagesFigure 2. Microassembly Workstation3.1 Calibration ResultsFor model-based visual servoing, an accurate calibrationof the optical system is required and it was accomplishedthrough a parametric model [7]. A round calibration pattern(Fig. 3) is used to establish the correspondence between theworld and image coordinates under 1X and 4X zoom levels.The center coordinates of the circles are calculated througha least square solution.Computed extrinsic parameters (rotation angles α , β ,γ; components of the translation vector, Tx, Ty, Tz) and in-trinsic parameters (total magnification M, objective focalFigure 3. Circular Calibration Patternlength f , tube length Top, working distance d and radialdistortion coefficient κ1), and the 3D reprojection errorsfor the calibration are tabulated in Table 1 and Table 2 re-spectively. It can be observed from Table 1 that the radialdistortion coefficient is very small. This proves that the mi-croscope lenses are machined very precisely. Moreover, βand γ angles have non-zero values which can be resultedfrom a mechanical tilt of the microscope stage or from aninaccurate design of the calibration pattern.Table 1. Computed Extrinsic and Intrinsic Parameters1X 4Xα (degrees) 90.7144 88.9825β (degrees) -2.7912 2.6331γ (degrees) 175.9179 0.9088Tx (µm ) -781.4 76.755Ty (µm ) -55.002 -156.58Tz (µm ) 204900 36370M 1.5893 6.3859d (µm ) 78750 4955.5f (µm ) 126150 31415Top (µm ) 200490 200610κ1 (µm−2 ) −8.4408×10−10 1.5399×10−11Table 2. 3D Reprojection Errors for 1X and 4X Zoom1X 4XMean Error (µm) 0.2202 0.0639Standard Deviation (µm) 0.3869 0.1321Maximum Error (µm) 1.7203 0.58433.2 Visual Servoing ResultsIn order to implement visual servoing algorithms real-timemeasurement of the image features are needed. This isachieved by the ESM algorithm [8], which is based on theminimization of the sum-of-squared-differences (SSD) be-tween the reference template and the current image usingparametric models.Model-based and model-free visual servoing (VS) al-gorithms were experimentally compared in microposition-ing and trajectory following tasks at 1X and 4X zoom lev-els. Micropositioning VS results are plotted in Figs. 4-7, and the trajectory following results for sinusoidal andsquare trajectories are depicted in Figs. 8-11.0 0.5 1 1.5 201020304050t (sec)x axis response(pixels)Micropositioning0 0.5 1 1.5 201020304050t (sec)y axis response(pixels)0 0.5 1 1.5 20100200300400500t (sec)U x (µm/sec)Control signal vs time for x−axis motion0 0.5 1 1.5 20100200300400500t (sec)U y (µm/sec)Control signal vs time for y−axis motionFigure 4. Step responses and control signals of model-based VS at 1XRegulation performances of both approaches for mi-cropositioning tasks in terms of settling time (ts), accuracyand precision are tabulated in Table 3. In the trajectory fol-lowing task, tracking performances of both approaches forsquare and sinusoidal trajectories are presented in Tables4-5.The experimental results illustrate that both of the vi-sual servoing approaches ensure convergence to the desiredtargets with sub-micron error when time considerations arenot primarily important. When the time performance haspriority for the task, the model-based, so called calibratedapproach performs better than model-free one in terms ofsettling time, accuracy and precision (Table 3). Moreover,the tracking performance of the calibrated approach is moreTable 3. Micropositioning for model-based and model-freeVSModel-based Model-freeStep ts Acc. Prec. ts Acc. Prec.(pix) (s) (µm) (µm) (s) (µm) (µm)1x 50 0.80 9.86 2.71 1.6 8.60 3.654x 50 0.45 1.35 0.57 1.6 4.74 1.920 0.5 1 1.5 20204060t (sec)x axis response(pixels)Micropositioning0 0.5 1 1.5 20204060t (sec)y axis response(pixels)0 0.5 1 1.5 2−500050010001500t (sec)U x  (µm/sec)Control signal vs time for x−axis motion0 0.5 1 1.5 2−2000020004000t (sec)U y  (µm/sec)Control signal vs time for y−axis motionFigure 5. Step responses and control signals of model-freeVS at 1X0 0.5 1 1.5 201020304050t (sec)x axis response(pixels)Micropositioning0 0.5 1 1.5 201020304050t (sec)y axis response(pixels)0 0.5 1 1.5 20100200300400500t (sec)U x (µm/sec)Control signal vs time for x−axis motion0 0.5 1 1.5 20100200300400500t (sec)U y (µm/sec)Control signal vs time for y−axis motionFigure 6. Step responses and control signals of model-based VS at 4X0 0.5 1 1.5 2−200204060t (sec)x axis response(pixels)Micropositioning0 0.5 1 1.5 2−200204060t (sec)y axis response(pixels)0 0.5 1 1.5 2−2000200400t (sec)U x  (µm/sec)Control signal vs time for x−axis motion0 0.5 1 1.5 20200400600t (sec)U y  (µm/sec)Control signal vs time for y−axis motionFigure 7. Step responses and control signals of model-freeVS at 4X240 260 280 300 320 340 360 380 400 420180200220240260280300320Sinusoidal trajectory followingx (pixels)y (pixels)0 20 40 60 80 100 12000.511.522.533.54Tracking error vs timet (sec)error (pixels)Figure 8. Actual sinusoidal trajectory and resulting track-ing error in model-based VS at 1Xaccurate and precise than the model-free one. Thus, the cal-ibrated method is more preferable, when accurate and pre-cise manipulation are strongly demanded in a limited time.However, at small magnifications such as M = 1.5893 and240 260 280 300 320 340 360 380 400 420180200220240260280300320x (pixels)y (pixels)Sinusoidal trajectory following0 50 100 150 20000.511.522.533.5t (sec)error (pixels)Tracking error vs timeFigure 9. Actual sinusoidal trajectory and resulting track-ing error in model-free VS at 1X240 260 280 300 320 340 360240260280300320340360Square trajectory followingx (pixels)y (pixels)0 10 20 30 40 50 60 70 8000.511.522.533.54Tracking error vs timet (sec)error (pixels)Figure 10. Actual square trajectory and resulting trackingerror in model-based VS at 1XM = 6.3859 over a large workspace (4× 3 mm2), only acoarse microvisual servoing task could be assumed. There-fore, the accuracy and precision of the model-free approachin the regulation and tracking problems are also acceptable,and the difference between two approaches are not that sig-nificant.220 240 260 280 300 320 340 360 380240260280300320340360x (pixels)y (pixels)Square trajectory following0 20 40 60 80 100 120 14000.511.522.533.544.55t (sec)error (pixels)Tracking error vs timeFigure 11. Actual square trajectory and resulting trackingerror in model-free VS at 1XTable 4. Trajectory tracking for model-based VSSquare SinusoidalAcc. Prec. Acc. Prec.(µm) (µm) (µm) (µm)1x 5.93 2.28 4.79 2.374x 1.47 1.19 1.12 1.31Table 5. Trajectory tracking for model-free VSSquare SinusoidalAcc. Prec. Acc. Prec.(µm) (µm) (µm) (µm)1x 8.65 2.70 6.14 2.744x 1.64 1.12 1.17 0.574 ConclusionModel-based and model-free visual servoing were ex-perimentally evaluated in micropositioning and trajectorytracking tasks. In these experiments, model-based ap-proach performed better in terms of accuracy, precision andsettling time than the model-free approach, however, thisdifference does not necessarily imply a superiority for acoarse manipulation strategy. In addition, the model-freevisual servoing is advantageous due to the fact that there isnot a requirement of the system model in the implementa-tion of the tasks and it can be adapted to different operatingmodes through a dynamic estimation of the composite Ja-cobian.5 AcknowledgementAuthors gratefully acknowledge the support provided bySU Internal Grant No. IACF06−00417.References[1] S. Hutchinson, G. Hager, and P. Corke, A Tutorialon Visual Servo Control, IEEE Trans. Robotics andAutomation, vol. 12, no. 5, 1996, 651-670.[2] F. Chaumette and S. Hutchinson, Visual Servo Con-trol Part 1: Basic Approaches, IEEE Robotics and Au-tomation Magazine, December 2006, 82-90.[3] D. Kragic and H. I. Christensen, Survey on visual ser-voing for manipulation. Technical report, Computa-tional Vision and Active Perception Laboratory, RoyalInstitute of Technology, 2002.[4] W. J.Wilson, C. C.W. Hulls, and G. S. Bell, Relativeend-effector control using Cartesian position-basedvisual servoing, IEEE Trans. Robot. Automat., vol.12, 684696, 1996.[5] P.I. Corke and S. A. Hutchinson, A new partitionedapproach to image-based visual servo control, Pro-ceedings of the International Symposium on Robotics,2000.[6] J.A. Piepmeier, Uncalibrated eye-in-hand visual ser-voing, The International Journal of Robotics Re-search, vol. 22, 2003, 805-819.[7] Y. Zhou and B.J. Nelson, Calibration of a parametricmodel of an optical microscope, Optical Engineering,vol.38, 1999, 1989-1995.[8] S. Benhimane and E. Malis, Real-time image-basedtracking of planes using efficient second order mini-mization, IEEE/RSJ International Conference on In-telligent Robots and Systems, vol. 1, 2004, 943-948.

Model-based vs. model-free visual servoing: A Performance evaluation in microsystems

https://research.sabanciuniv.edu/id/eprint/6731/

Model-based vs. model-free visual servoing: A Performance evaluation in microsystems

Abstract

Similar works

Full text

Available Versions

Sabanci University Research Database