This research report describes a new approach that estimates the velocity of the ego vehicle using a monocular setup to compute an approximation of the scene flow

De Charette, Raoul

Meyer, Alexis

de Charette, Raoul

INRIA a CCSD electronic archive server

HAL Id: hal-01620608https://hal.inria.fr/hal-01620608Submitted on 20 Oct 2017HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Computing Ego Velocity from Scene Flow estimationAlexis Meyer, Raoul de CharetteTo cite this version:Alexis Meyer, Raoul de Charette. Computing Ego Velocity from Scene Flow estimation. [ResearchReport] Inria Paris. 2016. ￿hal-01620608￿Computing Ego Velocity from Scene Flow estimationAlexis Meyer, Raoul de Charette*Abstract—This technical report describes a new approach thatestimates the velocity of the ego vehicle using a monocular setupto compute an approximation of the scene flow.Index Terms—Computer Vision, Image Flow, Ego motion,Intelligent Transportation SystemI. INTRODUCTIONIn the last two decades, several methods were proposed toestimate the ego motion of mobile robot as it is helps egolocalization, trackings of objects and more generally improvesscene understanding. Often ego motion extraction is doneusing LiDAR [1], [2] sometimes with an optional IMU asin [3], or using vision [4], [5], [6]. For the latter, the fashionway is to use a stereo setup and estimate the displacementof the vehicle from the shift of keypoints in left/right imageswhich provides a stereo disparity and thus RGB and Depth. Ashighlighted in [7] there is a large number of stereo techniquescompared to the few monocular ones though some recent paperwere proposed [8], [9], [10].We investigate in this document a method that requires onlya monocular setup and compute the ego motion of the vehiclefrom the approximation of scene flow using a flat-world as-sumption. Our approach is novel and - though preliminary - theresults obtained are promising. In the following we provide anoverview of our methodology, details on our flow computationand the extraction of the optical flow. The document ends withour preliminary evaluations.II. METHODOLOGYWe propose here a new approach to estimate ego velocityusing a monocular camera setup. In contradiction to stereo,the task is made significantly harder since monocular visioncan not provide any depth information.To overcome the lack of depth data, we use a strong as-sumption: the ground is locally planar and all pixels below thehorizon line are ground pixels. Using this assumption, we cancompute the scene flow (velocity in the scene) from temporalimage flow and calibration settings (section III). Althoughour assumption is strong and fails for many pixels (i.e. anynon-ground pixels) we have found that the error induced bythe latter is similar to a widely spread noise. Conversely, inthe velocity/angle space the pixels validating our assumption(i.e. ground pixels) are well grouped and can be extractedthrough Gaussian fitting (section IV). Having done this theestimation of the ego motion is straightforward since theground has a relative speed (i.e. in vehicle reference) equalsto the inverse of the world vehicle velocity. We compare alsoAuthors are with the Robotics and Intelligent Transportation Systems(RITS) Team, INRIA Paris, 2 rue Simone Iff, 75012 Paris, France{raoul.de-charette, alexis.meyer} @inria.frdifferent extraction methods: maximum, and various MeanShift variations. In section V we evaluate our method onthe widely used Kitti datasets [11] and discuss the promisingresults obtained.III. TEMPORAL IMAGE FLOW COMPUTATIONThe aim of this section is to explain how the temporal imageflow has been exploited. The word temporal highlights the factthat this method only requires one single camera. The inputsof this method are the camera calibration parameters : inter-nal parameters (focal length, magnification of the objective,optical center position) and the external ones (rotation andtranslation applied on the camera to pass from the vehicle tothe camera coordinate system).A. 2D image flowImage flow computes the relative shift of pixel patches fromimage t to image t + 1, often through the minimization of adistance function in a restricting neighborhood.In the literature there is two types of image flow, namely:sparse and dense. The former estimates shift of patchesindependently and fails for patches that could not be tracked(i.e. non-saliency patches). Dense image flow on the contraryinterpolates patch motion to fill missing values. In our study,we used the sparse image flow Farneback method [12] whichapproximates by a second degree polynomial, according to thefollowing signal model:f (x)∼ xT Ax+bT x+ c (1)The identification of parameters A, b and c results of aweighted least squares approximation of the signal, where theweighting function normally is a Gaussian. The minimizationof the image signal difference at time t and t + 1 in a givenneighborhood provides an estimation of the temporal shift.Figure 1 shows the temporal optical flow on the Kitti database.The following parameters were used: winsize = 9, polygon n= 5, polygon sigma = 1.1, iteration number = 15 (referto [12] for parameters meaning). Note that to compensatethe camera imaging noise a pre-smoothing is applied (9x9Gaussian kernel).In figure 1 the angle of the shifting was encoded asHue whereas shift magnitude is encoded as Brightness. Forexample, bright red indicates a top right displacement whereasa dark . In the image flow, static elements are eccentric fromthe optical center (when the car is moving forward) whereasdynamic elements have an almost unpredictable behavior.Conversely, a fast overtaking car will induce a concentricimage flow. As expected in such a dynamic scene the opticalflow suffers from various problems, mainly: tracking precision(a) Image at t(b) Image at t +1(c) Temporal optical flowFig. 1. Result of the sparse Farneback image flow [12] on Kitti dataset. Referto text for details.due to a lack of saliency, tracking loss due border effectsor large image displacement (more likely to occur with lowcamera frequency). In addition, the projection model impliessmaller image shift for far-away displacement rather than closeone, thus making harder to detect a dynamic object motionwhen far away.The 2D image flow here obtained cannot be exploited assuch. In fact, processing a 3D scene flow reveals to be thenext step to a better understanding of the scene motion.B. 3D scene flowForward projection geometry tells us that applying theprojection matrix to a 3D vector coordinate point (X ,Y,Z),we obtain its 2D imaged position (x,y):susvs= fx suv cu 00 fy cv 00 0 1 0txR3x3 tytz0 0 0 1XYZ1 (2)and xy1=susvs (3)where u,v the image coordinates, s a scaling constant, fx, fy thefocal length in pixels (respectively, width and height), cu,cv thecoordinates of the optical center in the image, R3x3 and tx, ty,tz the 3x3 rotation matrix and translation components to applyfor camera to vehicle transformation. Using equation 2 we cancompute a forward projection of a 3D point (scene to image),but we require backward projection (image to scene) which inpractice can be done due to the loss of depth information inthe projection process.In spite of the unknown depth, we use inverse calibrationprojection geometry to compute the light ray direction for eachpixel. The pseudo-inverse projection matrix Mi is a matrix thatif multipled from the right by the projection matrix is equal tothe identity. If we multiply from left with Mi, eq. 2 becomes:Mixy1=XcYcZc1 (4)With eq. 4 we are able to obtain the projection vector(Xc,Yc,Zc) of the pixel (x,y). Reader will note that becauseof the loss of s, there is no way to recover the depth alongthe vector. Instead our assumption is to assume that all pixelsbelow the horizon belong to the ground surface which isapproximated as locally planar. That is:a∗X +b∗Y + c∗Z = h (5)With n = (a,b,c) the normal vector of the plane (in our case(0,1,0) because we consider that the flat ground is collinearto the plane Y=0). Hence, the pixel depth can be obtainedfrom the intersection of the projection vector (Xc,Yc,Zc) andthe ground as expressed in eq. 5. This intersection occurs at(X ,Y,Z), with: X = (Xc− tx)d + txY = (Yc− ty)d + ty = hZ = (Zc− tz)d + tz(6)where (X ,Y,Z) are the real coordinates of a point of the scene,d the parameter of the system (the unknown variable) and hthe altitude of the ground.Now that we are able to couple one pixel to a positionin the scene we can turn the image displacement to scenedisplacement, in other words compute the scene flow fromthe image flow. Two components are needed to represent thevelocity vector : the magnitude v of the velocity (in km/h) andits orientation angle θ .v = |d| ∗ f ∗3.6|θ |= acos( VM .Vn|VM ||Vn| )sign(θ) = sign(VnX VMX −VnZVMZ )(7)where d is the displacement vector of a considered elementof the scene (of norm metric the meter) , Vn the unit Z axisvector of the scene and f the frequency of the camera. Notethat the multiplication 3.6 if for m/s to km/h conversion.Figure 2 shows an example output of our scene flowestimation, with the angle (θ ) encoded as Hue and velocity(v) encoded as Brightness. In the latter red (that is 0◦/360◦)correspond to a forward relative motion in the scene, andcyan (that is 180◦) is for backward relative scene motion.We can see that despite different optical flow our strongassumption leads to an accurate scene flow computation forall true ground pixels. This is visible in figure 2 where all road(a) Image flow(b) Scene flowFig. 2. Image flow (top) and our corresponding estimation of the scene flow(bottom). In the latter, the ground is represented with a unified color (darkcyan) which proves that our scene flow estimation is accurate for groundpixels.pixels have similar dark cyan color (i.e. same scene velocityangle and magnitude). Interestingly reader will notice that evenfor non ground pixels that can not satisfy our assumption, theestimated scene flow is coherent. For example even if the blackovertaking car is not the ground, the orientation angle of thevelocity is coherent to the reality.The 3D scene flow should now be analyzed in order toextract the ego motion of the vehicle.IV. EGO MOTION EXTRACTIONFig. 3. Scene flow histogram : Discretisation step of 5 km/h and 5 degrees.Figure 3 shows the histogram of magnitude versus angleof the scene flow velocity. Since many pixels satisfy ourassumption the histogram predominant distribution is locatedat the ground scene velocity location. We use histogramto extract the vehicle ego motion. To extract the velocityfrom angle/velocity distribution we compare three extractionmethods: a) Mean Shift, b) Mean shift with multiple seeds,c) Global maximum.a) Mean Shift density estimator [13]. The principle ofthe latter is to converge iteratively towards the maximumof the distribution, assuming an underlying local Gaussiandistribution. As for most density estimator the Mean Shiftrequires an initialization seed that is critical. In our case weused the maximum histogram as the initial seed. One of themain drawbacks of mean shift is the risk to extract a localmaximum.b) Mean Shift with multiple seeds was implemented toleverage the local maximum risk. Our 8 seeds our radiallydistributed on a circle centered on the maximum of thehistogram and with radius equal to the uncertainty limit of themodel. Our estimate here is the barycenter of the 8 meanshiftsconvergence.c) Global Maximum was also used as a naive comparison.We now discuss the evaluation of our complete methods andcompare the different extraction approaches.V. EVALUATION(a) Extraction on DB1(b) Extraction on DB3Fig. 4. Ego motion extraction on DB1 and DB3 (cf. text for naming).To evaluate the proposed method we used the Kitti bench-mark database with the ad-hoc calibrations settings, and theassociated velocity ground truth. Three datasets were evaluatedfrom the training database 2 that are sequence 2, 8 and9. Respectively refered as DB1, DB2, DB3. The completeextraction results of DB1 and DB3 are displayed in figure4.For all three databases we also detail the RMSE (Root MeanSquare Error) which is a frequently used metric to measurethe differences between two curves: here the ground truth andour extracted ego velocity.Besides the complexity of the tested datasets (shadows, lightsaturation, lack of saliency, non-planar surfaces, traffic), theRMSE (Km/h) DB1 DB2 DB3 MeanMaximum histogram 2.42 2.73 3.48 2.87Mean Shift one seed 2.52 3.47 4.07 3.35Mean Shift multiple seeds 2.49 2.85 3.41 2.91Fig. 5. RMSE calculation for each database. The maximum histogram seemsto have the best results with in average 2.87 km/h.Fig. 6. On the top the car turns to the right, in the middle the correspondingscene flow, in the bottom the scene flow histogram. When the car turns, thevelocity vectors correctly calculated through the method turn to the oppositedirection.extracted velocities are coherent to the reality. When the caris turning, the estimated gaussian is curving to the oppositeway as in fig. 6, which is logical since we observe the inverseego motion vectors. Even when the car stops, the extracted egomotion reaches 0 km/h. Those comments underline the validityof our approach. The database DB 3 has 800 frames andmany changes during the sequence (motion changes, lateralmotion) whereas the two others are linear (no lateral motion,less changes) explaining its higher RMSE (DB 3 column infigure 5). In fact while the car meets changes, the image flowtracking algorithm performances can decrease even slightlycausing the scene flow velocities to be wrongly estimated.For high velocity, the pitch angle of the car changes abruptly(due to relief, pot holes, etc.) causing temporal inconsistencieslocally in the scene flow extraction. As a consequence theextraction is more noisy for higher velocities.In term of quantitative results, we were surprised to notethat the maximum histogram reveals to be the best extractionamong the three implemented. Nevertheless the mean shiftmultiple obtains close results, but the use of barycenter isprobably not sufficient and better clustering should be appliedfor higher precision.VI. CONCLUSIONSThis paper aimed to reveal an interesting and innovative wayof ego motion extraction in the context of autonomous driving.The intentional use of a single camera setup made this ap-proach challenging. Although the results are really preliminaryand require additional work, they look promising and validateour strong assumption. Future works should consider usinga more complex extraction method and a temporal filtering.Undoubtedly, this will improve performances.ACKNOWLEDGMENTThis work has been conducted during an internship atINRIA Paris under the supervision of Raoul de Charette.REFERENCES[1] J. Zhang and S. Singh, “Visual-lidar odometry and mapping: Low-drift,robust, and fast,” in 2015 IEEE International Conference on Roboticsand Automation (ICRA). IEEE, 2015, pp. 2174–2181.[2] J. Zhang, M. Kaess, and S. Singh, “Real-time depth enhanced monocularodometry,” in 2014 IEEE/RSJ International Conference on IntelligentRobots and Systems. IEEE, 2014, pp. 4973–4980.[3] S. Scherer, J. Rehder, S. Achar, H. Cover, A. Chambers, S. Nuske, andS. Singh, “River mapping from a flying robot: state estimation, riverdetection, and obstacle mapping,” Autonomous Robots, vol. 33, no. 1-2,pp. 189–214, 2012.[4] I. Cvišić and I. Petrović, “Stereo odometry based on careful featureselection and tracking,” in Mobile Robots (ECMR), 2015 EuropeanConference on. IEEE, 2015, pp. 1–6.[5] M. Buczko and V. Willert, “Flow-decoupled normalized reprojectionerror for visual odometry.”[6] ——, “How to distinguish inliers from outliers in visual odometry forhigh-speed automotive applications,” in Intelligent Vehicles Symposium(IV), 2016 IEEE. IEEE, 2016, pp. 478–483.[7] F. Raudies and H. Neumann, “A review and evaluation of methodsestimating ego-motion,” Computer Vision and Image Understanding, vol.116, no. 5, pp. 606–633, 2012.[8] M. H. Mirabdollah and B. Mertsching, “Fast techniques for monocu-lar visual odometry,” in German Conference on Pattern Recognition.Springer, 2015, pp. 297–307.[9] S. Song, M. Chandraker, and C. C. Guest, “Parallel, real-time monocularvisual odometry,” in Robotics and Automation (ICRA), 2013 IEEEInternational Conference on. IEEE, 2013, pp. 4698–4705.[10] S. Song and M. Chandraker, “Robust scale estimation in real-timemonocular sfm for autonomous driving,” in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2014, pp.1566–1573.[11] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomousdriving? the KITTI vision benchmark suite,” in Proceedings of theIEEE Computer Society Conference on Computer Vision and PatternRecognition. IEEE, jun 2012, pp. 3354–3361. [Online]. Available:http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6248074[12] G. Farnebäck, “Two-frame motion estimation based on polynomialexpansion,” in Scandinavian conference on Image analysis. Springer,2003, pp. 363–370.[13] K. Fukunaga and L. Hostetler, “The estimation of the gradient of adensity function, with applications in pattern recognition,” jan 1975.

Computing Ego Velocity from Scene Flow estimation

Hal-Diderot

https://hal.inria.fr/hal-01620608/document

Computing Ego Velocity from Scene Flow estimation

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

Hal-Diderot