Firl, J.

Hoerter, M. H.

Lauer, M.

Stiller, Chr.

KITopen

Vehicle Detection, Classification and Position Estima-tion based on Monocular Video Data during Night-time  Jonas Firl, Marko H. Hoerter, Martin Lauer and Christoph Stiller  Keywords:  Automotive Lighting, Light-based Driver Assistance, Detection, Tracking, Classifi-cation, Light Sources. 1 Abstract This study describes an effective method for detecting, tracking and classifying vehicles during night-time in order to support automotive adaptive illumination applications. The hereby described software framework, which computes the rel-ative position, velocity and estimated class of all detected vehicles, integrates multiple processing stages. Firstly, an image segmentation using a threshold me-thod to detect all light sources in the image. Secondly, possible pairs of head- and taillight are clustered using geometrical information. Thirdly, all detected ob-jects are tracked using a Kalman-Filter to increase resolution and robustness of the algorithm. Lastly, a method for computing distance and velocity for all classi-fied objects (e.g. cars, trucks, bikes…) is presented. The system is tested to run in real-time and some results and conclusions are offered at the end. 2 Introduction Driving a vehicle during night-time is relatively more hazardous than at day-time, which was recently reviewed in detail by the German Federal Office of Statistics (DESTATIS, [1]). Technically inadequate illumination of the present road scenery as well as wrongly utilized light function (e.g. pure usage/ alternation between low- und high-beam) are just some examples on that. In order to run lighting based driving assistance system acting upon the maxim “best illumination without dazzling other traffic attendance”, it is mostly important to estimate the spatial position, velocity as well as the classified category of the detected light source. By having such data reliably acquired from the sensor side, different lighting based driving assistance system can be realized, such as Adap-tive Cut-Off-Line, Predictive Illumination Distance Control or Masked High Beam derivates [2]. 2.1 Objective of research project The intention of the hereby described research project is to detect, track and fi-nally classify light-sources based on real-world monocular image sequences. To implement a most suitable and efficient detection algorithm, different detection methods have been evaluated to each other with respect to their performance. For tracking purposes a linear and discretized Kalman-Filter has been utilized, which represents an iterative and stochastic state estimator by minimizing the middle error square. The applied classifier calculates the probability of error con-cerning to the found light source based on real-world image sequences. 2.2 Utilized Hard- and Software The above mentioned image sequences were acquired by utilizing the dedicated testing vehicle at the Department of Measurement and Control (see Figure 1 and Figure 2). The hereby used visual sensor (Point Grey, type “Firefly MV”) has been mounted on a camera platform close to the rear-view mirror and was driven at 30 frames per second. To guarantee timely synchronous data (e.g. vehicle immanent plus video data), a real-time data base (RTDB) framework, introduced in [4], has been exploited.  Figure 1 - AUDI Q7 as a testing vehicle at the Department of Measurement and Con-trol. [3]  Figure 2 - Dedicated camera platform mounted behind front window. [3] 3 Light source detection Within this section the used light sources detection algorithm will be described in detail. As a requirement, the detection algorithm should be capable not only to detect vehicle front lights (e.g. cars, trucks, motorcycles), but also other light sources with sufficient light intensity, like street lamps. In [3] different detection methods, like SURF, HOUGH-transformation or dedicated (multi-level) threshold operation with respect to performance and applicability have been evaluated. As a result of this evaluation finally a picture-row-based threshold operation, de-scribed in [5] has been utilized in the further proceeding (see Figure 3).    Figure 3 - Found parameter curve of the pic-ture-row-based threshold method; here ap-plied on a 640x480 pixel picture. [3]  Figure 4 - Outcome of the picture-row based threshold filtering method. [3] 3.1 Light source detection with threshold filter As described above, a picture-row-based threshold operation will be utilized to set the threshold value corresponding to the vertical pixel position. Like pointed out in [5], one of the most important benefits of this method is to detect even low-level light emitting sources (e.g. tail lights) in far distances in comparison to me-thods with a constant threshold. Based on the threshold filtered pictures con-nected blobs, which are basically two or more detected light sources, can be ex-tracted. Therefore, the so called connected component analysis, more specifical-ly detailed in [6], has been used in this work. Hereby one calculated blob inte-grates values as follows: midpoint, standard deviation and number of pixel. Due to their occurrence, detected light sources above the horizon are completely de-scribed by these values (see Figure 4), whereas the blobs underneath the hori-zon still need further processing to extract single light sources out of it. 3.2 Detection of reflections If single light sources do have sufficient distance between each other, the men-tioned detection will be done by the threshold filter operation itself. If not, two or more light sources appear as a single light source. This happens most likely with-in overexposed picture regions, for instance caused by reflection or dense traffic situations. The challenge on that is to evaluate the detected blobs of the given the picture, whether they consist of one or multiple light sources (e.g. two head lights plus two corresponding reflections, see Figure 5).   Figure 5 - Consolidated blobs resulting from threshold filtering. [3]  Figure 6 - Pair of head lights with corres-ponding reflections. [3]  Figure 7 - Separated blobs (red) plus cor-responding reflections (green). [3]   Figure 8 - Pairs of light sources grouped together by clustering algorithm. [3]  Within this work, a proper separation between single blobs and their reflections have been accomplished by summing up the pixel values column by column as well as row by row. A blob detected at its position  with-in a picture  results with its corresponding row sum vector  as well as its column sum vector :   As next step those stated vectors can be inspected for local maxima to get the number of separate light sources out of it. By illustrating Figure 6, we can detect two local maxima for each pixel direction, whereas at their intersection points the four corresponding light sources can be found. This makes it easy to distinguish between light sources and reflections by simple heuristic knowledge (reflection are most usual underneath its corresponding light sources) by considering a maximum gap between those two groups (see Figure 7). 3.3 To cluster pairs of front- and tail lights As a result of the detection stage, a list of detected light sources  per picture can be extracted. To be able to track obstacles within the traffic con-text, a clustering method has to pool potential pairs of front- and tail lights to-gether. In [7] a method is presented, which clusters light sources within a given picture based on geometrical properties. In our approach, this method has been adapted to the effect that the input data of two light sources  and  with geo-metric dimension is given as follows : The maximum and minimum pixel position , the correlated height and width , as well as the number of pixel corres-ponding to the light source . The two given light sources will be pooled togeth-er as soon as the following four properties are positively evaluated: (1) Vertical projections have adequate intersection areas: ),max(minmin jivpjiHHcvv <− , ),max(maxmax jivpjiHHcvv <− . (2) Height approximately at the same range: ),max( jijji HHcHH <− . (3) Number of pixel approximately at the same range: ajiji cAAAA >),max(),min(. (4) Horizontal distance as follows: ),,,max(2)()(minmin maxmaxjijidjjiih WWHHcuuuuD <−−−= . The parameters  have been justified manually through evaluating a huge amount of test data. As two major differences in comparison to [7], the cal-culation of the horizontal distance  of a dedicated light source has been adapted in two ways: Firstly, by not only taking the maximum height under con-sideration, but also the maximum width. This has been done due to the fact that light sources appear in reality not always in ideal round shape. Secondly, the threshold  has been increased to a higher level to be able to detect also very tiny appearing tail lights. As a result on that, a list of potential pairs of front- and tail lights can be stated as well as applied to the given sequence of pictures (see Figure 8).   Figure 9 - Camera and world coordinate sys-tem. [3] 020406080100120distance [meters]real distancemeasured distance Figure 10 – Results of the position estima-tion of an object in different distances. [3]   4 Position- and velocity estimation The aim of this section is to present a method to compute 3D positions in a fixed world coordinate system by given 2D image coordinates of detected light sources. These data are essential to get parameters like distance or velocity of detected vehicles in the road scenery in front of the own car. At the same time they will be the input data of the tracking algorithm described in section 5.  In most cases the problem of the under-determined transformation system is solved by using a stereo vision approach. Due to the hardware setup of this project - usage of only one camera in order to keep costs and packaging down - and to general problems using stereo vision systems in night time scenarios a restriction of the world has to be done: we assume a flat (2D) world. Hence the horizon can be assumed to be a constant horizontally line in the image. With the world coordinate system given in Figure 9, all objects have to be copla-nar parallel to the ),( WW ZX -plane, and therefore have a constant “height” d . This plane can be described by using e.g. the hessian normal form 0)0,1,0(,),,( =− dZYX TTWWW . With the known parameters from the camera calibration (i.e.: rotation matrix R , translation vector T , camera matrix A ) we can describe the transformation from a given image point ),( vu : TRvuARZYX TTTTWWW −= − )1,,(),,( 1λ , with: TTTTTvuARTRd)0,1,0(,)1,,()0,1,0(,1−+=λ  In Figure 10 the results of the transformation are shown (distance 22 WW ZXd += ). We can easily see the quadratic error increase due to small pixel errors, which normally occur with increasing object distances (see also [8]).  After the positions are estimated, the next step is to compute the velocity relativ to the own car. With avaible CAN-data (velocity and steering angle) it is easy to compute the vectoriell velocity using given positions tWWt ZXp ),(=→ at time t , e.g. using the formula presented in [9]:  mnppvnknmmk ktktt∑ ∑= + += −− −= 1 1 . 5 Clustering and Kalman-tracking In this section an algorithm to track possible vehicle-like objects in a sequence of images will be presented. Prior to the tracking it is required to extract possible vehicles like cars or bikes out of the blob list and generate a list of all objects in an image. Therefore we have to recognise, that vehicles can appear in the image as objects consisting of one (e.g.: bikes, far away cars) or of two blobs. Another important fact is that, with the flat world assumption of section 4 and the therefore constant skyline, vehicles can only appear under the constant image row of the horizon.  One-blob objects can be detected using information about there size and location in the image. Two-blob objects have to be clustered using the technique de-scribed in section 3.3.  After creating a complete list of objects in the image, a tracking algorithm can be applied, which will recover all vehicles over a sequence of images as well as compute their relative physical values. Therefore objects of the last image have to be associated with the current image data, taking account of the information gained during the past recovering process. One of the difficulties of this process is the fact that one object doesn’t have to consist of the same number of blobs over a set of frames.  After this data association a linear and discretized Kalman-Filter is used to in-crease robustness of the results and to improve spatial resolution of the distance estimation. The state vector x  and the measure vector y  are defined as follows, considering the flat world assumption: TZxWW vvZx ),,,(=   and   TXZy ),(= , with the velocities in WX - and WZ -direction Xv  and Zv . Assuming the model of approximately constant velocity (between two frames), the system matrix is given by: ⎟⎟⎟⎟⎟⎠⎞⎜⎜⎜⎜⎜⎝⎛=10000100010001TTA , with the camera frequency T/1 . The error, which occurs using this modell, is handled with the error matrix Q , which can be computed by using the modell of piecewise constant accelaration, simplified resulting in: ⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠⎞⎜⎜⎜⎜⎜⎜⎜⎜⎜⎝⎛=222322232324232402000220400204XXZZXXZZaTaTaTaTaTaTaTaTQ . Here, Za  denotes the maximal velocity in WZ -direction, Xa  respectively.  In Figure 11 the results of the Kalman-Filter are shown. The estimated and Kal-man-filtered distances are plotted over some frames from a scenario of one de-parting vehicle to point out the advantages of this tracking algorithm.  0204060801001201 21 41 61 81 101frame numberdistance [meter] estimateddistance Kalman‐filtereddistance Figure 11 - Results of the Kalman-Filter. [3] 6 Object classification The final stage of this project is to execute a classification process to all detected and tracked objects. We sort the detected objects into the following classes: Bike, Car, Truck and Unknown. Firstly a feature-vector imr  is defined for every object OOi ∈ , which has to be computed befor starting the classification: Tblobsi avsqdltnm ),,,,,(=r , with: ? blobsn :  Number of blobs the object consists of [-] ? lt :  Lifetime of the object (number of tracked images) [-] ? d :  Distance of the object [meters] ? sq :  Squareness of the object (width[pixel] / height[pixel]) ? v :  Velocity of the object [meters/sec²] ? a :  Area of the object [pixel²]  After interpreting a huge amount of training data, it is possible to formulate a probability function )|( mcp  for every class c  and for every element m  of the fea-ture-vector imr . The resulting probabilities are weighted and summed up to get the complete probability )|( imcpr  for every class and allow using a simple maxi-mum a posteriori classifier. Because of the difficulty in distinguishing between lights of a truck and those of a passenger car, the process shown in Figure 12 has delivered good classification results, which executes a detection of clearance lights for all possible trucks as a distinctive means.  As a result we obtain the corresponding class for each object (class Unknown is an indicator for an uncertain classification or any kind of noise) including its prob-ability.  compute probability for classes:- passenger car / truck- bike  >truckcarP /bikePbikePdistinguish between classes car and truck by detecting clearence lights    > cclass: car    >bikePclass: bikeclass: unknownclass: truckyes nonoyes yestruckcarP /truckcarP / c  Figure 12 – Classification process, the constant threshold c is set experimentally. [3] 7 Summary and Outlook With this paper we have presented a novel software framework to detect, track and classify light sources within a traffic context. By evaluating the computation time of the different software modules, it can be seen, that the over-all algorithm is capable to run under real-time conditions due to its highly efficient implementa-tion [3]. In the future, light-based driving assistance systems will be highly supported by advanced sensor systems, such as video, radar, laser, or lidar units to estimated physical values of the driver`s environment to guarantee an optimum of illumina-tion while driving at night-time. The presented approach in this paper shows first promising results, nevertheless further enhancement in the field of detection range and reliability have to be done in the future. 8 References [1] Statistisches Bundesamt Deutschland, „Entwicklung der Zahl der im Stra-ßenverkehr Getöteten 1953 bis 2008“, Section „Verkehr/ Verkehrsunfälle“, Ger-many: Wiesbaden, 2009 [2] Marko H. Hoerter et. al, „A Hardware and Software Framework for Auto-motive Intelligent Lighting“, Proceedings of IEEE Intelligent Vehicles Symposium 2009, China: Xi`an, 2009 [3] Jonas Firl, „Lichtquellendetektion, -klassifikation und Positionsbestimmung in nächtlichen monokularen Bildsequenzen“, University of Karlsruhe(TH), De-partment of Measurement and Control, Germany: Karlsruhe, 2009 [4] Matthias Goebl et al., “A Real-Time-capable Hard- and Software Architec-ture for Joint Image and Knowledge Processing in Cognitive Automobiles”, In Proc. IEEE Intelligent Vehicles Symposium, pages 734-740, Turkey: Istanbul, 2007 [5] M.Y. Chern et. al, “The lane recognition and vehicle detection at night for a camera-assisted car on highway”, In Proc. of IEEE International Conference on Robotics and Automation 2003, volume 2, 2003. [6] F. Chang et al., “A linear-time component-labeling algorithm using contour tracing technique”, Computer Vision and Image Understanding, 93(2):206–220, 2004. [7]  Yen-Lin Chen et al., „Nighttime vehicle detection for driver assistance and autonomous vehicles”, Proceedings of the 18th International Conference on Pat-tern Recognition, pages 687–690, USA: Washington, DC, 2006. [8]  Gideon P. Stein et. al, „Vision-based ACC with a Single Camera: Bounds on Range and Range Rate Accuracy“, Proceedings of the IEEE Intelligent Vehi-cles Symposium 2003, June 2003 [9] Kaiqi Huang et. al, „A real-time object detecting and tracking system for outdoor night surveillance“, Pattern Recognition, 41(1):432 – 444, 2008 [10] Yen-Lin Chen et. al, „Night-time Vehicle Detection for Driver Assistance and Autonomous Vehicles“, In ICPR ’06: Proceedings of the 18th International Conference on Pattern Recognition, pages 687–690, Washington, DC, USA, 2006. IEEE Computer Society. 

Vehicle Detection, Classification and Position Estimation based on Monocular Video Data during Night-time

https://publikationen.bibliothek.kit.edu/1000014692/1110930

Vehicle Detection, Classification and Position Estimation based on Monocular Video Data during Night-time

Abstract

Similar works

Full text

Available Versions

KITopen