A method for solving the dense disparity stereo correspondence problem is presented in this paper. This technique is designed specifically for pedestrian detection type applications. A new Ground Control Points (GCPs) scheme is introduced, using groundplane homography information to determine regions in which good GCPs are likely to occur. The method also introduces a dynamic disparity limit constraint to further improve GCP selection and dense disparity generation. The technique is applied to a real world pedestrian detection scenario with a background modeling system based on disparity and edges

Cooke, Eddie

Kelly, Philip

O'Connor, Noel E.

Smeaton, Alan F.

English

Name not available

3D IMAGE ANALYSIS FOR PEDESTRIAN DETECTIONPhilip Kelly ?, Eddie Cooke, Noel O’Connor, Alan SmeatonCentre for Digital Video Processing, Adaptive Information Cluster, Dublin City University, IrelandAbstract A method for solving the dense disparity stereocorrespondence problem is presented in this paper. Thistechnique is designed specifically for pedestrian detectiontype applications. A new Ground Control Points (GCPs)scheme is introduced, using groundplane homography in-formation to determine regions in which good GCPs arelikely to occur. The method also introduces a dynamicdisparity limit constraint to further improve GCP se-lection and dense disparity generation. The techniqueis applied to a real world pedestrian detection scenariowith a background modeling system based on disparityand edges.1 IntroductionMany computer vision based applications depend on ac-curate detection and segmentation of foreground objectsas a first step in their algorithmic process. Traditionalapproaches often fail in unconstrained real-world envi-ronments due to dynamic background conditions such asmoving backgrounds, changing lighting conditions andshadows, and the variability in a foreground objects lo-cal and global appearance. The use of stereo informationhas been proposed as a means to guide object segmen-tation, due to it having some distinct advantages overconventional 2D techniques [1].Many stereo correspondence techniques have beenproposed in literature, a taxonomy of many such tech-niques can be found in [2]. This is a difficult process,however, especially in areas of homogeneous colour or oc-clusion. To improve results, many algorithms make use ofone or more constraints, such as the disparity limit con-straint, which limits the region where a match for a pixelin one image can occur in a second image. This reducesambiguities as well as decreasing computational expense.? This material is based on works supported by ScienceFoundation Ireland under Grant No. 03/IN.3/I361.A second technique to improve results from stereo corre-spondence algorithms is to use highly reliable matchedpixels, known as Ground Control Points (GCPs) [3], tohelp guide results.In this paper we define a dynamic programming basedstereo correspondence technique, using GCPs and dy-namic disparity limit constraints, that has been devel-oped specifically for pedestrian surveillance type appli-cations. The technique for obtaining GCPs is a 3 stageprocess; (1) using groundplane homography information,regions are determined in which good GCPs are likely tobe found; (2) the best GCP disparities are selected basedon the region value built up from neighbouring pixels; (3)background GCPs are found using background disparityand edge models. In addition, we introduce a techniquefor obtaining a dynamic disparity limit constraint to fur-ther improve GCP selection and dense disparity genera-tion, in addition to reducing algorithmic complexity.This paper is organized as follows: Section 2 presentsthe details of the developed algorithmic approach. Firstly,an overview of homographic transformations is intro-duced; we then illustrate how GCPs are obtained anddense disparity is generated. In Section 3 we present ex-perimental results from a real world outdoor situation.Finally, Section 4 details conclusions and future work.2 Algorithm Details2.1 Groundplane SpaceFor convenience, we assume that the two input imagesare rectified via [4]. Rectification aligns the images ver-tically, so that epipolar lines are parallel. A second pre-processing step is then applied that aligns the imageshorizontally with respect to the groundplane. This tech-nique is based on the application of a groundplane ho-mography [5]. The homography maps the groundplanein one image, I1, to the groundplane in a second image,I2, using the equationx ∼= H12x′ (1)2 Philip Kelly, Eddie Cooke, Noel O’Connor, Alan Smeatonwhere ∼= denotes equality up to a scale factor, x is apoint in I1, x′ is a point in I2 and H12 denotes the planeinduced homography from I2 to I1. Applying this trans-formation to the input image I2 results in the alignmentof groundplane points between the two images I1 andH12I2, whereas points above or below the groundplanedo not correspond. Figure 1(a) shows two images I1 andH12I2 overlaid. Let the space wherein the two images I1and H12I2 are be known as groundplane space.2.2 Foreground Activity Regions (FARs)GCPs can be found to help guide stereo correspondencetechniques resulting in more accurate results. However,false GCPs can severely degrade the final matching re-sults [3]. Introducing stricter constraints on the selectionof GCPs could decrease the number of false GCPs, butcould also reduce the total number of GCPs, leading tothe possibility of an insufficient number of GCPs beingavailable to guide the matching process successfully.The properties of the groundplane space can be ap-plied to find areas, called Foreground Activity Regions(FARs), in which depth discontinuities are likely to oc-cur. These regions are typically a good place to extracta large number of strong GCPs as they tend to occur inareas of high texture. FARs occur due to foreground ob-jects being above the groundplane, and therefore they donot match in groundplane space, see Figure 1(a). For anillustrative example, take Figures 1(c) and (d), which arecorresponding sections from I1 and H12I2 respectively.As white region in the two images is part of the pedestri-ans hand, which is foreground, the hand in I1 and H12I2does not correspond, see Figure 1(e). This can be moreclearly seen by viewing the position of the edge pointsin I1 (in green), with respect to the corresponding edgepoints in H12I2 (in blue) in Figure 1(f). The key prop-erty is that, in groundplane space, neither the two sets ofedges nor any pixel between the two edges match in bothcolour intensity and gradient information. These pixelsare denoted in red in Figure 1(g). Let these regions ofnon-correspondence be called FARs. An important as-pect is that, in general, if there is a jump in disparity,then this disparity discontinuity is incorporated withina FAR.Once FARs are determined, the FAR is then searchedfor matching GCPs between images I1 and H12I2. How-ever, an advantage to obtaining FARs to determine GCPs,instead of just regions or points of high texture withinan image, is that a dynamic disparity limit constraint isdetermined for each GCP match. For a static disparitylimit, the closer an object is to the camera rig, the largerthe disparity limit should be. This is an ill-posed prob-lem as the distance of the object to the camera is notdetermined until the object’s depth is obtained. The con-straint is therefore usually fixed at the largest expecteddisparity that should occur in the scene. Using the FARs(a) (b)(c)(f)(d)(g)(e)(h)Fig. 1 (a) Groundplane Space; (b) FARs; (c) I1; (d) H12I2;(e) I1 and H12I2; (f) Edges; (g) FARs; (g) Disparityregions, it is possible to roughly estimate the maximumpossible disparity. In general, if there is a jump in dis-parity within a FAR then a match for every point in theFAR will be at a disparity less than the width of thatFAR. Observing Figure 1(h) notice how the width of theFAR, d, is greater than that of the disparity of the twoedges. The search for GCPs in each FAR is implementedusing this variable disparity limit constraint.2.3 Multiple Initial Ground Control Points (MIGCPs)Obtaining GCPs from FARs is a three stage process;(1) multiple initial GCP matches are found at variousdisparities; (2) the best choice of disparity for these GCPmatches are determined; (3) control point regions aredetermined.The first step involves finding Multiple Initial GroundControl Points (MIGCPs). In general it is more difficultto determine the accurate disparity of a point that hashomogeneous rather than heterogeneous neighbours. Forthis reason, we ensure that the matches for the MIGCPsare instantiated by edges that have an edge gradientgreater than a threshold, tsg, which is vertically orientedwith respect to the scanline. We refer to a point thatmeets this criteria as a maximum vertical edge, maxve.To determine the MIGCPs for a given FAR on aparticular scanline, each maxve within the FAR of I1,calledmax1ve, is compared to eachmaxve inH12I2, calledmax2ve, that is within a disparity of the width of theFAR from max1ve. The two maxve are compared us-ing the sum of absolute square distances (SAD) in theirRGB colour intensities. If the SAD is less than a thresh-old, tMaxAccept, then a possible match between the twomaxve exists. If this is the case then the process of ob-taining additional collaborating information is under-taken. Initially the number of the collaborating value,valcol, is 1. Collaborating information is gathered bymoving the current point in each image left by a sin-gle pixel along the scanline. The new image points failsto be collaborating the max1ve and max2ve match if1. The current point in either image is outside of theFAR or homogeneous in colour with its neighbours;2. SAD(curr1, curr2) ≥ tMaxAccept, where curr1 is thecurrent pixel in I1 and curr2 is the current pixel inH12I23D IMAGE ANALYSIS FOR PEDESTRIAN DETECTION 3If neither of the above is true, then valcol is incrementedby one, as another pixel match agrees with the choiceof MIGCP match. A similar technique is then appliedfrom max1ve and max2ve in the opposite direction alongthe scanline. The greater the value of valcol, the morelikely the match is of being correct.2.4 Final Ground Control Points (FGCPs)However, determining GCPs based on horizontal matchesalone, as is done in MIGCP, can result in false matches.This second step finds collaborating pixels vertically acrossscanlines, enforcing interline consistency for selecting GCPs.The technique to achieve this can be shown in the fol-lowing example.Let Figure 2(a) represent the set of MIGCPs in I1,so for example on scanline 1 there is 1 MIGCP at pixel1C1, where the subscript 1 represents I1. Let Figure2(b) represent the possible match disparities inH12I2 forthe MIGCPs in (a), therefore, for example, the MIGCPin scanline 1 has 3 possible matches at 1A2, 1C2 and1F2, where the subscript 2 represents H12I2. For eachMIGCP in I1 it is checked to see if there exists one ormore MIGCPs in the previous scanline within a distanceof 1 pixel. In this example, for 2B1 there is a MIGCPin the previous scanline at 1C1. If more than one exists,the MIGCP that has the closest colour intensity andgradient information to 2B1 is chosen. For each possibledisparity match of 2B1; determine if 1C1 has a corre-sponding possible disparity match in H12I2 in the samevicinity. In this case, the match 2B1 → 2E2, 1C1 hasa corresponding match from 1C1 → 1F2. However, thematch 2B1 → 2B2, has two corresponding matches from1C1 → 1A2 and 1C1 → 1C2. In this case the pixel from1A2 or 1C2 that is closest in colour intensity and gra-dient information to 2B2 is chosen, assume it is 1C2. Ifa corresponding match is found then the valcol for thetwo pixels in I1 for the matched disparity are added to-gether. In this case the valcol for matching 2B1 → 2E2and 1C1 → 1F2 are added together, as are the valcol formatching 2B1 → 2B2 and 1C1 → 1C2, see Figure 2(c).This process continues to the next MIGCPs in theimage. In scanline 4, there is only one match to 4C1,namely 4F2. This means that the chain in H12I2 con-taining 1C, 2B and 3A cannot continue. However thechain containing 1F, 2E and 3E grows longer to contain1F, now the valcol of any of the pixels in this chain isthe sum of all the initial MIGCP valcol in the chain. Byusing this technique, longer chains have larger values forvalcol. FGCPs are then found by obtaining the highestvalcol for each pixel. Finally, in post processing, we re-move all FGCPs that are not bidirectional, all FGCPswhere the highest valcol is not at least twice that of thesecond highest and all FGCPs that do not have a chainthat spans at least 3 scanlines. In addition we use dy-namic programming to enforce the ordering constraint(a) (b) (c)Fig. 2 FGCPs (a) I1; (b) H12I2; (c) H12I2 links;in FCGPs. These post processing steps ensure that mostambiguous FGCPs are removed.2.5 Ground Control Point RegionsAt this point it is possible to obtain more GCPs by ex-tending the FGCPs across regions of homogeneous tex-ture. An initial step for this process is to cluster FGCPsinto regions of homogeneous disparity using connectedcomponents, where two separate regions, r1 and r2, canmerge if √(rd1 − rd2)2 ≤rdx ∗ θσ(2)where rd1 is the disparity of r1, rd2 is the disparity of r2,rdx is the lowest disparity of the two regions and θ is themaximum disparity difference allowed for every σ pixelsof disparity that exists in rdx. In our experiments θ =1.5 and σ = 10, allowing 1.5 pixels disparity differencewhen rdx = 10, 3 pixels disparity difference when rdx =20, and so on. This allows pixels closer to the camerato cluster together easier, allowing for fluctuations inthe objects surface. Two separate FGCPs regions canthen be merged, if they are separated by a region ofhomogeneous texture. If this is the case then all pixelsbetween the two FGCPs regions are also classified asFGCPs.2.6 Background Ground Control Points (BGCPs)Background objects, such as walls, do not, in general,change position within an input image. If a sequence ofimages of the same scene is used, background disparityand edge models can be built. The initial backgrounddisparity model can be set to have the same disparityas the groundplane space. The model will then be up-dated over time to incorporate the background objectssuch as walls, bollards, etc. These background modelscan be used to eliminate the need for searching for GCPsthat arise from background objects. To implement this,we first stop looking for GCPs that occur due to back-ground objects by using the background disparity modelto detect FARs instead of the groundplane space. Back-ground objects will therefore not cause FARs, and thusno GCP will be searched for in these regions. BackgroundGCPs (BGCPs) are then determined using backgroundmaximum vertical edges, bmaxve, which are strong ver-tical edges that do not appear as foreground in eitherthe background edge or disparity model. If two bmaxve4 Philip Kelly, Eddie Cooke, Noel O’Connor, Alan Smeatonare separated vertically by a region of homogeneity thatdoes not contain a FAR then all points between the back-ground maximum vertical edges are defined as a BGCPregion. A BGCP region can propagate downward to thenext scanline iff there is no FAR anywhere inside theBGCP region on the next scanline. Finally a propagatedBGCP region can extend left and right until a FAR ora maximum vertical edge is reached. Figure 3(d) showsresults from obtaining FGCPs and BGCPs.2.7 Dense Disparity EstimationThe GCPs can be used as a basis to guide various dif-ferent stereo correspondence algorithms. We used themin conjunction with a single pass dynamic programmingbased approach, allowing GCPs to have zero cost in thematching process. It optimises the scanlines by minimis-ing an energy function that is similar to [3];E(d(x, y1)) =∑xC(x, y1, d(x, y1)) +∑x(λ(x, y1)ρ(d(x, y1)− d(x+ 1, y1)) +λ(x, y1)ρ(d(x, y1)− d(x, y1 − 1)))where y1 is the current scanline, ρ is the Potts modeland λ(x, y1) is a weight function, set to 30 in our exper-iments. A cost added for both a vertical and a horizon-tal difference in disparity to help enforce inter scanlineconsistency. In addition the idea of a dynamic disparitylimit constraint is used. The limit for a given scanline isfound as the maximum of the previous lines dense dis-parity, the current and next lines background model andGCP disparity.3 Experimental ResultsFigure 3(e) shows results of the dense disparity estima-tion technique using two cameras from a Digiclops [6]stereo camera rig. The end goal of this research is tocount pedestrian numbers in a scene. It is clear fromthe results that the disparity information obtained willgreatly aid pedestrian segmentation, but this informa-tion, in itself, is not enough. Take for example, Figure3(e), row 3, where a man has roughly the same disparityas the wall. We are are therefore unable to separate thepedestrian by stereo information alone. It is also impor-tant to also notice that only the pedestrians head andhands appear as FARs. This is due to the pedestrianstorso and the background having the same colour andtherefore there are no edges or indication of a disparityjump. This problem can also occur if one pedestrian isoccluded by another wearing the same coloured clothes.The result of this would be an undefined region betweenthe two pedestrians as there would be no visible dispar-ity discontinuity.(a) (b) (c) (d) (e)Fig. 3 (a) Groundplane Space; (b) FARs; (c) FGCPs; (d)FGCPs and BGCPs; (e) Disparity4 Conclusions and Future WorkThis paper described a technique for dense disparitymatching designed for pedestrian detection type appli-cations using GCPs and a dynamic disparity limit con-straint. In future work, the use of temporal data alongwith the improvement of the dense disparity algorithm toenforce better inter scanline consistency would increasethe accuracy of the dense disparity data. Current workincludes the use of disparity and biometric information,such as height/width, being applied to segment objects,even at the same depth, into separate objects and theclassification of that object as pedestrian or otherwise.References1. L. Zhao and C. Thorpe. Stereo and neural network-basedpedestrian detection. IEEE Trans. on Intelligent Trans-portation Systems, 1(3):148–154, September 2000.2. D. Scharstein and R. Szeliski. A taxonomy and evaluationof dense two-frame stereo correspondence algorithms. Int.Journal of Computer Vision,, 47(1/2/3):7–42, April 2002.3. B.T. Choi C. Kim, K.M. Lee and S.U. Lee. A dense stereomatching using two-pass dynamic programming with gen-eralized ground control points. In IEEE Conf. on Com-puter Vision and Pattern Recognition, June 2005.4. A. Fusiello, E. Trucco, and A. Verri. A compact algorithmfor rectification of stereo pairs. Machine Vision and Ap-plications, 12(1):16–22, 2000.5. P. Kelly, P. Beardsley, E. Cooke, N. O’Connor, andA. Smeaton. Detecting shadows and low-lying objects inindoor and outdoor scenes using homographies. In IEEConf. on Visual Information Engineering, April 2005.6. http://www.ptgrey.com/products/digiclops/index.html.

3D image analysis for pedestrian detection

DCU Online Research Access Service

A compact algorithm for rectiﬁcation of stereo pairs.

A dense stereo matching using two-pass dynamic programming with generalized ground control points.

A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.

Detecting shadows and low-lying objects in indoor and outdoor scenes using homographies.

Stereo and neural network-based pedestrian detection.

O\u27Connor, Noel E.

Irish Universities

http://doras.dcu.ie/381/1/wiamis_2006.pdf

3D image analysis for pedestrian detection

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service

Irish Universities