This paper deals with the problem of estimating the following road sign parameters: height, dimensions, visibility distance and partial occlusions. This work belongs to a framework whose main applications involve road sign maintenance, driver assistance, and inventory systems. From this paper we suggest a multisensory system composed from two cameras, a GPS receiver, and a distance measurement device,all of them installed in a car. The process consists of several steps which include road sign detection, recognition and tracking , and road signs parameters estimation. From some trigonometric properties, and a camera model, the information provided by the tracking subsystem and the distance measurement sensors, we estimate the road signs parameters.Results show that the described calculation methodology offers a correct
estimation for all types of traffic signs

Gil Jiménez, Pedro

Lafuente Arroyo, Sergio

López Sastre, Roberto Javier

Maldonado Bascón, Saturnino

Siegmann , Philip

English

Siegmann, Philip

e_Buah - Biblioteca Digital de la Universidad de Alcalá

Road Sign Analysis Using Multisensory DataR.J. López-Sastre, S. Lafuente-Arroyo, P. Gil-Jiménez, P. Siegmann,and S. Maldonado-BascónUniversity of Alcalá, Department of Signal Theory and CommunicationsPolytechnic School, A-2 Km. 33,600 - 28805 - Alcalá de Henares - Madrid, Spain{robertoj.lopez, sergio.lafuente, pedro.gil, philip.siegmann,saturnino.maldonado}@uah.esAbstract. This paper deals with the problem of estimating the follo-wing road sign parameters: height, dimensions, visibility distance andpartial occlusions. This work belongs to a framework whose main appli-cations involve road sign maintenance, driver assistance, and inventorysystems. From this paper we suggest a multisensory system composedfrom two cameras, a GPS receiver, and a distance measurement device,all of them installed in a car. The process consists of several steps whichinclude road sign detection, recognition and tracking , and road signsparameters estimation. From some trigonometric properties, and a ca-mera model, the information provided by the tracking subsystem and thedistance measurement sensors, we estimate the road signs parameters.Results show that the described calculation methodology offers a correctestimation for all types of traffic signs.1 IntroductionSeveral works have recently focused on traffic sign detection and recognition [1],[2], [3], [4], [5] and [6]. Specifically, in [7] it is described the framework that weuse in this research. In this paper we handle the task of automatically estimatingheight, dimensions, visibility distance and partial occlusions of road signs. Theaim of this work is to get an efficient framework which could be used in ainventory system, which could provide all of these estimated parameters for eachsign, and not only the type. From this work, we present a complete developmentbased on trigonometric relationships and a camera model to estimate the listedbefore parameters of road signs.This paper is organized as follows. Section II shows a global vision of oursystem, and in section III we describe all the implemented algorithms to achievethe traffic sign parameters estimation. Sections IV and V report the obtainedresults and present the conclusions of this work, respectively.2 System OverviewFor this work we have built a complete inventory system as it is shown in Fig. 1,which has the following subsystems:J. Mira and J.R. Álvarez (Eds.): IWINAC 2007, Part II, LNCS 4528, pp. 251–260, 2007.c© Springer-Verlag Berlin Heidelberg 2007252 R.J. López-Sastre et al.– Capture subsystem. We have mounted two firewire cameras on a car.The first one is entrusted to capture the road signs with a framerate of 15fps (frames per second), and with the second one we want to capture themilestones of the route. This subsystem synchronizes the capture of thesetwo cameras, and it saves the captured images in the hard disk of a laptop.– A Positioning subsystem which is composed of a GPS (Global PositioningSystem) receiver and a Distance Measurement Device (DMD). The aim ofthis block is to acquire the GPS position of the vehicle, each second, and tomeasure the covered distance by the car. This subsystem sends all of thesemeasured data to the Traffic sign analysis subsystem.– Traffic sign analysis subsystem, which will develop two task. Firstly, thesystems tries to detect, recognize and track the traffic signs in the images,and in a second step, the system performs the estimation of parameters.Fig. 1. Diagram with all subsystems which are part of the complete systemIn Fig. 2 we show a complete flow diagram with all the tasks that this sys-tem realizes. In the last step it is where we implement the parameters esti-mation process which is the main focus of this paper. It is important to notethat this estimation step is possible only after the system has completed theidentification of a road sign, which implies four stages: segmentation, detection,recognition and tracking. Specifically, in [7] the author describes all of thesesteps.Road Sign Analysis Using Multisensory Data 253Fig. 2. System flow diagram3 Traffic Sign Parameters EstimationThere are at least 350 of different road sign types. As we have described inthe previous section, our system allows to detect and identify road signs in asequence of video. This kind of systems require detailed knowledge of the sign,with the additional purpose of reducing false alarms and to make an inventorynot only with the type of the found road sign during the route, but also with someparameters of each sign: height, dimensions, visibility distance, partial occlusion,GPS position, distance of separation between them. For an inventory system,these kind of data could be used to verify if the dimensions are standard, if twoconsecutive road signs are too closer, or if it exists any kind of partial occlusionwhich complicates that a driver can perceive the signal.To obtain these estimations of parameters we have considered a pin-hole ca-mera model, as shows Fig. 3, where they are known the following parameters:the focal length (f), size of each unit cell (uch and ucw, which are the heightand width of a cell, respectively) in the Charge-Coupled Device (CCD) of thecamera, the height of the camera (hc), the height of the middle point of a cap-tured image (hm), and the distance between the camera plane and the middlepoint plane of a captured image dm. Then, before starting to record a route, wehave to complete a calibration step, which consists in to measure hc, hm and dm.We develop this step on a known area, where we can measure these distanceswithout difficulties.254 R.J. López-Sastre et al.Fig. 3. Calibration of the system3.1 Height EstimationOur first objective is to estimate the height of the sign. Ideally, we could considerthat the plane where is the road sign, and the plane where is projected the imagein the camera, are parallel, but it does not happen in a real situation, becausethe sign or the camera, or both, can be inclined. We have to take into accountthis consideration to design a model which lets us to estimate the height of anobject. Anyway, we must to define the model where camera and sign plane areparallel, and then, we will extend this explanation to the real situation.Fig. 4. Ideal situation where camera and road sign plane are parallelFigure 4 shows the created model to estimate the height of a road sign. Afterthe detection and recognition process, we have the position of the identifiedRoad Sign Analysis Using Multisensory Data 255road sign in the image hsi = hspuch, where hsp is the height in pixels and uchis the height in mm of each pixel. The Positioning subsystem gives the covereddistance between two images where the recognition subsystem had identified thesame road sign. Then, knowing these data and that hc = hm we can establishthattan (β) =hsif, (1)tan (β) =hs − hcd + x, (2)where x is the covered distance between two consecutive captured images wherethe signal had been recognized. For the second captured image we can definetan (β′) =h′sif, (3)tan (β′) =hs − hcd. (4)From both pair of equations we can writehsif=hs − hcd + x, (5)h′sif=hs − hcd. (6)These last two equations form the following system of equationshs −h′sifd = hc , (7)hs −hsifd = hc +hsifx , (8)where the unknowns are hs and d.Our objective is to obtain the height of the road sign hs which ishs = hc +h′sihsixf (h′si − hsi). (9)We only have considered the situation in which the road sign is over the middlepoint of the captured image. Of course, we have two possible situations more.The traffic sign can be at the same height that the middle point, then hs = hm.The last situation is that hs < hm, and then we can writetan (β) =hsif, (10)tan (β) =hc − hsd + x, (11)256 R.J. López-Sastre et al.tan (β′) =h′sif, (12)tan (β′) =hc − hsd. (13)Now hs is defined as followshs = hc −h′sihsixf (h′si − hsi). (14)In a real approach we must consider that the road sign and the camera planeare not parallel. Figure 5 shows this situation. Now, the angle α = 0, and in afirst stage of calibration we must measure this angle byα = arctan(hm − hcdm). (15)Fig. 5. Real situation where camera and road sign plane are not parallelFrom this point, with α > 0, we can defineβ = arctan(hsif), (16)tan (α + β) =hs − hcd + x. (17)For the next captured image we haveβ′ = arctan(h′sif), (18)Road Sign Analysis Using Multisensory Data 257tan (α + β′) =hs − hcd. (19)Then, we have the following system of equationshs − tan (α + β) d = hc + tan (α + β) x , (20)hs − tan (α + β′) d = hc . (21)From this point, following the same manner of reasoning that in the ideal situa-tion, we will have that hs must be estimated as followshs = hc −x tan (α + β) tan (α + β′)tan (α + β) − tan (α + β′) , (22)Also, if we consider the situation where hs < hm, then hs will be estimatedashs = hc −x tan (α − β) tan (α − β′)tan (α − β) − tan (α − β′) . (23)This mathematical study, about the trigonometric relationships between the realworld and the projected image in a camera, lets us to estimate the height of asign successfully.3.2 Dimensions EstimationTo estimate the dimensions of a road sign is a task which is related with the typeof the sign. We must know, before the dimensions estimation step, the answer tothe following question: what kind of traffic sign do we have recognized?. With thisinformation, we only have to apply the same algorithm that we have described inthe previous section, but not only for the botton point of the sign. The detectionand recognition subsystem gives us the vertexes of the rectangle where the roadsign is confined. For a correct dimensions estimation, we have to estimate theheight of the top and botton corner of this rectangle. Then, we can compute allthe dimensions for every type of road sign.3.3 Visibility Distance EstimationWe define the visibility distance as the distance where the road sign has beendetected for the first time by our system. This parameter is crucial for andinventory system which wants to measure the level of safety of a road. To estimatethis parameter, we have to take up again the approach to estimate the height ofa road sign. Figure 4 shows the model for a ideal situation where road sign andcamera plane are parallel. Equations (7) and (8) compose a systems of equations,where the unknowns are d and hs. We can work out the value of d, which is thevisibility distance that we are searching.d =hsixh′si − hsi. (24)258 R.J. López-Sastre et al.In a real situation we have to use the model presented in Fig. 5. Then, we canestimate the visibility distance asd =tan (α + β)xtan (α + β′) − tan (α + β) if hs > hm (25)d =tan (α − β)xtan (α − β′) − tan (α − β) if hs < hm (26)3.4 Partial Occlusion EstimationMany times, traffic signs appear occluded by other objects like trees, vehicles,other road signs, etc. This traffic signs analysis system gives an estimation ofpartial occlusions, and to have this information of each sign results crucial for acomplete inventory system which measures the level of maintenance of a road,including the safety. To estimate this parameter, we use the information providedby the tracking subsystem: if the same road sign is detected in non-consecutiveframes, we can determine that a partial occlusion was happen.4 ResultsThe current data of several traffic signs have been obtained for a sequence ina two ways road. For testing our approach we have measured the height anddimensions of some road signs, which are presented in Fig. 6.(a) (b) (c)(d) (e)Fig. 6. Some detected and recognized road signsRoad Sign Analysis Using Multisensory Data 259Table 1. Parameters estimation for some road signsRoad Sign Real Height (m) Estimated Height (m) Error (%)a) 1.95 1.90 2.5b) 2.16 2.13 1.38c) 1.84 1.81 1.6d) 1.92 1.87 2.6Our test sequences of images have been recorded with two firewire camerasfixed onto the hood and the roof of the car. The captured images have a resolutionof 640× 480, and in a calibration step we have measured hc, hm, f , dm and uch.Figure 6 shows in image e) a milestone detected by our system, and Table 1shows the estimated height for each type of sign and the error commited.We can observe that the error obtained is not too high, and the parameterestimated is closer to the real value. The worse the measurement in the cali-bration step, the larger the error in the estimation. The most important causeof error in the estimation process is the measurement of the variable x, whichindicates the covered distance between two captured images. The Positioningsubsystem refreshes this variable each 1.6 meters, because this is the resolutionof our DMD, then we do not have the exact value of this parameter. The pre-sented mathematical procedure works well when all the variables are measuredcorrectly, and the error obtained in this situation is about 0.8%. But the error inthis estimation can be caused by some other factors: lens distortion, an incorrectdetection of the sign in the image. The dimensions of each sign are estimatedfollowing the same process, but for two point of the signal, as we have describedin the previous sections.Table 2 shows some estimated visibility distances.Table 2. Visibility Distance EstimationRoad Sign Visibility Distance (m)a) 41.2b) 37.6c) 42.7d) 21.65 ConclusionsThis paper describes a complete framework to estimate height, dimensions, par-tial occlusions and visibility distance of traffic signs. The integration of the recog-nition, tracking and positioning subsystems, allows to estimate these parametersautomatically, following the mathematical approach presented from this work.260 R.J. López-Sastre et al.Future lines of work will focus on estimating more attributes of each sign,such as deformation level, distance from the sign to the margin of the road, levelof slope. On the other hand, we will work in a more complete system whichwill have calibrated cameras and a Positioning subsystems which improves theestimation of covered distance. Another future objective will be to complete thesystem with an stereo vision framework, in order to improve these estimationsof parameters.AcknowledgmentThis work was supported be the project of the Ministerio de Educación y Cienciaof Spain number TEC2004/03511/TCM.References1. Maldonado, S., Lafuente, S., Gil, P., Gómez, H., López, F.: Road-sign detectionand recognition based on support vector machines. IEEE Trans. on IntelligentTransportation Systems (2006)2. de la Escalera, A., Armingol, J.M., Pastor, J.M., Rodŕıguez, F.J.: Visual sign infor-mation extraction and identification by deformable models for intelligent vehicles.IEEE Trans. on Intelligent Transportation Systems 15 (2004) 57–683. Fang, C., Chen, S.: Road sign detection and tracking. IEEE Trans. on VehicularTechnology 52 (2003) 1329–13414. Lafuente Arroyo, S., Garćıa Dı́az, P., Acevedo Rodŕıguez, F., Gil Jiménez, P., Mal-donado Bascón, S.: Traffic signs classification invariant to rotations using supportvector machines. In: Proceedings of Advabced Concepts for Intelligent Vision Sys-tems, Brussels, Belgium (2004)5. de la Escalera, A., Moreno, L.E., Salichs, M.A., Armingol, J.M.: Road traffic signdetection and classification. IEEE Trans. on Industrial electronics 44 (1997) 848–8596. Aoyagui, Y., Asakura: A study on traffic sign recognition in scene image usinggenetic alhorithms and neural networks. In: Proceedings IEEE Int. Conf. IndustrialElectronics, Control and Instrumentation. Volume 3., Taipei, Taiwan (1996) 1838–18437. Lafuente-Arroyo, S., Maldonado-Bascón, S., Gil-Jiménez, P., Gómez-Moremo, H.,López-Ferreras, F.: Road sign tracking with a predictive filter solution. In: Proc. ofIEEE IECON, Paris, France (2006) 3314–3319

Road Sign Analysis Using Multisensory Data

https://ebuah.uah.es/dspace/bitstream/10017/1271/1/lopez2007b.pdf

Road Sign Analysis Using Multisensory Data

Abstract

Similar works

Full text

Available Versions

e_Buah - Biblioteca Digital de la Universidad de Alcalá