Dimitrievski, Martin

Jacobs, Lennert

Philips, Wilfried

Steendam, Heidi

Van Hamme, David

Veelaert, Peter

Ghent University Academic Bibliography

1Tracking Road Users by Cooperative Fusion ofRadar and Camera SensorsMartin Dimitrievski∗, David Van Hamme†, Lennert Jacobs‡, Peter Veelaert§, Heidi Steendam¶ and WilfriedPhilips‖TELIN-IPI, Ghent University - imec,St-Pietersnieuwstraat 41, B-9000 Gent, BelgiumEmail: ∗martin.dimitrievski@ugent.be, †david.vanhamme@ugent.be, ‡lennert.jacobs@ugent.be,§peter.veelaert@ugent.be, ¶heidi.steendam@ugent.be, ‖wilfried.philips@ugent.beAbstract—In this paper, we present the results and the lessonslearned during development of an intelligent automotive percep-tion system based on fusion of camera and radar. Our cooperativesensor fusion approach closely couples the detector and tracker,which allows to exploit the benefits of low-level fusion of the richsensor data without the need for high data bandwidth to thefusion center. To demonstrate the accuracy of our perceptionsystem, we captured a dataset consisting of various complextraffic scenarios in a European city center. For multiple targetsup to 10 m in front of the ego-vehicle, we measure accuracyimprovements of 20% over the camera-only system, with only afractional load to the network compared to low-level fusion.I. MOTIVATION AND SIGNIFICANCE OF THE TOPICAccurate detection and tracking of road users is essentialfor driver-less cars and many other smart mobility applica-tions. As no single sensor can provide the required accuracyand robustness, the output from several sensors needs to becombined. If we exclude practical limitations such as cost andease of integration, each sensor technology (e.g., radar, video,LiDAR, ultrasound) still has its own intrinsic limitations. Forinstance, cameras don’t work well at night-time, or in dazzlingsunlight. Radar can be confused by reflective metal objects,like rubbish bins or soda cans. LiDAR technology is affectedby atmospheric conditions that increase the light scattering ofthe air such as precipitation, mist, fog or fine dust. Fusingthe output of these different sensors is thus very important.While both vision based [1], [2] and ranging based [3] trackingof road users are mature research fields well described inliterature, fusion techniques between the two specifically forthe automotive context are much less covered in literature.Currently, sensor fusion happens at a relatively late stage,after each sensor has performed object detection based onits own limited collection of sensor data. Such late fusionhas obvious benefits on the system level, both in terms ofease of integration and robustness. However, a lot of sensorfusion potential is lost, especially in circumstances where onesensor underperforms compared to another. Employing so-called early or low-level fusion on the rich data gatheredfrom all sensors could improve the performance, but at thesame time significantly increases the required data bandwidthand processing power. For instance, the GoPro camera and TIAWR1443 automotive radar used in our experimental setuphave a raw data rate of about 30 Mbps (at 30 fps, using videocompression) and 160 Mbps (at 20 fps, uncompressed), respec-tively. A single contemporary automotive 3D LiDAR producesup to 9.6M points per second with data rates of up to 500Mbps.1 Despite these large numbers, the amount of sensorsadded to smart vehicles is expected to dramatically increase inorder to reach higher levels of autonomy. In addition, vehicle-to-everything (V2X) communication, which allows, e.g., toshare traffic information or sensor data among vehicles, high-precision navigation systems, and advanced infotainment willalso contribute to the anticipated explosion of data bandwidthin future cars. For this reason, car manufacturers are currentlyadopting high-speed Ethernet-based networks in addition toconventional data buses like CAN and LIN. Whereas 100Mbps (100BASE-T1) Ethernet over single-pair UTP cables hasbeen applied in cars today, 1 Gbps (1000BASE-T1) Ethernetwill be introduced in next-generation car platforms, and newstandards for 2.5-10 Gbps automotive Ethernet are beingdeveloped [4]. Nevertheless, the massive amount of sensordata requires to offload computation as much as possible fromthe Fusion Center (FC) towards the edge nodes to save bothbandwidth and computation power. In this regard, we havedeveloped a cooperative sensor fusion approach which closelycouples the detector and tracker without the need for highdata bandwidth. The cooperative approach embodies fusionon two levels. Firstly the processing pipelines of different sen-sors directly exchange mid-level information such as regionsof interest, which allows the sensors to resolve ambiguitiesduring the detection process. These detections then contin-uously update or spawn stable tracks, providing consistentand accurate object localization. In cases where detectionor data association fails, the tracker evaluates the likelihoodof multiple hypotheses by communicating the positions ofparticles to the radar and camera processing modules. Thisway, the tracker-detector feedback loop bypasses the originaldetection thresholds and thereby unlocks the potential of low-level data fusion without losing the aforementioned benefitsof independent sensor pipelines on the system level.II. CONCLUSIONSIt is well established that detection of road users usingradar produces targets with excellent range, but poor angular1https://velodynelidar.com/vls-128.html2Figure 1. Sample from the data captured in the city of Ghent showcasing data interaction at the sensor level. Left: camera frame with objects detected byFaster R-CNN; Center: uncertainty regions on the ground plane generated by back-projecting CV objects; Right: uncertainty regions on the ground planefused with radar (averaged over all Doppler velocities).resolution. On the other hand, by using Computer Vision (CV)and projective geometry, objects detected by a camera sensorhave poor range, but excellent angular resolution. Naïvely, onecan expect that combining such complementary informationwill lead to dramatic performance increase over any singularsensor. However, during our study we found out that trackingperformance is greatly hindered by ambiguities that exist evenin the fused data. For example, in figure 1 we see a typicalsituation where multiple pedestrians and cyclists are detectedin the camera frame. These regions of interest (BoundingBoxes) are communicated to the radar processing module andprojected on the ground plane (e.g. using expected personheight) taking into account the uncertainty on these projectedlocations. While augmenting this ground plane map with theradar signal greatly reduces the range uncertainty in spatiallyseparated objects such as the cyclists in the background, it isclear that ambiguities still exist.Candidate targets are obtained at the edge nodes by employ-ing Constant False Alarm Rate (CFAR) detection on the fuseddata, thereby avoiding transmission of raw data to the FC. Thealgorithm extracts detections with high signal-to-noise ratioswhich are then fed into our multiple object tracker. A particlefilter maximizes the belief in the state of each road user inan iterative prediction/update cycle. We have shown that inthe context of autonomous driving, particle filters [5] are wellsuited for tracking unpredictable pedestrian motion by keepingmultiple hypotheses. State updates are done using a joint-likelihood observation model. On the other hand, when tracksfail to associate with detections, or there are no detectionsabove a required threshold, most trackers in the literaturerely only on the motion model to predict the next state. Thistracking scenario occurs often in ambiguous situations, e.g. aperson is unobserved for multiple frames and can appear atmultiple locations, thus the probability density map of theirlocation is multi-modal. Our tracker is different because itcan intelligently update unassociated tracks by evaluating eachhypothesis (particle) over the low-level camera-radar data. Wecall these sub-threshold or soft-associations where the weightof each particle is inversely-proportional to the uncertainty inthe camera-radar data, figure 1 right.For each uncertain association, the particle filter com-municates 1000 predicted particles to the sensor processingmodules, with each particle consisting of 4 parameters (2Dposition and velocity). For a single track, we observe a peakbit rate of 2.4 Mbps streaming from the FC to the sensorsand 0.6 Mbps worth of likelihoods streamed in the otherdirection. This is rare occurrence though, as most of thetargets can be continuously updated with observations andvery rarely does the tracker need to evaluate likelihoodsof the full particle cloud. Experimental results on trackingvulnerable road users (pedestrians and cyclists) show that thebenefits of fusing camera with radar increase with the decreaseof scene complexity. We observed that the most significantimprovement in accuracy occurs when there is a single personin front of the ego-vehicle with localization errors improvingby 50% over the camera-only system. On the other hand, whenthe scene becomes very complex, smaller performance gainsare observed. Nonetheless, we measured improvements of 20%for multiple targets up to 10 m in front of the ego-vehicle.These results are remarkable since they show that although theradar cross section of a pedestrian is relatively small comparedto clutter such as fences, light poles etc. our smart system canextract more information from the two sensors exerting onlya fractional load to the network compared to low-level fusion.REFERENCES[1] S. Sivaraman and M. M. Trivedi, “Looking at vehicles on the road: Asurvey of vision-based vehicle detection, tracking, and behavior analy-sis,” IEEE Transactions on Intelligent Transportation Systems, vol. 14,pp. 1773–1795, Dec 2013.[2] A. W. M. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan,and M. Shah, “Visual tracking: An experimental survey,” IEEE Transac-tions on Pattern Analysis and Machine Intelligence, vol. 36, pp. 1442–1468, July 2014.[3] S. M. Patole, M. Torlak, D. Wang, and M. Ali, “Automotive radars:A review of signal processing techniques,” IEEE Signal ProcessingMagazine, vol. 34, pp. 22–35, March 2017.[4] G. W. d. Besten, “30.1 single-pair automotive phy solutions from 10mb/sto 10gb/s and beyond,” in 2019 IEEE International Solid- State CircuitsConference - (ISSCC), pp. 474–476, Feb 2019.[5] M. Dimitrievski, P. Veelaert, and W. Philips, “Behavioral pedestriantracking using a camera and lidar sensors on a moving vehicle,” Sensors,vol. 19, no. 2, 2019.

Tracking road users by cooperative fusion of radar and camera sensors

https://biblio.ugent.be/publication/8635644/file/8635649.pdf

Tracking road users by cooperative fusion of radar and camera sensors

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography