Mono-Camera 3D Multi-Object Tracking Using Deep Learning Detections and PMBM Filtering

Monocular cameras are one of the most commonly used sensors in the automotive industry for autonomous vehicles. One major drawback using a monocular camera is that it only makes observations in the two dimensional image plane and can not directly measure the distance to objects. In this paper, we aim at filling this gap by developing a multi-object tracking algorithm that takes an image as input and produces trajectories of detected objects in a world coordinate system. We solve this by using a deep neural network trained to detect and estimate the distance to objects from a single input image. The detections from a sequence of images are fed in to a state-of-the art Poisson multi-Bernoulli mixture tracking filter. The combination of the learned detector and the PMBM filter results in an algorithm that achieves 3D tracking using only mono-camera images as input. The performance of the algorithm is evaluated both in 3D world coordinates, and 2D image coordinates, using the publicly available KITTI object tracking dataset. The algorithm shows the ability to accurately track objects, correctly handle data associations, even when there is a big overlap of the objects in the image, and is one of the top performing algorithms on the KITTI object tracking benchmark. Furthermore, the algorithm is efficient, running on average close to 20 frames per second.Comment: 8 pages, 2 figures, for associated videos, see https://goo.gl/Aoydg

Ego-motion and Surrounding Vehicle State Estimation Using a Monocular Camera

Understanding ego-motion and surrounding vehicle state is essential to enable automated driving and advanced driving assistance technologies. Typical approaches to solve this problem use fusion of multiple sensors such as LiDAR, camera, and radar to recognize surrounding vehicle state, including position, velocity, and orientation. Such sensing modalities are overly complex and costly for production of personal use vehicles. In this paper, we propose a novel machine learning method to estimate ego-motion and surrounding vehicle state using a single monocular camera. Our approach is based on a combination of three deep neural networks to estimate the 3D vehicle bounding box, depth, and optical flow from a sequence of images. The main contribution of this paper is a new framework and algorithm that integrates these three networks in order to estimate the ego-motion and surrounding vehicle state. To realize more accurate 3D position estimation, we address ground plane correction in real-time. The efficacy of the proposed method is demonstrated through experimental evaluations that compare our results to ground truth data available from other sensors including Can-Bus and LiDAR

Using Panoramic Videos for Multi-person Localization and Tracking in a 3D Panoramic Coordinate

3D panoramic multi-person localization and tracking are prominent in many applications, however, conventional methods using LiDAR equipment could be economically expensive and also computationally inefficient due to the processing of point cloud data. In this work, we propose an effective and efficient approach at a low cost. First, we obtain panoramic videos with four normal cameras. Then, we transform human locations from a 2D panoramic image coordinate to a 3D panoramic camera coordinate using camera geometry and human bio-metric property (i.e., height). Finally, we generate 3D tracklets by associating human appearance and 3D trajectory. We verify the effectiveness of our method on three datasets including a new one built by us, in terms of 3D single-view multi-person localization, 3D single-view multi-person tracking, and 3D panoramic multi-person localization and tracking. Our code and dataset are available at \url{https://github.com/fandulu/MPLT}.Comment: 5 page

Spatiotemporal Constraints for Sets of Trajectories with Applications to PMBM Densities

In this paper we introduce spatiotemporal constraints for trajectories, i.e., restrictions that the trajectory must be in some part of the state space (spatial constraint) at some point in time (temporal constraint). Spatiotemporal contraints on trajectories can be used to answer a range of important questions, including, e.g., "where did the person that were in area A at time t, go afterwards?". We discuss how multiple constraints can be combined into sets of constraints, and we then apply sets of constraints to set of trajectories densities, specifically Poisson Multi-Bernoulli Mixture (PMBM) densities. For Poisson target birth, the exact posterior density is PMBM for both point targets and extended targets. In the paper we show that if the unconstrained set of trajectories density is PMBM, then the constrained density is also PMBM. Examples of constrained trajectory densities motivate and illustrate the key results