Lifting GIS Maps into Strong Geometric Context for Scene Understanding

Contextual information can have a substantial impact on the performance of visual tasks such as semantic segmentation, object detection, and geometric estimation. Data stored in Geographic Information Systems (GIS) offers a rich source of contextual information that has been largely untapped by computer vision. We propose to leverage such information for scene understanding by combining GIS resources with large sets of unorganized photographs using Structure from Motion (SfM) techniques. We present a pipeline to quickly generate strong 3D geometric priors from 2D GIS data using SfM models aligned with minimal user input. Given an image resectioned against this model, we generate robust predictions of depth, surface normals, and semantic labels. We show that the precision of the predicted geometry is substantially more accurate other single-image depth estimation methods. We then demonstrate the utility of these contextual constraints for re-scoring pedestrian detections, and use these GIS contextual features alongside object detection score maps to improve a CRF-based semantic segmentation framework, boosting accuracy over baseline models

A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects

Recently, Minimum Cost Multicut Formulations have been proposed and proven to be successful in both motion trajectory segmentation and multi-target tracking scenarios. Both tasks benefit from decomposing a graphical model into an optimal number of connected components based on attractive and repulsive pairwise terms. The two tasks are formulated on different levels of granularity and, accordingly, leverage mostly local information for motion segmentation and mostly high-level information for multi-target tracking. In this paper we argue that point trajectories and their local relationships can contribute to the high-level task of multi-target tracking and also argue that high-level cues from object detection and tracking are helpful to solve motion segmentation. We propose a joint graphical model for point trajectories and object detections whose Multicuts are solutions to motion segmentation {\it and} multi-target tracking problems at once. Results on the FBMS59 motion segmentation benchmark as well as on pedestrian tracking sequences from the 2D MOT 2015 benchmark demonstrate the promise of this joint approach

Real-time Physics Based Simulation for 3D Computer Graphics

Restoration of realistic animation is a critical part in the area of computer graphics. The goal of this sort of simulation is to imitate the behavior of the transformation in real life to the greatest extent. Physics-based simulation provides a solid background and proficient theories that can be applied in the simulation. In this dissertation, I will present real-time simulations which are physics-based in the area of terrain deformation and ship oscillations. When ground vehicles navigate on soft terrains such as sand, snow and mud, they often leave distinctive tracks. The realistic simulation of such vehicle-terrain interaction is important for ground based visual simulations and many video games. However, the existing research in terrain deformation has not addressed this issue effectively. In this dissertation, I present a new terrain deformation algorithm for simulating vehicle-terrain interaction in real time. The algorithm is based on the classic terramechanics theories, and calculates terrain deformation according to the vehicle load, velocity, tire size, and soil concentration. As a result, this algorithm can simulate different vehicle tracks on different types of terrains with different vehicle properties. I demonstrate my algorithm by vehicle tracks on soft terrain. In the field of ship oscillation simulation, I propose a new method for simulating ship motions in waves. Although there have been plenty of previous work on physics based fluid-solid simulation, most of these methods are not suitable for real-time applications. In particular, few methods are designed specifically for simulating ship motion in waves. My method is based on physics theories of ship motion, but with necessary simplifications to ensure real-time performance. My results show that this method is well suited to simulate sophisticated ship motions in real time applications

What Makes a Place? Building Bespoke Place Dependent Object Detectors for Robotics

This paper is about enabling robots to improve their perceptual performance through repeated use in their operating environment, creating local expert detectors fitted to the places through which a robot moves. We leverage the concept of 'experiences' in visual perception for robotics, accounting for bias in the data a robot sees by fitting object detector models to a particular place. The key question we seek to answer in this paper is simply: how do we define a place? We build bespoke pedestrian detector models for autonomous driving, highlighting the necessary trade off between generalisation and model capacity as we vary the extent of the place we fit to. We demonstrate a sizeable performance gain over a current state-of-the-art detector when using computationally lightweight bespoke place-fitted detector models.Comment: IROS 201