Playing for Data: Ground Truth from Computer Games

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.Comment: Accepted to the 14th European Conference on Computer Vision (ECCV 2016

Real-time architecture for robust motion estimation under varying illumination conditions

Motion estimation from image sequences is a complex problem which requires high computing resources and is highly affected by changes in the illumination conditions in most of the existing approaches. In this contribution we present a high performance system that deals with this limitation. Robustness to varying illumination conditions is achieved by a novel technique that combines a gradient-based optical flow method with a non-parametric image transformation based on the Rank transform. The paper describes this method and quantitatively evaluates its robustness to different illumination changing patterns. This technique has been successfully implemented in a real-time system using reconfigurable hardware. Our contribution presents the computing architecture, including the resources consumption and the obtained performance. The final system is a real-time device capable to computing motion sequences in real-time even in conditions with significant illumination changes. The robustness of the proposed system facilitates its use in multiple potential application fields.This work has been supported by the grants DEPROVI (DPI2004-07032), DRIVSCO (IST-016276-2) and TIC2007:”Plataforma Sw-Hw para sistemas de visión 3D en tiempo real”

Virtual Reality to Simulate Visual Tasks for Robotic Systems

Virtual reality (VR) can be used as a tool to analyze the interactions between the visual system of a robotic agent and the environment, with the aim of designing the algorithms to solve the visual tasks necessary to properly behave into the 3D world. The novelty of our approach lies in the use of the VR as a tool to simulate the behavior of vision systems. The visual system of a robot (e.g., an autonomous vehicle, an active vision system, or a driving assistance system) and its interplay with the environment can be modeled through the geometrical relationships between the virtual stereo cameras and the virtual 3D world. Differently from conventional applications, where VR is used for the perceptual rendering of the visual information to a human observer, in the proposed approach, a virtual world is rendered to simulate the actual projections on the cameras of a robotic system. In this way, machine vision algorithms can be quantitatively validated by using the ground truth data provided by the knowledge of both the structure of the environment and the vision system

Performance Characterization of Watson Ahumada Motion Detector Using Random Dot Rotary Motion Stimuli

The performance of Watson & Ahumada's model of human visual motion sensing is compared against human psychophysical performance. The stimulus consists of random dots undergoing rotary motion, displayed in a circular annulus. The model matches psychophysical observer performance with respect to most parameters. It is able to replicate some key psychophysical findings such as invariance of observer performance to dot density in the display, and decrease of observer performance with frame duration of the display

Assessment and application of wavelet-based optical flow velocimetry (wOFV) to wall-bounded turbulent flows

The performance of a wavelet-based optical flow velocimetry (wOFV) algorithm to extract high accuracy and high resolution velocity fields from particle images in wall-bounded turbulent flows is assessed. wOFV is first evaluated using synthetic particle images generated from a channel flow DNS of a turbulent boundary layer. The sensitivity of wOFV to the regularization parameter (lambda) is quantified and results are compared to PIV. Results on synthetic particle images indicated different sensitivity to under-regularization or over-regularization depending on which region of the boundary layer is analyzed. Synthetic data revealed that wOFV can modestly outperform PIV in vector accuracy across a broad lambda range. wOFV showed clear advantages over PIV in resolving the viscous sublayer and obtaining highly accurate estimates of the wall shear stress. wOFV was also applied to experimental data of a developing turbulent boundary layer. Overall, wOFV revealed good agreement with both PIV and PIV + PTV. However, wOFV was able to successfully resolve the wall shear stress and correctly normalize the boundary layer streamwise velocity to wall units where PIV and PIV + PTV showed larger deviations. Analysis of the turbulent velocity fluctuations revealed spurious results for PIV in close proximity to the wall, leading to significantly exaggerated and non-physical turbulence intensity. PIV + PTV showed a minor improvement in this aspect. wOFV did not exhibit this same effect, revealing that it is more accurate in capturing small-scale turbulent motion in the vicinity of boundaries. The enhanced vector resolution of wOFV enabled improved estimation of instantaneous derivative quantities and intricate flow structure both closer to the wall. These aspects show that, within a reasonable lambda range, wOFV can improve resolving the turbulent motion occurring in the vicinity of physical boundaries