1,109 research outputs found
Modeling Camera Effects to Improve Visual Learning from Synthetic Data
Recent work has focused on generating synthetic imagery to increase the size
and variability of training data for learning visual tasks in urban scenes.
This includes increasing the occurrence of occlusions or varying environmental
and weather effects. However, few have addressed modeling variation in the
sensor domain. Sensor effects can degrade real images, limiting
generalizability of network performance on visual tasks trained on synthetic
data and tested in real environments. This paper proposes an efficient,
automatic, physically-based augmentation pipeline to vary sensor effects
--chromatic aberration, blur, exposure, noise, and color cast-- for synthetic
imagery. In particular, this paper illustrates that augmenting synthetic
training datasets with the proposed pipeline reduces the domain gap between
synthetic and real domains for the task of object detection in urban driving
scenes
Rain rendering for evaluating and improving robustness to bad weather
Rain fills the atmosphere with water particles, which breaks the common
assumption that light travels unaltered from the scene to the camera. While it
is well-known that rain affects computer vision algorithms, quantifying its
impact is difficult. In this context, we present a rain rendering pipeline that
enables the systematic evaluation of common computer vision algorithms to
controlled amounts of rain. We present three different ways to add synthetic
rain to existing images datasets: completely physic-based; completely
data-driven; and a combination of both. The physic-based rain augmentation
combines a physical particle simulator and accurate rain photometric modeling.
We validate our rendering methods with a user study, demonstrating our rain is
judged as much as 73% more realistic than the state-of-theart. Using our
generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a
thorough evaluation of object detection, semantic segmentation, and depth
estimation algorithms and show that their performance decreases in degraded
weather, on the order of 15% for object detection, 60% for semantic
segmentation, and 6-fold increase in depth estimation error. Finetuning on our
augmented synthetic data results in improvements of 21% on object detection,
37% on semantic segmentation, and 8% on depth estimation.Comment: 19 pages, 19 figures, IJCV 2020 preprint. arXiv admin note: text
overlap with arXiv:1908.1033
GTAV-NightRain: Photometric Realistic Large-scale Dataset for Night-time Rain Streak Removal
Rain is transparent, which reflects and refracts light in the scene to the
camera. In outdoor vision, rain, especially rain streaks degrade visibility and
therefore need to be removed. In existing rain streak removal datasets,
although density, scale, direction and intensity have been considered,
transparency is not fully taken into account. This problem is particularly
serious in night scenes, where the appearance of rain largely depends on the
interaction with scene illuminations and changes drastically on different
positions within the image. This is problematic, because unrealistic dataset
causes serious domain bias. In this paper, we propose GTAV-NightRain dataset,
which is a large-scale synthetic night-time rain streak removal dataset. Unlike
existing datasets, by using 3D computer graphic platform (namely GTA V), we are
allowed to infer the three dimensional interaction between rain and
illuminations, which insures the photometric realness. Current release of the
dataset contains 12,860 HD rainy images and 1,286 corresponding HD ground truth
images in diversified night scenes. A systematic benchmark and analysis are
provided along with the dataset to inspire further research
Rain Removal in Traffic Surveillance: Does it Matter?
Varying weather conditions, including rainfall and snowfall, are generally
regarded as a challenge for computer vision algorithms. One proposed solution
to the challenges induced by rain and snowfall is to artificially remove the
rain from images or video using rain removal algorithms. It is the promise of
these algorithms that the rain-removed image frames will improve the
performance of subsequent segmentation and tracking algorithms. However, rain
removal algorithms are typically evaluated on their ability to remove synthetic
rain on a small subset of images. Currently, their behavior is unknown on
real-world videos when integrated with a typical computer vision pipeline. In
this paper, we review the existing rain removal algorithms and propose a new
dataset that consists of 22 traffic surveillance sequences under a broad
variety of weather conditions that all include either rain or snowfall. We
propose a new evaluation protocol that evaluates the rain removal algorithms on
their ability to improve the performance of subsequent segmentation, instance
segmentation, and feature tracking algorithms under rain and snow. If
successful, the de-rained frames of a rain removal algorithm should improve
segmentation performance and increase the number of accurately tracked
features. The results show that a recent single-frame-based rain removal
algorithm increases the segmentation performance by 19.7% on our proposed
dataset, but it eventually decreases the feature tracking performance and
showed mixed results with recent instance segmentation methods. However, the
best video-based rain removal algorithm improves the feature tracking accuracy
by 7.72%.Comment: Published in IEEE Transactions on Intelligent Transportation System
Gradual Network for Single Image De-raining
Most advances in single image de-raining meet a key challenge, which is
removing rain streaks with different scales and shapes while preserving image
details. Existing single image de-raining approaches treat rain-streak removal
as a process of pixel-wise regression directly. However, they are lacking in
mining the balance between over-de-raining (e.g. removing texture details in
rain-free regions) and under-de-raining (e.g. leaving rain streaks). In this
paper, we firstly propose a coarse-to-fine network called Gradual Network
(GraNet) consisting of coarse stage and fine stage for delving into single
image de-raining with different granularities. Specifically, to reveal
coarse-grained rain-streak characteristics (e.g. long and thick rain
streaks/raindrops), we propose a coarse stage by utilizing local-global spatial
dependencies via a local-global subnetwork composed of region-aware blocks.
Taking the residual result (the coarse de-rained result) between the rainy
image sample (i.e. the input data) and the output of coarse stage (i.e. the
learnt rain mask) as input, the fine stage continues to de-rain by removing the
fine-grained rain streaks (e.g. light rain streaks and water mist) to get a
rain-free and well-reconstructed output image via a unified contextual merging
sub-network with dense blocks and a merging block. Solid and comprehensive
experiments on synthetic and real data demonstrate that our GraNet can
significantly outperform the state-of-the-art methods by removing rain streaks
with various densities, scales and shapes while keeping the image details of
rain-free regions well-preserved.Comment: In Proceedings of the 27th ACM International Conference on Multimedia
(MM 2019
CARLA+: An Evolution of the CARLA Simulator for Complex Environment Using a Probabilistic Graphical Model
In an urban and uncontrolled environment, the presence of mixed traffic of autonomous vehicles, classical vehicles, vulnerable road users, e.g., pedestrians, and unprecedented dynamic events makes it challenging for the classical autonomous vehicle to navigate the traffic safely. Therefore, the realization of collaborative autonomous driving has the potential to improve road safety and traffic efficiency. However, an obvious challenge in this regard is how to define, model, and simulate the environment that captures the dynamics of a complex and urban environment. Therefore, in this research, we first define the dynamics of the envisioned environment, where we capture the dynamics relevant to the complex urban environment, specifically, highlighting the challenges that are unaddressed and are within the scope of collaborative autonomous driving. To this end, we model the dynamic urban environment leveraging a probabilistic graphical model (PGM). To develop the proposed solution, a realistic simulation environment is required. There are a number of simulators—CARLA (Car Learning to Act), one of the prominent ones, provides rich features and environment; however, it still fails on a few fronts, for example, it cannot fully capture the complexity of an urban environment. Moreover, the classical CARLA mainly relies on manual code and multiple conditional statements, and it provides no pre-defined way to do things automatically based on the dynamic simulation environment. Hence, there is an urgent need to extend the off-the-shelf CARLA with more sophisticated settings that can model the required dynamics. In this regard, we comprehensively design, develop, and implement an extension of a classical CARLA referred to as CARLA+ for the complex environment by integrating the PGM framework. It provides a unified framework to automate the behavior of different actors leveraging PGMs. Instead of manually catering to each condition, CARLA+ enables the user to automate the modeling of different dynamics of the environment. Therefore, to validate the proposed CARLA+, experiments with different settings are designed and conducted. The experimental results demonstrate that CARLA+ is flexible enough to allow users to model various scenarios, ranging from simple controlled models to complex models learned directly from real-world data. In the future, we plan to extend CARLA+ by allowing for more configurable parameters and more flexibility on the type of probabilistic networks and models one can choose. The open-source code of CARLA+ is made publicly available for researchers
- …