4 research outputs found
DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation through SwaftNet for Surrounding Sensing
Semantically interpreting the traffic scene is crucial for autonomous
transportation and robotics systems. However, state-of-the-art semantic
segmentation pipelines are dominantly designed to work with pinhole cameras and
train with narrow Field-of-View (FoV) images. In this sense, the perception
capacity is severely limited to offer higher-level confidence for upstream
navigation tasks. In this paper, we propose a network adaptation framework to
achieve Panoramic Annular Semantic Segmentation (PASS), which allows to re-use
conventional pinhole-view image datasets, enabling modern segmentation networks
to comfortably adapt to panoramic images. Specifically, we adapt our proposed
SwaftNet to enhance the sensitivity to details by implementing attention-based
lateral connections between the detail-critical encoder layers and the
context-critical decoder layers. We benchmark the performance of efficient
segmenters on panoramic segmentation with our extended PASS dataset,
demonstrating that the proposed real-time SwaftNet outperforms state-of-the-art
efficient networks. Furthermore, we assess real-world performance when
deploying the Detail-Sensitive PASS (DS-PASS) system on a mobile robot and an
instrumented vehicle, as well as the benefit of panoramic semantics for visual
odometry, showing the robustness and potential to support diverse navigational
applications.Comment: 8 pages, 10 figure
See Clearer at Night: Towards Robust Nighttime Semantic Segmentation through Day-Night Image Conversion
Currently, semantic segmentation shows remarkable efficiency and reliability
in standard scenarios such as daytime scenes with favorable illumination
conditions. However, in face of adverse conditions such as the nighttime,
semantic segmentation loses its accuracy significantly. One of the main causes
of the problem is the lack of sufficient annotated segmentation datasets of
nighttime scenes. In this paper, we propose a framework to alleviate the
accuracy decline when semantic segmentation is taken to adverse conditions by
using Generative Adversarial Networks (GANs). To bridge the daytime and
nighttime image domains, we made key observation that compared to datasets in
adverse conditions, there are considerable amount of segmentation datasets in
standard conditions such as BDD and our collected ZJU datasets. Our GAN-based
nighttime semantic segmentation framework includes two methods. In the first
method, GANs were used to translate nighttime images to the daytime, thus
semantic segmentation can be performed using robust models already trained on
daytime datasets. In another method, we use GANs to translate different ratio
of daytime images in the dataset to the nighttime but still with their labels.
In this sense, synthetic nighttime segmentation datasets can be generated to
yield models prepared to operate at nighttime conditions robustly. In our
experiment, the later method significantly boosts the performance at the
nighttime evidenced by quantitative results using Intersection over Union (IoU)
and Pixel Accuracy (Acc). We show that the performance varies with respect to
the proportion of synthetic nighttime images in the dataset, where the sweet
spot corresponds to most robust performance across the day and night.Comment: 13 pages, 7 figures, 2 tables, 2 equations. Artificial Intelligence
and Machine Learning in Defense Applications, SPIE Security + Defence 2019,
Strasbourg, France, September 201
Semantic Segmentation of Panoramic Images Using a Synthetic Dataset
Panoramic images have advantages in information capacity and scene stability
due to their large field of view (FoV). In this paper, we propose a method to
synthesize a new dataset of panoramic image. We managed to stitch the images
taken from different directions into panoramic images, together with their
labeled images, to yield the panoramic semantic segmentation dataset
denominated as SYNTHIA-PANO. For the purpose of finding out the effect of using
panoramic images as training dataset, we designed and performed a comprehensive
set of experiments. Experimental results show that using panoramic images as
training data is beneficial to the segmentation result. In addition, it has
been shown that by using panoramic images with a 180 degree FoV as training
data the model has better performance. Furthermore, the model trained with
panoramic images also has a better capacity to resist the image distortion.Comment: 15 pages, 12 figures, SPIE Security + Defence International Symposiu
Reinforced Wasserstein Training for Severity-Aware Semantic Segmentation in Autonomous Driving
Semantic segmentation is important for many real-world systems, e.g.,
autonomous vehicles, which predict the class of each pixel. Recently, deep
networks achieved significant progress w.r.t. the mean Intersection-over Union
(mIoU) with the cross-entropy loss. However, the cross-entropy loss can
essentially ignore the difference of severity for an autonomous car with
different wrong prediction mistakes. For example, predicting the car to the
road is much more servery than recognize it as the bus. Targeting for this
difficulty, we develop a Wasserstein training framework to explore the
inter-class correlation by defining its ground metric as misclassification
severity. The ground metric of Wasserstein distance can be pre-defined
following the experience on a specific task. From the optimization perspective,
we further propose to set the ground metric as an increasing function of the
pre-defined ground metric. Furthermore, an adaptively learning scheme of the
ground matrix is proposed to utilize the high-fidelity CARLA simulator.
Specifically, we follow a reinforcement alternative learning scheme. The
experiments on both CamVid and Cityscapes datasets evidenced the effectiveness
of our Wasserstein loss. The SegNet, ENet, FCN and Deeplab networks can be
adapted following a plug-in manner. We achieve significant improvements on the
predefined important classes, and much longer continuous playtime in our
simulator.Comment: Accepted to IEEE Transactions on Intelligent Transportation Systems
(T-ITS