In this paper, we focus on exploring effective methods for faster and
accurate semantic segmentation. A common practice to improve the performance is
to attain high-resolution feature maps with strong semantic representation. Two
strategies are widely used: atrous convolutions and feature pyramid fusion,
while both are either computationally intensive or ineffective. Inspired by the
Optical Flow for motion alignment between adjacent video frames, we propose a
Flow Alignment Module (FAM) to learn \textit{Semantic Flow} between feature
maps of adjacent levels and broadcast high-level features to high-resolution
features effectively and efficiently. Furthermore, integrating our FAM to a
standard feature pyramid structure exhibits superior performance over other
real-time methods, even on lightweight backbone networks, such as ResNet-18 and
DFNet. Then to further speed up the inference procedure, we also present a
novel Gated Dual Flow Alignment Module to directly align high-resolution
feature maps and low-resolution feature maps where we term the improved version
network as SFNet-Lite. Extensive experiments are conducted on several
challenging datasets, where results show the effectiveness of both SFNet and
SFNet-Lite. In particular, when using Cityscapes test set, the SFNet-Lite
series achieve 80.1 mIoU while running at 60 FPS using ResNet-18 backbone and
78.8 mIoU while running at 120 FPS using STDC backbone on RTX-3090. Moreover,
we unify four challenging driving datasets into one large dataset, which we
named Unified Driving Segmentation (UDS) dataset. It contains diverse domain
and style information. We benchmark several representative works on UDS. Both
SFNet and SFNet-Lite still achieve the best speed and accuracy trade-off on
UDS, which serves as a strong baseline in such a challenging setting. The code
and models are publicly available at https://github.com/lxtGH/SFSegNets.Comment: IJCV-2023; Extension of Previous work arXiv:2002.1012