Pre-training is prevalent in nowadays deep learning to improve the learned
model's performance. However, in the literature on federated learning (FL),
neural networks are mostly initialized with random weights. These attract our
interest in conducting a systematic study to explore pre-training for FL.
Across multiple visual recognition benchmarks, we found that pre-training can
not only improve FL, but also close its accuracy gap to the counterpart
centralized learning, especially in the challenging cases of non-IID clients'
data. To make our findings applicable to situations where pre-trained models
are not directly available, we explore pre-training with synthetic data or even
with clients' data in a decentralized manner, and found that they can already
improve FL notably. Interesting, many of the techniques we explore are
complementary to each other to further boost the performance, and we view this
as a critical result toward scaling up deep FL for real-world applications. We
conclude our paper with an attempt to understand the effect of pre-training on
FL. We found that pre-training enables the learned global models under
different clients' data conditions to converge to the same loss basin, and
makes global aggregation in FL more stable. Nevertheless, pre-training seems to
not alleviate local model drifting, a fundamental problem in FL under non-IID
data.Comment: Preprin