Jointly exploring client drift and catastrophic forgetting in dynamic learning

Abstract

Federated and Continual Learning have emerged as promising paradigms for the privacy-aware use of Deep Learning in dynamic environments by addressing spatial and temporal constraints on data availability. However, Client Drift and Catastrophic Forgetting are fundamental obstacles to ensuring robust performance. Existing work only addresses these problems separately, neglecting the fact that the root cause behind them, namely an unexpected shift in the data distribution, is connected. We propose a unified analysis framework for building a controlled test environment where we can jointly model spatial and temporal shifts, more closely emulating real dynamic settings. By generating a 3D landscape of the combined performance impact, we show that a moderate combination of both shifts can even improve the performance of the resulting model ("Generalization Bump"). We apply a simple and commonly used method from continual learning in the federated setting and observe this reoccurring phenomenon