We study the optimization aspects of personalized Federated Learning (FL). We
propose general optimizers that can be used to solve essentially any existing
personalized FL objective, namely a tailored variant of Local SGD and variants
of accelerated coordinate descent/accelerated SVRCD. By studying a general
personalized objective that is capable of recovering essentially any existing
personalized FL objective as a special case, we develop a universal
optimization theory applicable to all strongly convex personalized FL models in
the literature. We demonstrate the practicality and/or optimality of our
methods both in terms of communication and local computation. Surprisingly
enough, our general optimization solvers and theory are capable of recovering
best-known communication and computation guarantees for solving specific
personalized FL objectives. Thus, our proposed methods can be taken as
universal optimizers that make the design of task-specific optimizers
unnecessary in many cases.Comment: 65 pages, 5 figure