Federated learning (FL) allows multiple medical institutions to
collaboratively learn a global model without centralizing all clients data. It
is difficult, if possible at all, for such a global model to commonly achieve
optimal performance for each individual client, due to the heterogeneity of
medical data from various scanners and patient demographics. This problem
becomes even more significant when deploying the global model to unseen clients
outside the FL with new distributions not presented during federated training.
To optimize the prediction accuracy of each individual client for critical
medical tasks, we propose a novel unified framework for both Inside and Outside
model Personalization in FL (IOP-FL). Our inside personalization is achieved by
a lightweight gradient-based approach that exploits the local adapted model for
each client, by accumulating both the global gradients for common knowledge and
local gradients for client-specific optimization. Moreover, and importantly,
the obtained local personalized models and the global model can form a diverse
and informative routing space to personalize a new model for outside FL
clients. Hence, we design a new test-time routing scheme inspired by the
consistency loss with a shape constraint to dynamically incorporate the models,
given the distribution information conveyed by the test data. Our extensive
experimental results on two medical image segmentation tasks present
significant improvements over SOTA methods on both inside and outside
personalization, demonstrating the great potential of our IOP-FL scheme for
clinical practice. Code will be released at https://github.com/med-air/IOP-FL.Comment: Submitted to IEEE TMI special issue on federated learning for medical
imagin