Learning from the collective knowledge of data dispersed across private
sources can provide neural networks with enhanced generalization capabilities.
Federated learning, a method for collaboratively training a machine learning
model across remote clients, achieves this by combining client models via the
orchestration of a central server. However, current approaches face two
critical limitations: i) they struggle to converge when client domains are
sufficiently different, and ii) current aggregation techniques produce an
identical global model for each client. In this work, we address these issues
by reformulating the typical federated learning setup: rather than learning a
single global model, we learn N models each optimized for a common objective.
To achieve this, we apply a weighted distance minimization to model parameters
shared in a peer-to-peer topology. The resulting framework, Iterative Parameter
Alignment, applies naturally to the cross-silo setting, and has the following
properties: (i) a unique solution for each participant, with the option to
globally converge each model in the federation, and (ii) an optional
early-stopping mechanism to elicit fairness among peers in collaborative
learning settings. These characteristics jointly provide a flexible new
framework for iteratively learning from peer models trained on disparate
datasets. We find that the technique achieves competitive results on a variety
of data partitions compared to state-of-the-art approaches. Further, we show
that the method is robust to divergent domains (i.e. disjoint classes across
peers) where existing approaches struggle.Comment: Published at IEEE Big Data 202