Federated Learning trains machine learning models on distributed devices by
aggregating local model updates instead of local data. However, privacy
concerns arise as the aggregated local models on the server may reveal
sensitive personal information by inversion attacks. Privacy-preserving
methods, such as homomorphic encryption (HE), then become necessary for FL
training. Despite HE's privacy advantages, its applications suffer from
impractical overheads, especially for foundation models. In this paper, we
present FedML-HE, the first practical federated learning system with efficient
HE-based secure model aggregation. FedML-HE proposes to selectively encrypt
sensitive parameters, significantly reducing both computation and communication
overheads during training while providing customizable privacy preservation.
Our optimized system demonstrates considerable overhead reduction, particularly
for large foundation models (e.g., ~10x reduction for ResNet-50, and up to ~40x
reduction for BERT), demonstrating the potential for scalable HE-based FL
deployment