Privacy, security and data governance constraints rule out a brute force
process in the integration of cross-silo data, which inherits the development
of the Internet of Things. Federated learning is proposed to ensure that all
parties can collaboratively complete the training task while the data is not
out of the local. Vertical federated learning is a specialization of federated
learning for distributed features. To preserve privacy, homomorphic encryption
is applied to enable encrypted operations without decryption. Nevertheless,
together with a robust security guarantee, homomorphic encryption brings extra
communication and computation overhead. In this paper, we analyze the current
bottlenecks of vertical federated learning under homomorphic encryption
comprehensively and numerically. We propose a straggler-resilient and
computation-efficient accelerating system that reduces the communication
overhead in heterogeneous scenarios by 65.26% at most and reduces the
computation overhead caused by homomorphic encryption by 40.66% at most. Our
system can improve the robustness and efficiency of the current vertical
federated learning framework without loss of security