13 research outputs found
Vertical Semi-Federated Learning for Efficient Online Advertising
As an emerging secure learning paradigm in leveraging cross-silo private
data, vertical federated learning (VFL) is expected to improve advertising
models by enabling the joint learning of complementary user attributes
privately owned by the advertiser and the publisher. However, the 1) restricted
applicable scope to overlapped samples and 2) high system challenge of
real-time federated serving have limited its application to advertising
systems.
In this paper, we advocate new learning setting Semi-VFL (Vertical
Semi-Federated Learning) as a lightweight solution to utilize all available
data (both the overlapped and non-overlapped data) that is free from federated
serving. Semi-VFL is expected to perform better than single-party models and
maintain a low inference cost. It's notably important to i) alleviate the
absence of the passive party's feature and ii) adapt to the whole sample space
to implement a good solution for Semi-VFL. Thus, we propose a carefully
designed joint privileged learning framework (JPL) as an efficient
implementation of Semi-VFL. Specifically, we build an inference-efficient
single-party student model applicable to the whole sample space and meanwhile
maintain the advantage of the federated feature extension. Novel feature
imitation and ranking consistency restriction methods are proposed to extract
cross-party feature correlations and maintain cross-sample-space consistency
for both the overlapped and non-overlapped data.
We conducted extensive experiments on real-world advertising datasets. The
results show that our method achieves the best performance over baseline
methods and validate its effectiveness in maintaining cross-view feature
correlation
Semi-Federated Learning of an Embedding Space Across Multiple Machine Clusters
Provided are systems and methods for privacy-preserving learning of a shared embedding space for data split across multiple separate clusters of computing machines. In one example, the multiple separate clusters of computing machines can correspond to multiple separate data silos
Semi-federated learning: convergence analysis and optimization of a hybrid learning framework
Under the organization of the base station (BS), wireless federated learning (FL) enables collaborative model training among multiple devices. However, the BS is merely responsible for aggregating local updates during the training process, which incurs a waste of the computational resource at the BS. To tackle this issue, we propose a semi-federated learning (SemiFL) paradigm to leverage the computing capabilities of both the BS and devices for a hybrid implementation of centralized learning (CL) and FL. Specifically, each device sends both local gradients and data samples to the BS for training a shared global model. To improve communication efficiency over the same time-frequency resources, we integrate over-the-air computation for aggregation and non-orthogonal multiple access for transmission by designing a novel transceiver structure. To gain deep insights, we conduct convergence analysis by deriving a closed-form optimality gap for SemiFL and extend the result to two extra cases. In the first case, the BS uses all accumulated data samples to calculate the CL gradient, while a decreasing learning rate is adopted in the second case. Our analytical results capture the destructive effect of wireless communication and show that both FL and CL are special cases of SemiFL. Then, we formulate a non-convex problem to reduce the optimality gap by jointly optimizing the transmit power and receive beamformers. Accordingly, we propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers. Extensive simulation results on two real-world datasets corroborate our theoretical analysis, and show that the proposed SemiFL outperforms conventional FL and achieves 3.2% accuracy gain on the MNIST dataset compared to state-of-the-art benchmarks
Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning
The increasing popularity of the federated learning (FL) framework due to its
success in a wide range of collaborative learning tasks also induces certain
security concerns. Among many vulnerabilities, the risk of Byzantine attacks is
of particular concern, which refers to the possibility of malicious clients
participating in the learning process. Hence, a crucial objective in FL is to
neutralize the potential impact of Byzantine attacks and to ensure that the
final model is trustable. It has been observed that the higher the variance
among the clients' models/updates, the more space there is for Byzantine
attacks to be hidden. As a consequence, by utilizing momentum, and thus,
reducing the variance, it is possible to weaken the strength of known Byzantine
attacks. The centered clipping (CC) framework has further shown that the
momentum term from the previous iteration, besides reducing the variance, can
be used as a reference point to neutralize Byzantine attacks better. In this
work, we first expose vulnerabilities of the CC framework, and introduce a
novel attack strategy that can circumvent the defences of CC and other robust
aggregators and reduce their test accuracy up to %33 on best-case scenarios in
image classification tasks. Then, we propose a new robust and fast defence
mechanism that is effective against the proposed and other existing Byzantine
attacks.Comment: IEEE Transactions on Information Forensics and Security 202
Semi-Supervised Federated Adaptive Label Learning with Disambiguation Prototype
Traditional federated learning depends on having data that is categorized or labeled for its training processes. However, obtaining such specific labeled data can be a complex task due to concerns about privacy, the expensive nature of labeling, or a shortage of specialized knowledge for categorizing data in certain fields. To overcome these obstacles, researchers have introduced Federated Semi-Supervised Learning (FSSL). This approach merges a limited quantity of categorized data with a vast amount of uncategorized data, thereby improving the effectiveness of models while maintaining the privacy and security of the data. Despite its advantages, FSSL encounters challenges such as gradual progress in learning and diminished precision, especially in situations where the distribution of data categories is uneven.
To mitigate these problems, we have introduced an innovative framework named Federated Adaptive Label Learning (FedALL). FedALL integrates strategies from transfer learning, partial label learning, and prototype learning. It constructs an Adaptive Label List through transfer learning and devises a Label Disambiguation Prototype via representation learning. These strategies significantly reduce the adverse impacts caused by incorrect one-hot-label predictions for unlabeled data on the client's model. Additionally, incorporating pre-trained weights substantially accelerates the convergence speed of FedALL.
In the experimental section of our paper, we compared FedALL against other baseline models in both Independent and Identically Distributed (IID) and Non-Independent and Identically Distributed (Non-IID) settings across three distinct datasets. The results demonstrate that FedALL outperforms all baselines in all scenarios, especially in Non-IID environments. Moreover, we utilized the Banach Fixed Point Theorem to prove the convergence of the Label Disambiguation Prototype
Vertical Federated Learning:A Structured Literature Review
Federated Learning (FL) has emerged as a promising distributed learning paradigm with an added advantage of data privacy. With the growing interest in having collaboration among data owners, FL has gained significant attention of organizations. The idea of FL is to enable collaborating participants train machine learning (ML) models on decentralized data without breaching privacy. In simpler words, federated learning is the approach of ``bringing the model to the data, instead of bringing the data to the mode''. Federated learning, when applied to data which is partitioned vertically across participants, is able to build a complete ML model by combining local models trained only using the data with distinct features at the local sites. This architecture of FL is referred to as vertical federated learning (VFL), which differs from the conventional FL on horizontally partitioned data. As VFL is different from conventional FL, it comes with its own issues and challenges. In this paper, we present a structured literature review discussing the state-of-the-art approaches in VFL. Additionally, the literature review highlights the existing solutions to challenges in VFL and provides potential research directions in this domain
How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning
Federated learning (FL) has attracted vivid attention as a privacy-preserving
distributed learning framework. In this work, we focus on cross-silo FL, where
clients become the model owners after training and are only concerned about the
model's generalization performance on their local data. Due to the data
heterogeneity issue, asking all the clients to join a single FL training
process may result in model performance degradation. To investigate the
effectiveness of collaboration, we first derive a generalization bound for each
client when collaborating with others or when training independently. We show
that the generalization performance of a client can be improved only by
collaborating with other clients that have more training data and similar data
distribution. Our analysis allows us to formulate a client utility maximization
problem by partitioning clients into multiple collaborating groups. A
hierarchical clustering-based collaborative training (HCCT) scheme is then
proposed, which does not need to fix in advance the number of groups. We
further analyze the convergence of HCCT for general non-convex loss functions
which unveils the effect of data similarity among clients. Extensive
simulations show that HCCT achieves better generalization performance than
baseline schemes, whereas it degenerates to independent training and
conventional FL in specific scenarios
Vertical Federated Learning: A Structured Literature Review
Federated Learning (FL) has emerged as a promising distributed learning
paradigm with an added advantage of data privacy. With the growing interest in
having collaboration among data owners, FL has gained significant attention of
organizations. The idea of FL is to enable collaborating participants train
machine learning (ML) models on decentralized data without breaching privacy.
In simpler words, federated learning is the approach of ``bringing the model to
the data, instead of bringing the data to the mode''. Federated learning, when
applied to data which is partitioned vertically across participants, is able to
build a complete ML model by combining local models trained only using the data
with distinct features at the local sites. This architecture of FL is referred
to as vertical federated learning (VFL), which differs from the conventional FL
on horizontally partitioned data. As VFL is different from conventional FL, it
comes with its own issues and challenges. In this paper, we present a
structured literature review discussing the state-of-the-art approaches in VFL.
Additionally, the literature review highlights the existing solutions to
challenges in VFL and provides potential research directions in this domain
Vertical Federated Learning
Vertical Federated Learning (VFL) is a federated learning setting where
multiple parties with different features about the same set of users jointly
train machine learning models without exposing their raw data or model
parameters. Motivated by the rapid growth in VFL research and real-world
applications, we provide a comprehensive review of the concept and algorithms
of VFL, as well as current advances and challenges in various aspects,
including effectiveness, efficiency, and privacy. We provide an exhaustive
categorization for VFL settings and privacy-preserving protocols and
comprehensively analyze the privacy attacks and defense strategies for each
protocol. In the end, we propose a unified framework, termed VFLow, which
considers the VFL problem under communication, computation, privacy, and
effectiveness constraints. Finally, we review the most recent advances in
industrial applications, highlighting open challenges and future directions for
VFL