191 research outputs found
Study on the dynamics of golf swing and impedance control for a golf swing robot
高知工科大学博士(工学) 平成19年3月20日授与 (甲第111号
Reducing Communication for Split Learning by Randomized Top-k Sparsification
Split learning is a simple solution for Vertical Federated Learning (VFL),
which has drawn substantial attention in both research and application due to
its simplicity and efficiency. However, communication efficiency is still a
crucial issue for split learning. In this paper, we investigate multiple
communication reduction methods for split learning, including cut layer size
reduction, top-k sparsification, quantization, and L1 regularization. Through
analysis of the cut layer size reduction and top-k sparsification, we further
propose randomized top-k sparsification, to make the model generalize and
converge better. This is done by selecting top-k elements with a large
probability while also having a small probability to select non-top-k elements.
Empirical results show that compared with other communication-reduction
methods, our proposed randomized top-k sparsification achieves a better model
performance under the same compression level.Comment: Accepted by IJCAI 202
Federated Unlearning via Active Forgetting
The increasing concerns regarding the privacy of machine learning models have
catalyzed the exploration of machine unlearning, i.e., a process that removes
the influence of training data on machine learning models. This concern also
arises in the realm of federated learning, prompting researchers to address the
federated unlearning problem. However, federated unlearning remains
challenging. Existing unlearning methods can be broadly categorized into two
approaches, i.e., exact unlearning and approximate unlearning. Firstly,
implementing exact unlearning, which typically relies on the
partition-aggregation framework, in a distributed manner does not improve time
efficiency theoretically. Secondly, existing federated (approximate) unlearning
methods suffer from imprecise data influence estimation, significant
computational burden, or both. To this end, we propose a novel federated
unlearning framework based on incremental learning, which is independent of
specific models and federated settings. Our framework differs from existing
federated unlearning methods that rely on approximate retraining or data
influence estimation. Instead, we leverage new memories to overwrite old ones,
imitating the process of \textit{active forgetting} in neurology. Specifically,
the model, intended to unlearn, serves as a student model that continuously
learns from randomly initiated teacher models. To preserve catastrophic
forgetting of non-target data, we utilize elastic weight consolidation to
elastically constrain weight change. Extensive experiments on three benchmark
datasets demonstrate the efficiency and effectiveness of our proposed method.
The result of backdoor attacks demonstrates that our proposed method achieves
satisfying completeness
Federated Large Language Model: A Position Paper
Large scale language models (LLM) have received significant attention and
found diverse applications across various domains, but their development
encounters challenges in real-world scenarios. These challenges arise due to
the scarcity of public domain data availability and the need to maintain
privacy with respect to private domain data. To address these issues, federated
learning (FL) has emerged as a promising technology that enables collaborative
training of shared models while preserving decentralized data. We propose the
concept of federated LLM, which comprises three key components, i.e., federated
LLM pre-training, federated LLM fine-tuning, and federated LLM prompt
engineering. For each component, we discuss its advantage over traditional LLM
training methods and propose specific engineering strategies for
implementation. Furthermore, we explore the novel challenges introduced by the
integration of FL and LLM. We analyze existing solutions and identify potential
obstacles faced by these solutions within the context of federated LLM.Comment: 11 pages, 4 figure
Freshness or Accuracy, Why Not Both? Addressing Delayed Feedback via Dynamic Graph Neural Networks
The delayed feedback problem is one of the most pressing challenges in
predicting the conversion rate since users' conversions are always delayed in
online commercial systems. Although new data are beneficial for continuous
training, without complete feedback information, i.e., conversion labels,
training algorithms may suffer from overwhelming fake negatives. Existing
methods tend to use multitask learning or design data pipelines to solve the
delayed feedback problem. However, these methods have a trade-off between data
freshness and label accuracy. In this paper, we propose Delayed Feedback
Modeling by Dynamic Graph Neural Network (DGDFEM). It includes three stages,
i.e., preparing a data pipeline, building a dynamic graph, and training a CVR
prediction model. In the model training, we propose a novel graph convolutional
method named HLGCN, which leverages both high-pass and low-pass filters to deal
with conversion and non-conversion relationships. The proposed method achieves
both data freshness and label accuracy. We conduct extensive experiments on
three industry datasets, which validate the consistent superiority of our
method
- …