150 research outputs found
FwdLLM: Efficient FedLLM using Forward Gradient
Large Language Models (LLMs) are transforming the landscape of mobile
intelligence. Federated Learning (FL), a method to preserve user data privacy,
is often employed in fine-tuning LLMs to downstream mobile tasks, an approach
known as FedLLM. Though recent efforts have addressed the network issue induced
by the vast model size, they have not practically mitigated vital challenges
concerning integration with mobile devices, such as significant memory
consumption and sluggish model convergence.
In response to these challenges, this work introduces FwdLLM, an innovative
FL protocol designed to enhance the FedLLM efficiency. The key idea of FwdLLM
to employ backpropagation (BP)-free training methods, requiring devices only to
execute ``perturbed inferences''. Consequently, FwdLLM delivers way better
memory efficiency and time efficiency (expedited by mobile NPUs and an expanded
array of participant devices). FwdLLM centers around three key designs: (1) it
combines BP-free training with parameter-efficient training methods, an
essential way to scale the approach to the LLM era; (2) it systematically and
adaptively allocates computational loads across devices, striking a careful
balance between convergence speed and accuracy; (3) it discriminatively samples
perturbed predictions that are more valuable to model convergence.
Comprehensive experiments with five LLMs and three NLP tasks illustrate
FwdLLM's significant advantages over conventional methods, including up to
three orders of magnitude faster convergence and a 14.6x reduction in memory
footprint. Uniquely, FwdLLM paves the way for federated learning of
billion-parameter LLMs such as LLaMA on COTS mobile devices -- a feat
previously unattained.Comment: under revie
Federated NLP in Few-shot Scenarios
Natural language processing (NLP) sees rich mobile applications. To support
various language understanding tasks, a foundation NLP model is often
fine-tuned in a federated, privacy-preserving setting (FL). This process
currently relies on at least hundreds of thousands of labeled training samples
from mobile clients; yet mobile users often lack willingness or knowledge to
label their data. Such an inadequacy of data labels is known as a few-shot
scenario; it becomes the key blocker for mobile NLP applications.
For the first time, this work investigates federated NLP in the few-shot
scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and
prompt learning, we first establish a training pipeline that delivers
competitive accuracy when only 0.05% (fewer than 100) of the training data is
labeled and the remaining is unlabeled. To instantiate the workflow, we further
present a system FFNLP, addressing the high execution cost with novel designs.
(1) Curriculum pacing, which injects pseudo labels to the training workflow at
a rate commensurate to the learning progress; (2) Representational diversity, a
mechanism for selecting the most learnable data, only for which pseudo labels
will be generated; (3) Co-planning of a model's training depth and layer
capacity. Together, these designs reduce the training delay, client energy, and
network traffic by up to 46.0, 41.2 and 3000.0,
respectively. Through algorithm/system co-design, FFNLP demonstrates that FL
can apply to challenging settings where most training samples are unlabeled
Towards Practical Few-shot Federated NLP
Transformer-based pre-trained models have emerged as the predominant solution
for natural language processing (NLP). Fine-tuning such pre-trained models for
downstream tasks often requires a considerable amount of labeled private data.
In practice, private data is often distributed across heterogeneous mobile
devices and may be prohibited from being uploaded. Moreover, well-curated
labeled data is often scarce, presenting an additional challenge. To address
these challenges, we first introduce a data generator for federated few-shot
learning tasks, which encompasses the quantity and skewness of scarce labeled
data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a
prompt-based federated learning system that exploits abundant unlabeled data
for data augmentation. Our experiments indicate that AUG-FedPrompt can perform
on par with full-set fine-tuning with a limited amount of labeled data.
However, such competitive performance comes at a significant system cost.Comment: EuroSys23 worksho
A Comprehensive Survey on Orbital Edge Computing: Systems, Applications, and Algorithms
The number of satellites, especially those operating in low-earth orbit
(LEO), is exploding in recent years. Additionally, the use of COTS hardware
into those satellites enables a new paradigm of computing: orbital edge
computing (OEC). OEC entails more technically advanced steps compared to
single-satellite computing. This feature allows for vast design spaces with
multiple parameters, rendering several novel approaches feasible. The mobility
of LEO satellites in the network and limited resources of communication,
computation, and storage make it challenging to design an appropriate
scheduling algorithm for specific tasks in comparison to traditional
ground-based edge computing. This article comprehensively surveys the
significant areas of focus in orbital edge computing, which include protocol
optimization, mobility management, and resource allocation. This article
provides the first comprehensive survey of OEC. Previous survey papers have
only concentrated on ground-based edge computing or the integration of space
and ground technologies. This article presents a review of recent research from
2000 to 2023 on orbital edge computing that covers network design, computation
offloading, resource allocation, performance analysis, and optimization.
Moreover, having discussed several related works, both technological challenges
and future directions are highlighted in the field.Comment: 18 pages, 9 figures and 5 table
DialogRE^C+: An Extension of DialogRE to Investigate How Much Coreference Helps Relation Extraction in Dialogs
Dialogue relation extraction (DRE) that identifies the relations between
argument pairs in dialogue text, suffers much from the frequent occurrence of
personal pronouns, or entity and speaker coreference. This work introduces a
new benchmark dataset DialogRE^C+, introducing coreference resolution into the
DRE scenario. With the aid of high-quality coreference knowledge, the reasoning
of argument relations is expected to be enhanced. In DialogRE^C+ dataset, we
manually annotate total 5,068 coreference chains over 36,369 argument mentions
based on the existing DialogRE data, where four different coreference chain
types namely speaker chain, person chain, location chain and organization chain
are explicitly marked. We further develop 4 coreference-enhanced graph-based
DRE models, which learn effective coreference representations for improving the
DRE task. We also train a coreference resolution model based on our annotations
and evaluate the effect of automatically extracted coreference chains
demonstrating the practicality of our dataset and its potential to other
domains and tasks.Comment: Accepted by NLPCC 202
Study on the composition optimization method for improving the fluidity of cast TiAlNb alloy and its mechanism
In this paper, the effects of Al, Nb main elements, Fe, Mo, W, Co, B, Si and
their contents on the fluidity of Ti-22Al-25Nb alloy were investigated. The
composition that was beneficial to improve the fluidity was screened through
the thermodynamic software calculating thermophysical parameters affecting the
fluidity of TiAlNb alloy, the numerical simulation test of its fluidity and
the verification test of the fluidity of optimized alloys. Finally, the
improvement mechanism of the alloy fluidity was discussed. Results showed that
the appropriate reduction of Nb element was better than Al element for the
improvement of fluidity. The addition of trace Fe, B and Si elements were
beneficial to the improvement of fluidity, the improvement effect of B element
was best, while the addition of trace Mo, W, Co were not conducive to the
improvement of fluidity. The cessation mechanism of TiAlNb alloy is the
cessation mechanism of the alloy with a wide crystallization temperature range.
The composition which was most beneficial to improve the fluidity was
Ti-22Al-24Nb-0.1B. The main reasons for the improvement of the fluidity had two
sides: on the one hand, the reduction of 1at% Nb and the addition of 0.1at% B
not only increased the superheat and crystallization latent heat of the alloy,
but also reduced the melt viscosity and thermal conductivity, thus improving
the fluidity. On the other hand, the TiB phase refined the grains, the fine
grains prevented the dendrite from growing into developed dendrite networks,
inhibited the adverse effect of the increase in the width of the solidification
zone on the fluidity, reduced the flow resistance of the molten metal, and
further improved the fluidity of the alloy.Comment: 23 pages, 14 figures, research pape
Expression of a LINE-1 endonuclease variant in gastric cancer: its association with clinicopathological parameters
BACKGROUND: Long interspersed nuclear element-1 (LINE-1 or L1), the most abundant and only autonomously active family of non-LTR retrotransposons in the human genome, expressed not only in the germ lines but also in somatic tissues. It contributes to genetic instability, aging, and age-related diseases, such as cancer. Our previous study identified in human gastric adenocarcinoma an upregulated transcript GCRG213, which shared 88% homology with human L1 sequence and contained a putative conserved apurinic/apyrimidinic endonucleas1 domain. METHODS: Immunohistochemistry was carried out by using a monoclonal mouse anti-human GCRG213 protein (GCRG213p) antibody produced in our laboratory, on tissue microarray constructed with specimens from 175 gastric adenocarcinoma patients. The correlation between GCRG213p expression and patient clinicopathological parameters was evaluated. GCRG213p expression in gastric cancer cell lines were studied using Western blotting analysis. L1 promoter methylation status of gastric cancer cells was tested using methylation-specific PCR. BLASTP was used at the NCBI Blast server to identify GCRG213p sequence to any alignments in the Protein Data Bank databases. RESULTS: Most primary gastric cancer, lymph node metastases and gastric intestinal metaplasia glands showed positive GCRG213p immunoreactivity. High GCRG213p immunostaining score in the primary gastric cancer was positively correlated with tumor differentiation (well differentiated, p = 0.001), Lauren’s classification (intestinal type, p < 0.05) and a late age onset of gastric adenocarcinoma (≥65 yrs; p < 0.05). GCRG213p expression has no association with other clinicopathological parameters, including survival. Western blotting analysis of GCRG213p expression in gastric cancer cells indicated that GCRG213p level was higher in gastric cancer cell lines than in human normal gastric epithelium immortalized cell line GES-1. Partial methylation of L1 in gastric cancer cells was confirmed by methylation-specific PCR. BLASTP program analysis revealed that GCRG213p peptide shared 83.0% alignment with the C-terminal region of L1 endonuclease (L1-EN). GCRG213p sequence possesses the important residues that compose the conserved features of L1-EN. CONCLUSIONS: GCRG213p could be a variant of L1-EN, a functional member of L1-EN family. Overexpression of GCRG213p is common in both primary gastric cancer and lymph node metastasis. These findings provide evidence of somatic L1 expression in gastric cancer, and its potential consequences in the form of tumor
- …