76 research outputs found
D: Decentralized Training over Decentralized Data
While training a machine learning model using multiple workers, each of which
collects data from their own data sources, it would be most useful when the
data collected from different workers can be {\em unique} and {\em different}.
Ironically, recent analysis of decentralized parallel stochastic gradient
descent (D-PSGD) relies on the assumption that the data hosted on different
workers are {\em not too different}. In this paper, we ask the question: {\em
Can we design a decentralized parallel stochastic gradient descent algorithm
that is less sensitive to the data variance across workers?} In this paper, we
present D, a novel decentralized parallel stochastic gradient descent
algorithm designed for large data variance \xr{among workers} (imprecisely,
"decentralized" data). The core of D is a variance blackuction extension of
the standard D-PSGD algorithm, which improves the convergence rate from
to where
denotes the variance among data on different workers. As a result, D is
robust to data variance among workers. We empirically evaluated D on image
classification tasks where each worker has access to only the data of a limited
set of labels, and find that D significantly outperforms D-PSGD
BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual Pragmatic Knowledge
Pre-trained language models like ChatGPT have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks. Moreover, in bioinformatics, generating
functional programs poses additional notable challenges due to the amount of
domain knowledge, the need for complicated data operations, and intricate
functional dependencies between the operations. Here, we present BioCoder, a
benchmark developed to evaluate existing pre-trained models in generating
bioinformatics code. In relation to function-code generation, BioCoder covers
potential package dependencies, class declarations, and global variables. It
incorporates 1026 functions and 1243 methods in Python and Java from GitHub and
253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing
framework for evaluation, and we have applied it to evaluate many models
including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+,
InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes
the importance of domain knowledge, pragmatic code generation, and contextual
understanding. Our dataset, benchmark, Docker images, and scripts required for
testing are all available at https://github.com/gersteinlab/biocoder
Large Language Models are Effective Table-to-Text Generators, Evaluators, and Feedback Providers
Large language models (LLMs) have shown remarkable ability on controllable
text generation. However, the potential of LLMs in generating text from
structured tables remains largely under-explored. In this paper, we study the
capabilities of LLMs for table-to-text generation tasks, particularly aiming to
investigate their performance in generating natural language statements that
can be logically entailed by a provided table. First, we investigate how LLMs
compare to state-of-the-art table-to-text fine-tuned models, and demonstrate
that LLMs can generate statements with higher faithfulness compared with
previous state-of-the-art fine-tuned models. Given this finding, we next
explore whether LLMs can serve as faithfulness-level automated evaluation
metrics. Through human evaluation, we show that evaluation metrics adopted from
LLMs correlates better with human judgments compared with existing
faithfulness-level metrics. Finally, we demonstrate that LLMs using
chain-of-thought prompting can generate high-fidelity natural language feedback
for other table-to-text models' generations, provide insights for future work
regarding the distillation of text generation capabilities from LLMs to smaller
models.Comment: work in progres
Hop: Heterogeneity-Aware Decentralized Training
Recent work has shown that decentralized algorithms can deliver superior
performance over centralized ones in the context of machine learning. The two
approaches, with the main difference residing in their distinct communication
patterns, are both susceptible to performance degradation in heterogeneous
environments. Although vigorous efforts have been devoted to supporting
centralized algorithms against heterogeneity, little has been explored in
decentralized algorithms regarding this problem.
This paper proposes Hop, the first heterogeneity-aware decentralized training
protocol. Based on a unique characteristic of decentralized training that we
have identified, the iteration gap, we propose a queue-based synchronization
mechanism that can efficiently implement backup workers and bounded staleness
in the decentralized setting. To cope with deterministic slowdown, we propose
skipping iterations so that the effect of slower workers is further mitigated.
We build a prototype implementation of Hop on TensorFlow. The experiment
results on CNN and SVM show significant speedup over standard decentralized
training in heterogeneous settings
Exogenous Gamma-Aminobutyric Acid Application Induced Modulations in the Performance of Aromatic Rice Under Lead Toxicity
Gamma-aminobutyric acid (GABA) is a non-protein amino acid and has a multi-functional role in abiotic stress tolerance. A pot experiment was conducted to assess the role of exogenous gamma-aminobutyric acid (GABA) application to modulate the growth, yield, and related physio-biochemical mechanisms in two aromatic rice cultivars, that is, Guixiangzhan (GXZ) and Nongxiang 18 (NX-18), under Pb toxic and normal conditions. The experimental treatments were comprised of Ck: without Pb and GABA (control), GABA: 1 mM GABA is applied under normal conditions (without Pb), Pb + GABA: 1 mM GABA is applied under Pb toxicity (800 mg kg−1 of soil), and Pb= only Pb (800 mg kg−1 of soil) is applied (no GABA). The required concentrations of GABA were applied as a foliar spray. Results revealed that Pb stress induced oxidative damage in terms of enhanced malondialdehyde (MDA), electrolyte leakage (EL), and H2O2 contents, while exogenous GABA application improved leaf chlorophyll, proline, protein and GABA contents, photosynthesis and gas exchange, and antioxidant defense under Pb toxicity in both rice cultivars. Moreover, glutamine synthetase (GS) and nitrate reductase (NR) activities were variably affected due to GABA application under Pb stress. The yield and related traits, that is, productive tillers/pot, grains/panicle, filled grain %, 1,000-grain weight, and grain yield were 13.64 and 10.29, 0.37% and 2.26%, 3.89 and 19.06%, 7.35 and 12.84%, and 17.92 and 40.56 lower under Pb treatment than Pb + GABA for GXZ and NX-18, respectively. Furthermore, exogenous GABA application in rice reduced Pb contents in shoot, leaves, panicle, and grains compared with Pb-exposed plants without GABA. Overall, GXZ performed better than NX-18 under Pb toxic conditions
- …