D2^2: Decentralized Training over Decentralized Data

While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from different workers can be {\em unique} and {\em different}. Ironically, recent analysis of decentralized parallel stochastic gradient descent (D-PSGD) relies on the assumption that the data hosted on different workers are {\em not too different}. In this paper, we ask the question: {\em Can we design a decentralized parallel stochastic gradient descent algorithm that is less sensitive to the data variance across workers?} In this paper, we present D$^2$, a novel decentralized parallel stochastic gradient descent algorithm designed for large data variance \xr{among workers} (imprecisely, "decentralized" data). The core of D$^2$ is a variance blackuction extension of the standard D-PSGD algorithm, which improves the convergence rate from $O\left({\sigma \over \sqrt{nT}} + {(n\zeta^2)^{\frac{1}{3}} \over T^{2/3}}\right)$ to $O\left({\sigma \over \sqrt{nT}}\right)$ where $\zeta^{2}$ denotes the variance among data on different workers. As a result, D$^2$ is robust to data variance among workers. We empirically evaluated D$^2$ on image classification tasks where each worker has access to only the data of a limited set of labels, and find that D$^2$ significantly outperforms D-PSGD

BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual Pragmatic Knowledge

Pre-trained language models like ChatGPT have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks. Moreover, in bioinformatics, generating functional programs poses additional notable challenges due to the amount of domain knowledge, the need for complicated data operations, and intricate functional dependencies between the operations. Here, we present BioCoder, a benchmark developed to evaluate existing pre-trained models in generating bioinformatics code. In relation to function-code generation, BioCoder covers potential package dependencies, class declarations, and global variables. It incorporates 1026 functions and 1243 methods in Python and Java from GitHub and 253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing framework for evaluation, and we have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes the importance of domain knowledge, pragmatic code generation, and contextual understanding. Our dataset, benchmark, Docker images, and scripts required for testing are all available at https://github.com/gersteinlab/biocoder

Large Language Models are Effective Table-to-Text Generators, Evaluators, and Feedback Providers

Large language models (LLMs) have shown remarkable ability on controllable text generation. However, the potential of LLMs in generating text from structured tables remains largely under-explored. In this paper, we study the capabilities of LLMs for table-to-text generation tasks, particularly aiming to investigate their performance in generating natural language statements that can be logically entailed by a provided table. First, we investigate how LLMs compare to state-of-the-art table-to-text fine-tuned models, and demonstrate that LLMs can generate statements with higher faithfulness compared with previous state-of-the-art fine-tuned models. Given this finding, we next explore whether LLMs can serve as faithfulness-level automated evaluation metrics. Through human evaluation, we show that evaluation metrics adopted from LLMs correlates better with human judgments compared with existing faithfulness-level metrics. Finally, we demonstrate that LLMs using chain-of-thought prompting can generate high-fidelity natural language feedback for other table-to-text models' generations, provide insights for future work regarding the distillation of text generation capabilities from LLMs to smaller models.Comment: work in progres

Hop: Heterogeneity-Aware Decentralized Training

Recent work has shown that decentralized algorithms can deliver superior performance over centralized ones in the context of machine learning. The two approaches, with the main difference residing in their distinct communication patterns, are both susceptible to performance degradation in heterogeneous environments. Although vigorous efforts have been devoted to supporting centralized algorithms against heterogeneity, little has been explored in decentralized algorithms regarding this problem. This paper proposes Hop, the first heterogeneity-aware decentralized training protocol. Based on a unique characteristic of decentralized training that we have identified, the iteration gap, we propose a queue-based synchronization mechanism that can efficiently implement backup workers and bounded staleness in the decentralized setting. To cope with deterministic slowdown, we propose skipping iterations so that the effect of slower workers is further mitigated. We build a prototype implementation of Hop on TensorFlow. The experiment results on CNN and SVM show significant speedup over standard decentralized training in heterogeneous settings

Exogenous Gamma-Aminobutyric Acid Application Induced Modulations in the Performance of Aromatic Rice Under Lead Toxicity

Gamma-aminobutyric acid (GABA) is a non-protein amino acid and has a multi-functional role in abiotic stress tolerance. A pot experiment was conducted to assess the role of exogenous gamma-aminobutyric acid (GABA) application to modulate the growth, yield, and related physio-biochemical mechanisms in two aromatic rice cultivars, that is, Guixiangzhan (GXZ) and Nongxiang 18 (NX-18), under Pb toxic and normal conditions. The experimental treatments were comprised of Ck: without Pb and GABA (control), GABA: 1 mM GABA is applied under normal conditions (without Pb), Pb + GABA: 1 mM GABA is applied under Pb toxicity (800 mg kg−1 of soil), and Pb= only Pb (800 mg kg−1 of soil) is applied (no GABA). The required concentrations of GABA were applied as a foliar spray. Results revealed that Pb stress induced oxidative damage in terms of enhanced malondialdehyde (MDA), electrolyte leakage (EL), and H2O2 contents, while exogenous GABA application improved leaf chlorophyll, proline, protein and GABA contents, photosynthesis and gas exchange, and antioxidant defense under Pb toxicity in both rice cultivars. Moreover, glutamine synthetase (GS) and nitrate reductase (NR) activities were variably affected due to GABA application under Pb stress. The yield and related traits, that is, productive tillers/pot, grains/panicle, filled grain %, 1,000-grain weight, and grain yield were 13.64 and 10.29, 0.37% and 2.26%, 3.89 and 19.06%, 7.35 and 12.84%, and 17.92 and 40.56 lower under Pb treatment than Pb + GABA for GXZ and NX-18, respectively. Furthermore, exogenous GABA application in rice reduced Pb contents in shoot, leaves, panicle, and grains compared with Pb-exposed plants without GABA. Overall, GXZ performed better than NX-18 under Pb toxic conditions