Rethinking Learning Rate Tuning in the Era of Large Language Models

Large Language Models (LLMs) represent the recent success of deep learning in achieving remarkable human-like predictive performance. It has become a mainstream strategy to leverage fine-tuning to adapt LLMs for various real-world applications due to the prohibitive expenses associated with LLM training. The learning rate is one of the most important hyperparameters in LLM fine-tuning with direct impacts on both fine-tuning efficiency and fine-tuned LLM quality. Existing learning rate policies are primarily designed for training traditional deep neural networks (DNNs), which may not work well for LLM fine-tuning. We reassess the research challenges and opportunities of learning rate tuning in the coming era of Large Language Models. This paper makes three original contributions. First, we revisit existing learning rate policies to analyze the critical challenges of learning rate tuning in the era of LLMs. Second, we present LRBench++ to benchmark learning rate policies and facilitate learning rate tuning for both traditional DNNs and LLMs. Third, our experimental analysis with LRBench++ demonstrates the key differences between LLM fine-tuning and traditional DNN training and validates our analysis

SP1 enhances Zbtb7A gene expression via direct binding to GC box in HePG2 cells

<p>Abstract</p> <p>Background</p> <p>Zbtb7A is a proto-oncogenic transcriptional regulator that plays an important role in adipogenesis, osteogenesis and oncogenesis, but little is known about the regulation of Zbtb7A gene expression which is of importance in the function uncovering of this gene.</p> <p>Finding</p> <p>Here, a 5'-flanking region of the human Zbtb7A gene was cloned and characterized. It was found that the GC box within Zbtb7A promoter is necessary for the promoter activity. Furthermore, we identified that Sp1 acts as an activator in the regulation of Zbtb7A promoter activity and the physical interaction between Sp1 and GC box is responsible for the activation of Zbtb7A gene promoter.</p> <p>Conclusion</p> <p>Our results confirmed that Sp1 upregulates Zbtb7A gene expression via direct binding to GC box within the promoter.</p

Explanation-Guided Backdoor Attacks on Model-Agnostic RF Fingerprinting

Despite the proven capabilities of deep neural networks (DNNs) for radio frequency (RF) fingerprinting, their security vulnerabilities have been largely overlooked. Unlike the extensively studied image domain, few works have explored the threat of backdoor attacks on RF signals. In this paper, we analyze the susceptibility of DNN-based RF fingerprinting to backdoor attacks, focusing on a more practical scenario where attackers lack access to control model gradients and training processes. We propose leveraging explainable machine learning techniques and autoencoders to guide the selection of positions and values, enabling the creation of effective backdoor triggers in a model-agnostic manner. To comprehensively evaluate our backdoor attack, we employ four diverse datasets with two protocols (Wi-Fi and LoRa) across various DNN architectures. Given that RF signals are often transformed into the frequency or time-frequency domains, this study also assesses attack efficacy in the time-frequency domain. Furthermore, we experiment with potential defenses, demonstrating the difficulty of fully safeguarding against our attacks

Preventing Discriminatory Decision-making in Evolving Data Streams

Bias in machine learning has rightly received significant attention over the last decade. However, most fair machine learning (fair-ML) work to address bias in decision-making systems has focused solely on the offline setting. Despite the wide prevalence of online systems in the real world, work on identifying and correcting bias in the online setting is severely lacking. The unique challenges of the online environment make addressing bias more difficult than in the offline setting. First, Streaming Machine Learning (SML) algorithms must deal with the constantly evolving real-time data stream. Second, they need to adapt to changing data distributions (concept drift) to make accurate predictions on new incoming data. Adding fairness constraints to this already complicated task is not straightforward. In this work, we focus on the challenges of achieving fairness in biased data streams while accounting for the presence of concept drift, accessing one sample at a time. We present Fair Sampling over Stream ($FS^2$), a novel fair rebalancing approach capable of being integrated with SML classification algorithms. Furthermore, we devise the first unified performance-fairness metric, Fairness Bonded Utility (FBU), to evaluate and compare the trade-off between performance and fairness of different bias mitigation methods efficiently. FBU simplifies the comparison of fairness-performance trade-offs of multiple techniques through one unified and intuitive evaluation, allowing model designers to easily choose a technique. Overall, extensive evaluations show our measures surpass those of other fair online techniques previously reported in the literature

Association between salivary microbiota and renal function in renal transplant patients during the perioperative period

IntroductionRenal transplantation is an effective treatment for the end stage renal disease (ESRD). However, how salivary microbiota changes during perioperative period of renal transplant recipients (RTRs) has not been elucidated.MethodsFive healthy controls and 11 RTRs who had good recovery were enrolled. Saliva samples were collected before surgery and at 1, 3, 7, and 14 days after surgery. 16S rRNA gene sequencing was performed.ResultsThere was no significant difference in the composition of salivary microbiota between ESRD patients and healthy controls. The salivary microbiota of RTRs showed higher operational taxonomic units (OTUs) amount and greater alpha and beta diversity than those of ESRD patients and healthy controls, but gradually stabilized over time. At the phylum level, the relative abundance of Actinobacteria, Tenericutes and Spirochaetes was about ten times different from ESRD patients or healthy controls for RTRs overall in time. The relative abundance of Bacteroidetes, Fusobacteria, Patescibacteria, Leptotrichiaceae and Streptococcaceae was correlated with serum creatinine (Scr) after renal transplantation.DiscussionIn short, salivary microbiota community altered in the perioperative period of renal transplantation and certain species of salivary microbiota had the potential to be a biomarker of postoperative recovery

Electrochemical Lithium Extraction with Gas Flushing of Porous Electrodes

Electrochemical extraction of lithium from seawater/brine is receiving more and more attention because of its environment-friendly and energy-saving features. In this work, an electrochemical lithium extraction system with gas flushing of porous electrodes is proposed. We verified that the operation of multiple gas washes can significantly reduce the consumption of ultrapure water during the solution exchange and save the time required for the continuous running of the system. The water consumption of multiple gas flush operations is only 1/60 of that of a normal single flush to obtain a purity close to 100% in the recovery solution. By comparing the ion concentration distribution on the electrode surface in flow-through and flow-by-flow modes, we demonstrate that the flow-through mode performs better. We also verified the lithium extraction performance of the whole system, achieving a purity close to 100% and average energy consumption of 0.732 kWh∙kg−1 in each cycle from the source solution of the simulated Atacama salt lake water. These results provide a feasible approach for the large-scale operation of electrochemical lithium extraction from seawater/brine