Safe RLHF: Safe Reinforcement Learning from Human Feedback

With the development of large language models (LLMs), striking a balance between the performance and safety of AI systems has never been more critical. However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training. To address this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly decouples human preferences regarding helpfulness and harmlessness, effectively avoiding the crowdworkers' confusion about the tension and allowing us to train separate reward and cost models. We formalize the safety concern of LLMs as an optimization task of maximizing the reward function while satisfying specified cost constraints. Leveraging the Lagrangian method to solve this constrained problem, Safe RLHF dynamically adjusts the balance between the two objectives during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we demonstrate a superior ability to mitigate harmful responses while enhancing model performance compared to existing value-aligned algorithms. Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with collected human preferences, significantly improving its helpfulness and harmlessness according to human evaluations

Distributed and Deep Vertical Federated Learning with Big Data

In recent years, data are typically distributed in multiple organizations while the data security is becoming increasingly important. Federated Learning (FL), which enables multiple parties to collaboratively train a model without exchanging the raw data, has attracted more and more attention. Based on the distribution of data, FL can be realized in three scenarios, i.e., horizontal, vertical, and hybrid. In this paper, we propose to combine distributed machine learning techniques with Vertical FL and propose a Distributed Vertical Federated Learning (DVFL) approach. The DVFL approach exploits a fully distributed architecture within each party in order to accelerate the training process. In addition, we exploit Homomorphic Encryption (HE) to protect the data against honest-but-curious participants. We conduct extensive experimentation in a large-scale cluster environment and a cloud environment in order to show the efficiency and scalability of our proposed approach. The experiments demonstrate the good scalability of our approach and the significant efficiency advantage (up to 6.8 times with a single server and 15.1 times with multiple servers in terms of the training time) compared with baseline frameworks.Comment: To appear in CCPE (Concurrency and Computation: Practice and Experience

Facile and Scalable Preparation of Graphene Oxide-Based Magnetic Hybrids for Fast and Highly Efficient Removal of Organic Dyes

This study reports the facile preparation and the dye removal efficiency of nanohybrids composed of graphene oxide (GO) and Fe[subscript 3]O[subscript 4] nanoparticles with various geometrical structures. In comparison to previously reported GO/Fe[subscript 3]O[subscript 4] composites prepared through the one-pot, in situ deposition of Fe[subscript 3]O[subscript 4] nanoparticles, the GO/Fe[subscript 3]O[subscript 4] nanohybrids reported here were obtained by taking advantage of the physical affinities between sulfonated GO and Fe[subscript 3]O[subscript 4] nanoparticles, which allows tuning the dimensions and geometries of Fe3O4 nanoparticles in order to decrease their contact area with GO, while still maintaining the magnetic properties of the nanohybrids for easy separation and adsorbent recycling. Both the as-prepared and regenerated nanohybrids demonstrate a nearly 100% removal rate for methylene blue and an impressively high removal rate for Rhodamine B. This study provides new insights into the facile and controllable industrial scale fabrication of safe and highly efficient GO-based adsorbents for dye or other organic pollutants in a wide range of environmental-related applications

RobustState: Boosting Fidelity of Quantum State Preparation via Noise-Aware Variational Training

Quantum state preparation, a crucial subroutine in quantum computing, involves generating a target quantum state from initialized qubits. Arbitrary state preparation algorithms can be broadly categorized into arithmetic decomposition (AD) and variational quantum state preparation (VQSP). AD employs a predefined procedure to decompose the target state into a series of gates, whereas VQSP iteratively tunes ansatz parameters to approximate target state. VQSP is particularly apt for Noisy-Intermediate Scale Quantum (NISQ) machines due to its shorter circuits. However, achieving noise-robust parameter optimization still remains challenging. We present RobustState, a novel VQSP training methodology that combines high robustness with high training efficiency. The core idea involves utilizing measurement outcomes from real machines to perform back-propagation through classical simulators, thus incorporating real quantum noise into gradient calculations. RobustState serves as a versatile, plug-and-play technique applicable for training parameters from scratch or fine-tuning existing parameters to enhance fidelity on target machines. It is adaptable to various ansatzes at both gate and pulse levels and can even benefit other variational algorithms, such as variational unitary synthesis. Comprehensive evaluation of RobustState on state preparation tasks for 4 distinct quantum algorithms using 10 real quantum machines demonstrates a coherent error reduction of up to 7.1 $\times$ and state fidelity improvement of up to 96\% and 81\% for 4-Q and 5-Q states, respectively. On average, RobustState improves fidelity by 50\% and 72\% for 4-Q and 5-Q states compared to baseline approaches.Comment: Accepted to FASTML @ ICCAD 2023. 14 pages, 20 figure

Process modeling studies of physical mechanisms of the formation of an anticyclonic eddy in the central Red Sea

Author Posting. © American Geophysical Union, 2014. This article is posted here by permission of American Geophysical Union for personal use, not for redistribution. The definitive version was published in Journal of Geophysical Research: Oceans 119 (2014): 1445–1464, doi:10.1002/2013JC009351.Surface drifters released in the central Red Sea during April 2010 detected a well-defined anticyclonic eddy around 23°N. This eddy was ∼45–60 km in radius, with a swirl speed up to ∼0.5 m/s. The eddy feature was also evident in monthly averaged sea surface height fields and in current profiles measured on a cross-isobath, shipboard CTD/ADCP survey around that region. The unstructured-grid, Finite-Volume Community Ocean Model (FVCOM) was configured for the Red Sea and process studies were conducted to establish the conditions necessary for the eddy to form and to establish its robustness. The model was capable of reproducing the observed anticyclonic eddy with the same location and size. Diagnosis of model results suggests that the eddy can be formed in a Red Sea that is subject to seasonally varying buoyancy forcing, with no wind, but that its location and structure are significantly altered by wind forcing, initial distribution of water stratification and southward coastal flow from the upstream area. Momentum analysis indicates that the flow field of the eddy was in geostrophic balance, with the baroclinic pressure gradient forcing about the same order of magnitude as the surface pressure gradient forcing.This project was supported by the King Abdullah University of Science and Technology (KAUST). The development of Global-FVCOM was supported by NSF grants ARC0712903, ARC0732084, ARC0804029 and OCE-1203393. C. Chen’s contributions were also supported by the International Center for Marine Studies at Shanghai Ocean University through the ‘‘Shanghai Universities First-class Disciplines Project.’’ L. Pratt was also supported by National Science Foundation Grant OCE0927017.2014-08-2

The Long Noncoding RNA TUG1 Promotes Laryngeal Cancer Proliferation and Migration

Background/Aims: Researchers have shown that long noncoding RNAs are closely associated with the pathogenesis of laryngeal squamous cell carcinoma (LSCC). However, the role of the long noncoding RNA taurine-upregulated gene 1 (TUG1) in the pathogenesis of LSCC remains unclear, although it is recognized as an oncogenic regulator for several types of squamous cell carcinoma. Methods: qRT-PCR was performed to measure the expression of TUG1 in LSCC tissues and cell lines. 3-(4,5-dimethyl-2-thiazolyl)-2,5-diphenyl-2-H-tetrazolium bromide (MTT) was used to measure the effect of TUG1 on cell proliferation. Transwell assay and flow cytometry were employed to determine the effect of TUG1 on cell migration and invasion. Western-blot were performed to explore the relation of TUG1 and p53 mRNA. Results: Higher TUG1 expression in LSCC than in paired normal tumor-adjacent tissue specimens (N = 64) was observed using quantitative real-time polymerase chain reaction. Also, high TUG1 expression was positively associated with advanced T category, worse lymph node metastasis and late clinical stage. Furthermore, in vitro experiments demonstrated that silencing of TUG1 markedly inhibited proliferation, cell-cycle progression, migration, and invasion of LSCC cells, whereas depletion of TUG1 led to increased apoptosis. Conclusion: These findings demonstrated that upregulated TUG1 expression exerted oncogenic effects by promoting proliferation, migration, and invasion, and inhibiting apoptosis in LSCC cells