Safe RLHF: Safe Reinforcement Learning from Human Feedback

With the development of large language models (LLMs), striking a balance between the performance and safety of AI systems has never been more critical. However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training. To address this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly decouples human preferences regarding helpfulness and harmlessness, effectively avoiding the crowdworkers' confusion about the tension and allowing us to train separate reward and cost models. We formalize the safety concern of LLMs as an optimization task of maximizing the reward function while satisfying specified cost constraints. Leveraging the Lagrangian method to solve this constrained problem, Safe RLHF dynamically adjusts the balance between the two objectives during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we demonstrate a superior ability to mitigate harmful responses while enhancing model performance compared to existing value-aligned algorithms. Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with collected human preferences, significantly improving its helpfulness and harmlessness according to human evaluations

Sulfur and mercury MIF suggest volcanic contributions to Earth’s atmosphere at 2.7 Ga

This study received funding from a Natural Environment Research Council Standard Grant NE/M001156/1 (ALZ, EGN), and from the European Research Council under the European Union’s Horizon 2020 research and innovation programme (Grant 678812 to MWC).The Archean eon is associated with large-scale changes in Earth’s geosphere and biosphere, including the onset of plate tectonics and the expansion of oxygenic photosynthesis, although the full impacts of these changes on the atmosphere remain unclear. Here we present coupled records of mass independent fractionation of sulfur (S-MIF) and mercury (Hg-MIF) isotopes from well preserved sediments of the ∼2.7 billion year old (Ga) Manjeri Formation, Belingwe Greenstone Belt, Zimbabwe. These palaeoatmospheric proxies record different trends for S-MIF and odd number Hg-MIF versus even number Hg-MIF, providing novel constraints on atmospheric chemistry during this time. S-MIF and odd number Hg-MIF values are muted in comparison to values preserved in later Archean sediments, representing a combination of enhanced volcanic input and local mixing. Even number Hg-MIF is absent from these sediments, consistent with complete photo-oxidation of gaseous Hg0, which could have been driven by increased halogen emissions from arc volcanism. When considered within a global geodynamic context, these MIF data suggest an important role for subduction zone-related volcanism associated with early plate tectonics in modulating the ∼2.7 Ga atmosphere.Publisher PDFPeer reviewe

Effects of heat treatment on the structure, digestive property, and absorptivity of holoferritin

Ferritin, as an iron storage protein, has been considered to be a well-utilized iron supplement. However, the typical thermal processing of food rich in ferritin may affect the structure and function of ferritin as an iron supplement. Here, a plant ferritin (soybean seed ferritin, SSF) and an animal ferritin (donkey spleen ferritin, DSF) were used to analyze the changes in fundamental structure and iron content after thermal treatments (68 °C for 30 min, 100 °C for 10 min). Then, SSF and DSF after thermal treatment were administered intragastrically to mice to further evaluate its digestive stability and absorptivity after thermal processing. Results showed the secondary structure, oligomeric states, iron content, and digestive stability of DSF were maintained better than that of SSF after thermal treatments, indicating that DSF has a higher thermostability than SSF. Both SSF and DSF after thermal treatment exhibited higher absorptivity than untreated ferritins. SSF showed higher absorptivity than DSF whether heated or not

A vector spectrum analyzer of 55.1 THz spectral bandwidth and 99 kHz frequency resolution

The analysis of optical spectra - emission or absorption - has been arguably the most powerful approach for discovering and understanding matters. The invention and development of many kinds of spectrometers have equipped us with versatile yet ultra-sensitive diagnostic tools for trace gas detection, isotope analysis, and resolving hyperfine structures of atoms and molecules. With proliferating data and information, urgent and demanding requirements have been placed today on spectrum analysis with ever-increasing spectral bandwidth and frequency resolution. These requirements are especially stringent for broadband laser sources that carry massive information, and for dispersive devices used in information processing systems. In addition, spectrum analyzers are expected to probe the device's phase response where extra information is encoded. Here we demonstrate a novel vector spectrum analyzer (VSA) that is capable to characterize passive devices and active laser sources in one setup. Such a dual-mode VSA can measure loss, phase response and dispersion property of passive devices. It also can coherently map a broadband laser spectrum into the RF domain. The VSA features a bandwidth of 55.1 THz (1260 to 1640 nm), frequency resolution of 99 kHz, and dynamic range of 56 dB. Meanwhile, our fiber-based VSA is compact and robust. It requires neither high-speed modulators and photodetectors, nor any active feedback control. Finally, we successfully employ our VSA for applications including characterization of integrated dispersive waveguides, mapping frequency comb spectra, and coherent light detection and ranging (LiDAR). Our VSA presents an innovative approach for device analysis and laser spectroscopy, and can play a critical role in future photonic systems and applications for sensing, communication, imaging, and quantum information processing

Large AI Models in Health Informatics: Applications, Challenges, and the Future

Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which can reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A prime example is ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our lives. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multi-modal data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents a comprehensive review of large AI models, from background to their applications. We identify seven key sectors in which large AI models are applicable and might have substantial influence, including 1) bioinformatics; 2) medical diagnosis; 3) medical imaging; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.Comment: This article has been accepted for publication in IEEE Journal of Biomedical and Health Informatic

Baichuan 2: Open Large-scale Language Models

Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.Comment: Baichuan 2 technical report. Github: https://github.com/baichuan-inc/Baichuan