SALMON: Self-Alignment with Principle-Following Reward Models

Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents. However, a significant limitation of such an approach is its dependency on high-quality human annotations, making its application to intricate tasks challenging due to difficulties in obtaining consistent response demonstrations and in-distribution response preferences. This paper presents a novel approach, namely SALMON (Self-ALignMent with principle-fOllowiNg reward models), to align base language models with minimal human supervision, using only a small set of human-defined principles, yet achieving superior performance. Central to our approach is a principle-following reward model. Trained on synthetic preference data, this model can generate reward scores based on arbitrary human-defined principles. By merely adjusting these principles during the RL training phase, we gain full control over the preferences with the reward model, subsequently influencing the behavior of the RL-trained policies, and eliminating the reliance on the collection of online human preferences. Applying our method to the LLaMA-2-70b base language model, we developed an AI assistant named Dromedary-2. With only 6 exemplars for in-context learning and 31 human-defined principles, Dromedary-2 significantly surpasses the performance of several state-of-the-art AI systems, including LLaMA-2-Chat-70b, on various benchmark datasets. We have open-sourced the code and model weights to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, improved controllability, and scalable oversight.Comment: Project page: https://github.com/IBM/SALMO

Genetic evaluation for production and body size traits using different animal models in purebred-Duroc pigs

Duroc pigs are popular crossbred terminal sires, and accurate assessment of genetic parameters in the population can help to rationalize breeding programmes. The principle aim of this study were to evaluate the genetic parameters of production (birth weight, BW; age at 115 kg, AGE; feed conversion ratio, FCR) and body size (body length, BL; body height, BH; front cannon circumference, FCC) traits of Duroc pigs. The second objective was to analyze the fit of different genetic assessment models. The variance components and correlations of BW (28,348 records), AGE (28,335 records), FCR (11,135 records), BL (31,544 records), BH (21,862 records), and FCC (14,684 records) traits were calculated by using DMU and AIREMLF90 from BLUPF90 package. In the common environment model, the heritability of BW, AGE, FCR, BL, BH, and FCC traits were 0.17 ± 0.014, 0.30 ± 0.019, 0.28 ± 0.024, 0.16 ± 0.013, 0.14 ± 0.017, and 0.081 ± 0.016, with common litter effect values of 0.25, 0.20, 0.18, 0.23, 0.19, and 0.16, respectively. According to the results of the Akaike information criterion (AIC) calculations, models with smaller AIC values have a better fit. We found that the common environment model with litter effects as random effects for estimating genetic parameters had a better fit. In this Model, the estimated genetic correlations between AGE with BW, FCR, BL, BH, and FCC traits were −0.28 (0.040), 0.76 (0.038), −0.71 (0.036), −0.44 (0.060), and −0.60 (0.073), respectively, with phenotypic correlations of −0.17, 0.52, −0.22, −0.13 and −0.24, respectively. In our analysis of genetic trends for six traits in the Duroc population from 2012 to 2021, we observed significant genetic trends for AGE, BL, and BH. Particularly noteworthy is the rapid decline in the genetic trend for AGE, indicating an enhancement in the pig's growth rate through selective breeding. Therefore, we believe that some challenging-to-select traits can benefit from the genetic correlations between traits. By selecting easily measurable traits, they can gain from synergistic selection effects, leading to genetic progress. Conducting population genetic parameter analysis can assist us in devising breeding strategies

Quantitative characterization of the disturbance of groundwater system in typical coal seam mining in contiguous area of Inner Mongolia and Shaanxi

High-intensity mining of coal mines in ecologically fragile areas disturbs the groundwater system and causes water resources loss. The typical shallow and deep mining mines in the middle section of the Yellow River basin are selected as the research objects, and the development height of the water-conducting fracture zone in the shallow composite coal seam mining and the deep single coal seam mining is comprehensively determined by using the theoretical analysis, numerical simulation of overburden damage and the underground measurement, and the comparison with surrounding mines with similar conditions. According to the spatial location relationship between the water-conducting fracture zone and the main roof aquifer/aquiclude, the water filling model of Zhiluo Formation - Zhidan Group aquifer in shallow coal seam mining and that of Zhiluo Formation aquifer in deep coal seam mining are divided; By generalizing the groundwater flow system of different aquifer water filling modes, the three-dimensional unsteady flow mathematical model of groundwater in typical coal seam mining is constructed, and the numerical model of groundwater flow in two working conditions is established by using Visual Modflow software. The quantitative prediction of groundwater flow field and water resource loss in the mining of the continuous working face is carried out, and the results show that the groundwater from Zhidan Group and Zhiluo Formation aquifers in the disturbed aquifer is converging towards the goaf at the same time, the flow field also gradually recovers with the formation of the secondary stable structure of the roof after mining. Around the goaf, the groundwater flow field shows a local high hydraulic gradient phenomenon; In addition, the average annual water loss of the two coal mining conditions in the continuous period is more than 2 million m3 and 7.3 million m3 respectively. The research results provide a reference for the advanced fine control of regional water resources loss and the utilization of mine water resources

Exploring the Fecal Microbial Composition and Metagenomic Functional Capacities Associated With Feed Efficiency in Commercial DLY Pigs

Gut microbiota has indispensable roles in nutrient digestion and energy harvesting, especially in processing the indigestible components of dietary polysaccharides. Searching for the microbial taxa and functional capacity of the gut microbiome associated with feed efficiency (FE) can provide important knowledge to increase profitability and sustainability of the swine industry. In the current study, we performed a comparative analysis of the fecal microbiota in 50 commercial Duroc × (Landrace × Yorkshire) (DLY) pigs with polarizing FE using 16S rRNA gene sequencing and shotgun metagenomic sequencing. There was a different microbial community structure in the fecal microbiota of pigs with different FE. Random forest analysis identified 24 operational taxonomic units (OTUs) as potential biomarkers to improve swine FE. Multiple comparison analysis detected 8 OTUs with a significant difference or tendency toward a difference between high- and low-FE pigs (P < 0.01, q < 0.1). The high-FE pigs had a greater abundance of OTUs that were from the Lachnospiraceae and Prevotellaceae families and the Escherichia-Shigella and Streptococcus genera than low-FE pigs. A sub-species Streptococcus gallolyticus subsp. gallolyticus could be an important candidate for improving FE. The functional capacity analysis found 18 KEGG pathways and CAZy EC activities that were different between high- and low-FE pigs. The fecal microbiota in high FE pigs have greater functional capacity to degrade dietary cellulose, polysaccharides, and protein and may have a greater abundance of microbes that can promote intestinal health. These results provided insights for improving porcine FE through modulating the gut microbiome

Production of Transgenic Pigs Mediated by Pseudotyped Lentivirus and Sperm

Sperm-mediated gene transfer can be a very efficient method to produce transgenic pigs, however, the results from different laboratories had not been widely repeated. Genomic integration of transgene by injection of pseudotyped lentivirus to the perivitelline space has been proved to be a reliable route to generate transgenic animals. To test whether transgene in the lentivirus can be delivered by sperm, we studied incubation of pseudotyped lentiviruses and sperm before insemination. After incubation with pig spermatozoa, 62±3 lentiviral particles were detected per 100 sperm cells using quantitative real-time RT-PCR. The association of lentivirus with sperm was further confirmed by electron microscopy. The sperm incubated with lentiviral particles were artificially inseminated into pigs. Of the 59 piglets born from inseminated 5 sows, 6 piglets (10.17%) carried the transgene based on the PCR identification. Foreign gene and EGFP was successfully detected in ear tissue biopsies from two PCR-positive pigs, revealed via in situ hybridization and immunohistochemistry. Offspring of one PCR-positive boar with normal sows showed PCR-positive. Two PCR-positive founders and offsprings of PCR-positive boar were further identified by Southern-blot analysis, out of which the two founders and two offsprings were positive in Southern blotting, strongly indicating integration of foreign gene into genome. The results indicate that incubation of sperm with pseudotyped lentiviruses can incorporated with sperm-mediated gene transfer to produce transgenic pigs with improved efficiency

Spatial inequity in access to healthcare facilities at a county level in a developing country: a case study of Deqing County, Zhejiang, China

Background The inequities in healthcare services between regions, urban and rural, age groups and diverse income groups have been growing rapidly in China. Equal access to basic medical and healthcare services has been recognized as “a basic right of the people” by Chinese government. Spatial accessibility to healthcare facilities has received huge attention in Chinese case studies but been less studied particularly at a county level due to limited availability of high-resolution spatial data. This study is focused on measuring spatial accessibility to healthcare facilities in Deqing County. The spatial inequity between the urban (town) and rural is assessed and three scenarios are designed and built to examine which scenario is instrumental for better reducing the spatial inequity. Methods This study utilizes highway network data, Digital Elevation Model (DEM), location of hospitals and clinics, 2010 census data at the finest level – village committee, residential building footprint and building height. Areal weighting method is used to disaggregate population data from village committee level to residential building cell level. Least cost path analysis is applied to calculate the travel time from each building cell to its closest healthcare facility. Then an integral accessibility will be calculated through weighting the travel time to the closest facility between three levels. The spatial inequity in healthcare accessibility between the town and rural areas is examined based on the coverages of areas and populations. The same method is used to compare three scenarios aimed at reducing such spatial inequity – relocation of hospitals, updates of weighting values, and the combination of both. Results 50.03 % of residents can reach a county hospital within 15 min by driving, 95.77 % and 100 % within 30 and 60 min respectively. 55.14 % of residents can reach a town hospital within 5 min, 98.04 % and 100 % within 15 and 30 min respectively. 57.86 % of residential building areas can reach a village clinic within 5 min, 92.65 % and 99.22 % within 10 and 15 min. After weighting the travel time between the three-level facilities, 30.87 % of residents can reach a facility within 5 min, 80.46 %% and 99.88 % within 15 and 30 min respectively. Conclusions The healthcare accessibility pattern of Deqing County has exhibited spatial inequity between the town and rural areas, with the best accessibility in the capital of the county and poorest in the West of the county. There is a high negative correlation between population ageing and healthcare accessibility. Allocation of more advanced medical and healthcare equipment and highly skillful doctors and nurses to village clinics will be an efficient means of reducing the spatial inequity and further consolidating the national medical security system. GIS (Geographical Information Systems) methods have proven successful method of providing quantitative evidence for policy analysis although the data sets and methods could be further improved

A Resource Recommendation Algorithm for Online English Learning Systems Based on Learning Ability Evaluation

Mature online English learning platforms should provide students with necessary learning resources, ensure efficient access to learning projects, and offer the optimal learning experience. However, the traditional recommendation methods for English learning resources cannot satisfy the in-depth learning demand of students. To solve the problem, this paper designs a resource recommendation algorithm for online English learning systems based on learning ability evaluation. Firstly, the workflow of the designed algorithm was introduced, and a four-layer test system was developed for students’ English learning ability evaluation. Next, an English learning ability evaluation method was proposed based on the maximum expectation algorithm, as well as the estimation methods for parameters like learning ability, degree of discrimination, difficulty, and guess coefficient. Experimental results demonstrated the good effect of the proposed resource recommendation algorithm. The research findings provide a reference for resource recommendation of other online learning systems

Space-borne SAR Three-dimensional Imaging by Joint Multiple Azimuth Angle Doppler Frequency Rate Estimation

Using azimuth beam steering, space-borne Synthetic Aperture Radar (SAR) can observe from multiple azimuth angle in single pass. During multiple azimuth angle observation, the satellite orbit is equivalent to long three-dimensional (3-D) curvilinear array, which has the potential for 3-D imaging. Sampling by space-borne multiple azimuth angle SAR is sparse in height dimension, which makes unambiguous 3-D imaging by 3-D fast Fourier transform infeasible. Besides, the complex relationship between the targets’projection in multi-angle SAR images and the height error is difficult to determine. To overcome this limitation, in this paper, we present a 3-D imaging method based on joint multiple azimuth angle Doppler frequency rate estimation. First, a relationship is proposed between the height error and Doppler frequency rate at different azimuth angle. Then, the Doppler frequency modulation rate error is estimated by Map Drift (MD) technique. Next, the height estimation results of different azimuth angle are combined to improve the estimation accuracy. From the estimated height and the target location in the SAR images, 3-D geometric information is retrieved and 3-D imaging is achieved. Our simulation experiments validate that the height estimation can achieve an accuracy of few meters with the proposed method