The Trickle-down Impact of Reward (In-)consistency on RLHF

Standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against a Reward Model (RM), which itself is trained to reflect human preferences for desirable generations. A notable subject that is understudied is the (in-)consistency of RMs -- whether they can recognize the semantic changes to different prompts and appropriately adapt their reward assignments -- and their impact on the downstream RLHF model. In this paper, we visit a series of research questions relevant to RM inconsistency: (1) How can we measure the consistency of reward models? (2) How consistent are the existing RMs and how can we improve them? (3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training? We propose Contrast Instructions -- a benchmarking strategy for the consistency of RM. Each example in Contrast Instructions features a pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard ranking objective fail miserably on Contrast Instructions compared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques ConvexDA and RewardFusion, which enhance reward consistency through extrapolation during the RM training and inference stage, respectively. We show that RLHF models trained with a more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-down effect on the downstream RLHF process

Agilicious: Open-source and open-hardware agile quadrotor for vision-based flight

Autonomous, agile quadrotor flight raises fundamental challenges for robotics research in terms of perception, planning, learning, and control. A versatile and standardized platform is needed to accelerate research and let practitioners focus on the core problems. To this end, we present Agilicious, a codesigned hardware and software framework tailored to autonomous, agile quadrotor flight. It is completely open source and open hardware and supports both model-based and neural network–based controllers. Also, it provides high thrust-to-weight and torque-to-inertia ratios for agility, onboard vision sensors, graphics processing unit (GPU)–accelerated compute hardware for real-time perception and neural network inference, a real-time flight controller, and a versatile software stack. In contrast to existing frameworks, Agilicious offers a unique combination of flexible software stack and high-performance hardware. We compare Agilicious with prior works and demonstrate it on different agile tasks, using both model-based and neural network–based controllers. Our demonstrators include trajectory tracking at up to 5g and 70 kilometers per hour in a motion capture system, and vision-based acrobatic flight and obstacle avoidance in both structured and unstructured environments using solely onboard perception. Last, we demonstrate its use for hardware-in-the-loop simulation in virtual reality environments. Because of its versatility, we believe that Agilicious supports the next generation of scientific and industrial quadrotor research

Autoinflammatory Diseases: a Rapidly Advancing Field

Autoinflammatory diseases (AIDs) are a group of genetic disorders characterized by generalized inflammatory responses and multiorgan involvement primarily caused by dysregulated innate immunity. Since the introduction of this concept, AIDs has been a rapidly advancing research field including at least 56 diseases, deepening the understanding of the interaction between innate and adaptive immunity. Despite distinct features displayed by AIDs of different categories, genetic testing remains essential for highly suspected cases. The diagnosis of undifferentiated systemic autoinflammatory diseases, omics-powered precision stratification and targeted therapy for AIDs are promising research areas in the future. This article introduces the rapid progresses in AIDs concept, mechanism, and classification. We present a summary of the characteristic clinical phenotype, as well as the current diagnostic challenges and treatment experiences, in the hope of raising the awareness of these disorders

Protocol for the Development of the Guidelines for the Genetic Diagnosis of Autoinflammatory Diseases in China

Autoinflammatory diseases (AIDs) were defined over 20 years ago, and since then, 56 monogenic AIDs have been discovered. Due to the overlapping symptoms, AIDs are prone to misdiagnosis and mistreatment. Different AIDs are characterized by distinct genetic variants, and some AIDs have shown unique genetic characteristics in the Chinese population. Currently, there is no evidence-based guideline for the genetic diagnosis of AIDs in China. To further standardize the early recognition and precision diagnosis of AIDs, it is urgently needed to develop evidence-based clinical practice guidelines on the genetic diagnosis of AIDs to provide scientific guidance for the clinical management of AIDs. The Chinese Pharmacists Association Rare Diseases Medication Working Committee and the Subspecialty Group of Immunology from the Society of Pediatrics of the Chinese Medical Association have jointly launched a protocol for the development of the Guidelines for the Genetic Diagnosis of Autoinflammatory Diseases in China, which was registered and written following WHO Handbook for Guideline Development with the help of nationwide multidisciplinary experts organized by the Department of Pediatrics, Peking Union Medical College Hospital. A formal document of the guidelines will be devised and published following the workflow of evidence-based guideline development. This paper introduces the Guidelines for the Genetic Diagnosis of Autoinflammatory Diseases in China, including its background, significance, objectives, target population, guideline users, guideline working group members, and the workflow of guideline development

Analysis on Agronomic Characters and Quality of Erjingtiao Varieties in Different Production Areas of Sichuan

【Objective】A comprehensive evaluation on germplasm resources of high-quality local Erjingtiao pepper varieties in Sichuan was conducted, with a view to giving full play to their roles in new variety breeding and industrial processing.【Method】Erjingtiao pepper varieties from nine different production areas of Sichuan including Jianyang, Fushun, Xichong, Mianyang, Santai, Jiajiang, Shuangliu, Huidong and Qionglai were selected as the research materials, and their agronomic traits, nutritional quality indexes and flavor characteristics were comprehensively evaluated by field observation, quality detection and electronic nose analysis.【Result】In terms of agronomic traits, there were significant differences among different varieties in the node of first flower, main stem height, stem diameter, plant height, crown length, crown width, leaf length and leaf width. The mean values were 6.83~12.67, 18.08~27.33 cm, 9.08~15.39 mm, 98.17~141.5 cm, 86.33~136.67 cm, 69.75~117.50 cm, 9.62~15.86 cm and 2.74~4.56 cm, respectively. The differences in the appearance of different varieties of fruits were relatively small, the fruit shape indexes were all above 10, with red mature fruits, glossy fruit surface, soft fruit texture and wrinkled fruit skin, which were the typical fruit characteristics of Erjingtiao pepper. However, there was a great difference in fruit length among different varieties, with the longest fruit length of Qionglai Erjingtiao (25.02 cm) and the shortest fruit length of Huidong Erjingtiao (11.23 cm). However, the difference of fruit width was little, ranging from 1.08 to 1.76 cm. In terms of nutritional quality, there were significant differences among different varieties. The variable coefficients were between 8.35 to 45.72, of which capsaicin content had the greatest difference, with the highest content in Qionglai Erjingtiao (0.54 mg/g), and the lowest one in Xichong Erjingtiao (0.13 mg/g). However, except for Jianyang Erjingtiao, with the lowest color value of only 7.35, there was no significant difference in color value among other varieties. Meanwhile, there were some correlations among different quality indexes, and soluble sugar was significantly positively correlated with total protein and Vc, which was conducive to screening pepper varieties with high nutrition. The first three varieties of Erjingtiao with better comprehensive quality traits were selected by membership function analysis. The Qionglai Erjingtiao, ranking first, belonged to the high-spicy and high-nutrient variety, followed by Shuangliu Erjingtiao and Xichong Erjingtiao, belonged to the varieties of medium-spicy and low-spicy with high nutrition, respectively. The fruit flavor characteristics of different Erjingtiao varieties were similar, and only some varieties could be distinguished by electronic nose, but the flavor of the pulp and seed of the same pepper variety could be distinguished well.【Conclusion】The varieties of Erjingtiao pepper in different production areas of Sichuan have formed various agronomic traits and quality characteristics through long-term natural and artificial selection, and they can be screened according to different needs in the process of germplasm resources utilization and product selection

Reproducibility and Validity of a Food Frequency Questionnaire for Assessing Dietary Consumption via the Dietary Pattern Method in a Chinese Rural Population.

This study was conducted to assess the reproducibility and validity of a food frequency questionnaire (FFQ) that was developed to assess the overall dietary consumption via dietary pattern method in a rural population in southwest China.A total of 179 participants aged between 40 and 70 years old were included in this study. Participants administered FFQ at baseline (FFQ1) and one year later (FFQ2) to assess the reproducibility. Six 3-day 24-hour recalls (24HRs) were completed between the administrations of two FFQs to determine the validity. Dietary patterns from three separate dietary sources were derived by using principle component factor analysis. Comparisons between dietary pattern scores were made by using Pearson or intraclass correlation coefficient, cross-classification analysis, weighted kappa (κ) statistic and Bland-Altman analysis. The de-attenuated method was adopted to correct the monthly and seasonally variation and the partial correlation analysis was used correct the influence by total energy intake.Two major dietary factors, labeled as prudent pattern and processed food pattern, were identified. The prudent pattern was characterized by higher factor loadings of wheat, rice, fresh vegetables, bean products, nuts, red meat, white meat and fresh eggs; and the processed food pattern was characterized by higher factor loadings of pickled vegetables, preserved vegetables and salted meat. Between Two FFQs, intraclass correlation coefficients were 0.57 for prudent pattern and 0.55 for processed food pattern, partial Pearson correlation coefficients were 0.51 for the prudent pattern and 0.56 for the processed food pattern; weighted κ statistic ranged from 0.45 (for the prudent pattern) to 0.56 (for the processed food pattern). Between FFQs and 24HRs, de-attenuated correlation coefficients ranged from 0.54 to 0.78 for the prudent pattern and from 0.55 to 0.61 for the processed food pattern; partial Pearson correlation coefficients ranged from 0.41 to 0.56 for the prudent pattern and from 0.42 to 0.44 for the processed food pattern; weighted κ statistic ranged from 0.42 to 0.46 for prudent pattern and from 0.43 to 0.60 for processed food pattern. The Bland-Altman plots and limits of agreement indicated that the deviation/divergence was not obvious for both of the patterns between FFQ1 and FFQ2 and between FFQs and 24HRs.The study suggests that the FFQ is reasonably reproducible and valid to assess the overall dietary consumption via dietary pattern methods in the Chinese rural population

Abscisic acid and polyamines coordinately regulate strawberry drought responses

Abscisic acid (ABA) and polyamines (PAs) play an important role in plant stress responses; however, their cooperative mechanisms are not fully understood. Here, using the detached-, PEG 6000- and pot-drought strawberry leaves, we investigated the expression changes of genes key to ABA and PA homeostasis by a combination of qRT-PCR, western blotting, RNA-sequencing, physiological analyses, and pharmacological tests. The results showed that early drought may promote the production of both putrescine (Put) by FaADC1 and ABA by FaNCED1, then these productions could be further facilitated by the accumulated Put. Meanwhile, the high accumulation of ABA not only further accelerated ABA biosynthesis by FaNCED3, but also controlled PA homeostasis by a decrease in FaADC1 expression and an increase in FaSAMDC1 and FaPAO5 expression. Based on our RNA-seq data, Put was involved in photosynthesis and oxidative phosphorylation, while ABA was implicated in sugar and flavonoid metabolism, both of which were linked to auxin (IAA). Altogether, in response to strawberry drought stress, FaNCED1/3 and FaADC1/FaSAMDC1 participated separately in ABA and PA accumulation, mostly FaNCED1/FaADC1 was induced by stress and FaNCED3/FaSAMDC1 was induced by ABA. Finally, a balance between ABA and PAs for the cooperative control of plant growth and adaptation is proposed through IAA, photosynthesis, oxidative phosphorylation, and flavonoid metabolism, among which ABA may act in a Put-dependent manner

Hsa_circ_0009092/miR-665/NLK signaling axis suppresses colorectal cancer progression via recruiting TAMs in the tumor microenvironment

Abstract Background It has been demonstrated that circularRNA (circRNAs) plays a critical role in various cancers. While the potential molecular mechanism of circRNAs in the progression of colorectal cancer (CRC) remains uncertain. Methods Differentially expressed circRNAs were identified by RNA sequencing. RT-qPCR detected the expression of circ_0009092, miR-665, and NLK in CRC tissues and cells. Functions of circ_0009092 on tumor cell proliferation, migration, and invasion were investigated by a series of in vitro assays. The underlying mechanism of circ_0009092 was explored by bioinformatics analysis, RNA immunoprecipitation (RIP) and luciferase assays. A co-culture assay in vitro was performed to detect the affection of circ_0009092 on macrophage recruitment in the tumor microenvironment (TME). A xenograft mouse model was used to explore the effect of circ_0009092 on tumor growth. Results Circ_0009092 was downregulated in CRCand predicted a good prognosis. Overexpression of circ_0009092 reduced tumor cell EMT, proliferation, migration, and invasion in vitro and in vivo. Mechanistically, circ_0009092 elevated the NLK expression via sponging miR-665 and suppressed the Wnt/β-catenin signaling pathway. EIF4EA3 induced circ_0009092 expression in CRC cells. In addition, NLK regulates phosphorylation and O-GlcNAcylation of STAT3 by binding to STAT3, thereby inhibiting CCL2 expression, in which it inhibits macrophage recruitment in the tumor microenvironment (TME). Conclusion EIF4A3 suppressed circ_0009092 biogenesis, whichinhibits CRC progression by sponging miR-665 to downregulate NLK. Circ_0009092/miR-665/NLK suppressed tumor EMT, proliferation, migration, and invasion by acting on the Wnt/β-catenin signaling pathway. NLK directly interacted with STAT3 and decreased the CCL2 expression, inhibiting the recruitment of tumor-associated macrophages (TAMs) in the TME. Our study provided novel insights into the roles of circ_0009092 as a novel promising prognostic and therapeutic target in CRC

Comparative Genome Analysis of Scutellaria baicalensis and Scutellaria barbata Reveals the Evolution of Active Flavonoid Biosynthesis

Scutellaria baicalensis (S. baicalensis) and Scutellaria barbata (S. barbata) are common medicinal plants of the Lamiaceae family. Both produce specific flavonoid compounds, including baicalein, scutellarein, norwogonin, and wogonin, as well as their glycosides, which exhibit antioxidant and antitumor activities. Here, we report chromosome-level genome assemblies of S. baicalensis and S. barbata with quantitative chromosomal variation (2n = 18 and 2n = 26, respectively). The divergence of S. baicalensis and S. barbata occurred far earlier than previously reported, and a whole-genome duplication (WGD) event was identified. The insertion of long terminal repeat elements after speciation might be responsible for the observed chromosomal expansion and rearrangement. Comparative genome analysis of the congeneric species revealed the species-specific evolution of chrysin and apigenin biosynthetic genes, such as the S. baicalensis-specific tandem duplication of genes encoding phenylalanine ammonia lyase and chalcone synthase, and the S. barbata-specific duplication of genes encoding 4-CoA ligase. In addition, the paralogous duplication, colinearity, and expression diversity of CYP82D subfamily members revealed the functional divergence of genes encoding flavone hydroxylase between S. baicalensis and S. barbata. Analyzing these Scutellaria genomes reveals the common and species-specific evolution of flavone biosynthetic genes. Thus, these findings would facilitate the development of molecular breeding and studies of biosynthesis and regulation of bioactive compounds