A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

Offline reinforcement learning (RL) provides a promising solution to learning an agent fully relying on a data-driven paradigm. However, constrained by the limited quality of the offline dataset, its performance is often sub-optimal. Therefore, it is desired to further finetune the agent via extra online interactions before deployment. Unfortunately, offline-to-online RL can be challenging due to two main challenges: constrained exploratory behavior and state-action distribution shift. To this end, we propose a Simple Unified uNcertainty-Guided (SUNG) framework, which naturally unifies the solution to both challenges with the tool of uncertainty. Specifically, SUNG quantifies uncertainty via a VAE-based state-action visitation density estimator. To facilitate efficient exploration, SUNG presents a practical optimistic exploration strategy to select informative actions with both high value and high uncertainty. Moreover, SUNG develops an adaptive exploitation method by applying conservative offline RL objectives to high-uncertainty samples and standard online RL objectives to low-uncertainty samples to smoothly bridge offline and online stages. SUNG achieves state-of-the-art online finetuning performance when combined with different offline RL methods, across various environments and datasets in D4RL benchmark.Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessibl

Evaluating the Individualized Treatment of Traditional Chinese Medicine: A Pilot Study of N-of-1 Trials

Purpose. To compare the efficacy of individualized herbal decoction with controlled decoction for individual patients with stable bronchiectasis. Methods. We conducted N-of-1 RCTs (single-patient, double-blind, randomized, multiple crossover design) in 3 patients with stable bronchiectasis. The primary outcome was patient self-rated symptom scores on visual analogue scales. Secondary outcome was 24-hour sputum volume. A clinical efficacy criterion which combined symptoms score and medication preference was also formulated. Results. All three patients showed various degrees of improvement on their symptoms and one patient’s (Case 3) 24 h sputum volume decreased from 70 mL to 30 mL. However, no significant differences were found between individualized herbal decoction and control decoction on symptoms score, or on 24-hour sputum volume. One patient (Case 2) had clear preference for the individualized herbal decoction over the standard one with the confirmation after unblinding. We therefore considered this case as clinically important. Discussion. N-of-1 trials comply with individualized philosophy of TCM clinical practice and had good compliance. It is necessary to set up clinical efficacy criteria and to consider the interference of acute exacerbation

Ionic Liquid-Based Microwave-Assisted Extraction of Flavonoids from Bauhinia championii (Benth.) Benth.

An ionic liquids (IL)-based microwave-assisted approach for extraction and determination of flavonoids from Bauhinia championii (Benth.) Benth. was proposed for the first time. Several ILs with different cations and anions and the microwave-assisted extraction (MAE) conditions, including sample particle size, extraction time and liquid-solid ratio, were investigated. Two M 1-butyl-3-methylimidazolium bromide ([bmim] Br) solution with 0.80 M HCl was selected as the optimal solvent. Meanwhile the optimized conditions a ratio of liquid to material of 30:1, and the extraction for 10 min at 70 °C. Compared with conventional heat-reflux extraction (CHRE) and the regular MAE, IL-MAE exhibited a higher extraction yield and shorter extraction time (from 1.5 h to 10 min). The optimized extraction samples were analysed by LC-MS/MS. IL extracts of Bauhinia championii (Benth.)Benth consisted mainly of flavonoids, among which myricetin, quercetin and kaempferol, β-sitosterol, triacontane and hexacontane were identified. The study indicated that IL-MAE was an efficient and rapid method with simple sample preparation. LC-MS/MS was also used to determine the chemical composition of the ethyl acetate/MAE extract of Bauhinia championii (Benth.) Benth, and it maybe become a rapid method to determine the composition of new plant extracts

Ionic Liquid-Based Microwave-Assisted Extraction of Flavonoids from Bauhinia championii (Benth.) Benth.

An ionic liquids (IL)-based microwave-assisted approach for extraction and determination of flavonoids from Bauhinia championii (Benth.) Benth. was proposed for the first time. Several ILs with different cations and anions and the microwave-assisted extraction (MAE) conditions, including sample particle size, extraction time and liquid-solid ratio, were investigated. Two M 1-butyl-3-methylimidazolium bromide ([bmim] Br) solution with 0.80 M HCl was selected as the optimal solvent. Meanwhile the optimized conditions a ratio of liquid to material of 30:1, and the extraction for 10 min at 70 °C. Compared with conventional heat-reflux extraction (CHRE) and the regular MAE, IL-MAE exhibited a higher extraction yield and shorter extraction time (from 1.5 h to 10 min). The optimized extraction samples were analysed by LC-MS/MS. IL extracts of Bauhinia championii (Benth.)Benth consisted mainly of flavonoids, among which myricetin, quercetin and kaempferol, β-sitosterol, triacontane and hexacontane were identified. The study indicated that IL-MAE was an efficient and rapid method with simple sample preparation. LC-MS/MS was also used to determine the chemical composition of the ethyl acetate/MAE extract of Bauhinia championii (Benth.) Benth, and it maybe become a rapid method to determine the composition of new plant extracts

Chlorogenic acid ameliorates heart failure by attenuating cardiomyocyte ferroptosis

Objective: To elucidate the effects of chlorogenic acid (CGA), a bioactive polyphenol compound prevalent in traditional Chinese medicine and various foods, including Lonicera japonica Thunb. (Jin Yin Hua), Eucommia ulmoides Oliv. (Du Zhong Ye), tea, and coffee, on cardiomyocyte ferroptosis and heart failure. Methods: We assessed the effect of CGA on cardiac function using a mouse model of heart failure induced by transverse aortic constriction (TAC). These indicators included the left ventricular ejection fraction (LVEF), fractional shortening (LVFS), end-systolic volume (LVESV), end-diastolic volume (LVEDV), end-systolic diameter (LVESD), and end-diastolic diameter (LVEDD). An isoprenaline hydrochloride (ISO)-induced H9c2 cardiomyocyte cell model was also established, and the cells were treated with various concentrations of CGA. To assess the effect of CGA on ferroptosis in cardiomyocytes, we measured cell viability and evaluated the levels of intracellular reactive oxygen species (ROS), ferrous ions (Fe2+), and lipid peroxidation using fluorescent staining. To clarify the ferroptosis signaling pathway regulated by CGA, western blotting was used to examine the expression of ferroptosis biomarkers, specifically solute carrier family 7 member 11 (SLC7A11) and glutathione peroxidase 4 (GPX4), in H9c2 cardiomyocytes and mouse myocardial tissues. Results: CGA significantly enhanced cardiac performance indices such as LVEF, LVFS, LVESV, LVEDV, LVESD, and LVEDD. H9c2 cardiomyocytes exposed to ISO showed decreased cell viability and increased ROS levels, Fe2+ content, and lipid peroxidation levels. However, CGA treatment significantly ameliorated these changes. Additionally, in both H9c2 cardiomyocytes and myocardial tissue obtained from mice with TAC, CGA increased the expression of ferroptosis-related proteins, including SLC7A11 and GPX4. Conclusion: CGA has the potential to enhance cardiac function and diminish lipid peroxidation and ROS levels in cardiomyocytes via the SLC7A11/GPX4 signaling pathway. This process alleviates ferroptosis in cardiomyocytes. These results provide new insights into the clinical use of CGA and the management of heart failure

Identification of angiotensin‐converting enzyme inhibitory peptides from peanut meal (Arachis hypogaea Linn) fermented by Lactobacillus pentosus using MALDI‐TOF–MS and LC–MS/MS

Abstract This study focused on the production of angiotensin‐converting enzyme inhibitory peptides (ACEIPs) from peanut meal (Arachis hypogaea Linn) fermented by Lactobacillus pentosus. The fermentation process was optimized using the response surface methodology with ACE inhibitory activity as the experimental indicator. ACEIPs were further purified after fermentation using ultrafiltration and Sephadex G‐25 gel chromatography. The effect of different molecular weights (ranging from 0.5 to 1.5 kDa) of ACEIP on ACE inhibitory activity was investigated, and a maximum inhibitory rate of 48.83% was achieved. The content of ACEIP was 78.95%. Amino acid analysis revealed that the hydrophobic amino acids accounted for 43.09% of the total content. Among the identified amino acids, glutamic acid had the highest content of 14.94%, followed by leucine and aspartic acid. Matrix‐assisted laser desorption/ionization time‐of‐flight–mass spectrometry (MS) and liquid chromatography–tandem MS were used to identify the molecular weights of the selected ACEIPs, yielding six ACEIPs with good stability and high hydrophilicity. Flexible docking of the six ACEIPs with ACE was simulated using AutoDock Vina (v1.5.7). The result showed that the ACEIPs formed 11, 8, 7, 9, 7, and 6 hydrogen bonds with ACE residues, and the lowest binding energies between them were −9.8, −8.1, −9.0, −9.3, −8.2, and−9.1 kcal/mol, respectively. Among them, GFGINAENNHRIF exhibited superior ACE inhibitory activity and binding stability

Investigation into the Individualized Treatment of Traditional Chinese Medicine through a Series of N-of-1 Trials

Purpose. To compare the efficacy of individualized herbal decoction with standard decoction for patients with stable bronchiectasis through N-of-1 trials. Methods. We conducted a single center N-of-1 trials in 17 patients with stable bronchiectasis. Each N-of-1 trial contains three cycles. Each cycle is divided into two 4-week intervention including individualized decoction and fixed decoction (control). The primary outcome was patient self-reported symptoms scores on a 1–7 point Likert scale. Secondary outcomes were 24-hour sputum volume and CAT scores. Results. Among 14 completed trials, five showed that the individualized decoction was statistically better than the control decoction on symptom scores (P<0.05) but was not clinically significant. The group data of all the trials showed that individualized decoction was superior to control decoction on symptom scores (2.13±0.58 versus 2.30±0.65, P=0.002, mean difference and 95% CI: 0.18 (0.10, 0.25)), 24 h sputum volume (P=0.009), and CAT scores (9.69±4.89 versus 11.64±5.59, P=0.013, mean difference and 95% CI: 1.95 (1.04, 2.86)) but not clinically significant. Conclusion. Optimizing the combined analysis of individual and group data and the improvement of statistical models may make contribution in establishing a method of evaluating clinical efficacy in line with the characteristics of traditional Chinese medicine individual diagnosis and treatment

A Series of N-of-1 Trials for Traditional Chinese Medicine Using a Bayesian Method: Study Rationale and Protocol

Background. Our previous studies showed that N-of-1 trials could reflect the individualized characteristics of traditional Chinese medicine (TCM) syndrome differentiation with good feasibility, but the sensitivity was low. Therefore, this study will use hierarchical Bayesian statistical method to improve the sensitivity and applicability of N-of-1 trials of TCM. Methods/Design. This is a randomized, double-blind, placebo-controlled, three-pair crossover trial for a single subject, including 4–8 weeks of run-in period and 24 weeks of formal trial. In this study, we will recruit a total of 30 participants who are in the stable stage of bronchiectasis. The trial will be divided into three pairs (cycles), and one cycle contains two observation periods. The medications will be taken for three weeks and stopped for one week in the last week of each observation period. The order of syndrome differentiation decoction and placebo will be randomly determined. Patient self-reported symptom score (on a 7-point Likert scale) is the primary outcome. Discussion. Some confounding variables (such as TCM syndrome type and potential carryover effect of TCM) will be introduced into hierarchical Bayesian statistical method to improve the sensitivity and applicability of N-of-1 trials of TCM, and the use of prior available information (e.g., “borrowing from strength” of previous trial results) within the analysis may improve the sensitivity of the results of a series of N-of-1 trials, from both the individual and population level to study the efficacy of TCM syndrome differentiation. It is the exploration of improving the objective evaluation method of the clinical efficacy of TCM and may provide reference value for clinical trials of TCM in other chronic diseases. This trial is registered with ClinicalTrials.gov (ID: NCT04601792)