Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian

Offline reinforcement learning (RL), which refers to decision-making from a previously-collected dataset of interactions, has received significant attention over the past years. Much effort has focused on improving offline RL practicality by addressing the prevalent issue of partial data coverage through various forms of conservative policy learning. While the majority of algorithms do not have finite-sample guarantees, several provable conservative offline RL algorithms are designed and analyzed within the single-policy concentrability framework that handles partial coverage. Yet, in the nonlinear function approximation setting where confidence intervals are difficult to obtain, existing provable algorithms suffer from computational intractability, prohibitively strong assumptions, and suboptimal statistical rates. In this paper, we leverage the marginalized importance sampling (MIS) formulation of RL and present the first set of offline RL algorithms that are statistically optimal and practical under general function approximation and single-policy concentrability, bypassing the need for uncertainty quantification. We identify that the key to successfully solving the sample-based approximation of the MIS problem is ensuring that certain occupancy validity constraints are nearly satisfied. We enforce these constraints by a novel application of the augmented Lagrangian method and prove the following result: with the MIS formulation, augmented Lagrangian is enough for statistically optimal offline RL. In stark contrast to prior algorithms that induce additional conservatism through methods such as behavior regularization, our approach provably eliminates this need and reinterprets regularizers as "enforcers of occupancy validity" than "promoters of conservatism."Comment: 49 pages, 1 figur

Three-Pole Tunable Filters with High Rejection using Mixed Quarter-Lambda and Asymmetric Half-Lambda Resonators

A novel three-pole tunable bandpass filter using varactor-loaded quarter-wavelength combline and asym-metric half-wavelength resonators is proposed in this paper. A nearly constant 3-dB absolute bandwidth is 150 ± 13 MHz (10.7%~6.9% fractional bandwidth) within the tuning range of 1.4-2.0 GHz (42.8%). The filter is designed on a Rogers substrate with relative permittivity 2.2 and h=1 mm with its insertion loss varying from 3.6 dB to 2.8 dB and return loss better than 10 dB over the entire tuning range. The creation of two transmission zeros near the passband edges is analyzed by the even-odd-method. By using dissimilar resonators, the proposed tunable filter could obtain > 33 dB rejection levels at the second harmonics. The measured results show good agreement with the simulated ones

Prognostic value of ALDH1 and Nestin in advanced cancer: a systematic meta-analysis with trial sequential analysis

Background: Novel prognostic markers and therapeutic targets for advanced cancer are urgently needed. This report with trial sequential analysis (TSA) was first conducted to provide robust estimates of the correlation between aldehyde dehydrogenase 1 (ALDH1) and Nestin and clinical outcomes of advanced cancer patients. Methods: Hazard ratios (HRs) with 95% confidence intervals (CIs) were summarized for overall survival (OS), disease-free survival (DFS), progression-free survival (PFS), cancer-specific survival (CSS), relapse/recurrence-free survival (RFS), and metastasis-free survival (MFS) from multivariable analysis. TSA was performed to control for random errors. Results: A total of 20 studies with 2050 patients (ALDH1: 15 studies with 1557 patients and Nestin: 5 studies with 493 patients) were identified. ALDH1 (HR = 2.28, p < 0.001) and Nestin (HR = 2.39, p < 0.001) were associated with a worse OS, as confirmed by TSA. Nestin positivity was linked to a poor PFS (HR = 2.08, p < 0.001), but ALDH1 was not linked to DFS, RFS, MFS, or PFS, and TSA showed that more studies were needed. Subgroup analysis by tumor type indicated that ALDH1 positivity may be associated with shorter OS in breast, head and neck cancers, but there was no association with colorectal cancer. Subgroup analysis by study source showed that ALDH1 positivity was correlated with a worse OS for Japanese (HR = 1.94, p = 0.002) and European patients (HR = 4.15, p < 0.001), but there was no association for Chinese patients. Subgroup analysis by survival rate showed that ALDH1 positivity correlated with poor OS at ⩾ 5 years (HR = 2.33, p < 0.001) or 10 years (HR = 1.76, p = 0.038). Conclusions: ALDH1 may be more valuable as an effective therapeutic target than Nestin for improving the long-term survival rate of advanced cancer. Additional prospective clinical trials are needed across different cancer types

Efficacy and Safety of Hou Gu Mi Xi on Spleen Qi Deficiency in Patients with Nonorganic Gastrointestinal Disorders: Protocol for a Multicenter, Randomized, Placebo-Controlled Trial

Background. There is a worldwide epidemic of nonorganic gastrointestinal disorders (NOGDs), which are a class of disorders that cause various discomforts and ultimately progress into organic gastrointestinal diseases. Because of the unsatisfactory efficacy of Western medical treatments, traditional Chinese medicine (TCM) is becoming a promising complementary and alternative treatment to manage NOGDs. Objectives. To investigate the efficacy and safety of Hou Gu Mi Xi (HGMX), a newly developed dietary TCM formula, on the syndrome of spleen qi deficiency (SQD) in patients with NOGDs. Methods/Design. This study is a multicenter, randomized, double-blinded, parallel, and placebo-controlled trial that will last for 2 years. All qualified subjects with NOGDs and SQD will be included. The study population will be divided into the HGMX and placebo groups. To assess the efficacy of HGMX, we will mainly focus on changes in SQD symptoms scored by a Spleen Qi Deficiency Symptoms Grading and Quantifying Scale and evaluate changes in gastrin-17, the negative Helicobacter pylori conversion rate, body weight, body mass index, and gastroscopy findings. The safety of HGMX will be assessed by recording adverse events (AEs), severe AEs, treatment-related AEs and withdrawal due to AEs. Discussion. This trial is part of our study series that intends to validate the potential of HGMX in the management of chronic gastrointestinal diseases. This series of RCTs is the first committed to the evaluation of a dietary TCM formula and will hopefully establish an evidence-based clinical research model for dietary TCM formulas. Ethics. The protocol was approved by Ethics Committee of five research hospitals and was registered in Clinicaltrials.gov (NCT03019042)