Guarded Policy Optimization with Imperfect Online Demonstrations

The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.Comment: Accepted at ICLR 2023 (top 25%

MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning

Driving safely requires multiple capabilities from human and intelligent agents, such as the generalizability to unseen environments, the safety awareness of the surrounding traffic, and the decision-making in complex multi-agent settings. Despite the great success of Reinforcement Learning (RL), most of the RL research works investigate each capability separately due to the lack of integrated environments. In this work, we develop a new driving simulation platform called MetaDrive to support the research of generalizable reinforcement learning algorithms for machine autonomy. MetaDrive is highly compositional, which can generate an infinite number of diverse driving scenarios from both the procedural generation and the real data importing. Based on MetaDrive, we construct a variety of RL tasks and baselines in both single-agent and multi-agent settings, including benchmarking generalizability across unseen scenes, safe exploration, and learning multi-agent traffic. The generalization experiments conducted on both procedurally generated scenarios and real-world scenarios show that increasing the diversity and the size of the training set leads to the improvement of the generalizability of the RL agents. We further evaluate various safe reinforcement learning and multi-agent reinforcement learning algorithms in MetaDrive environments and provide the benchmarks. Source code, documentation, and demo video are available at https://metadriverse.github.io/metadrive . More research projects based on MetaDrive simulator are listed at https://metadriverse.github.ioComment: Source code, documentation, and demo video are available at https://metadriverse.github.io/metadrive . More research projects based on MetaDrive simulator are listed at https://metadriverse.github.i

Cloning and Functional Analysis of the MADS-box CiMADS9 Gene from Carya illinoinensis

AbstractA MADS-box gene, CiMADS9, was cloned from the male flowers of Carya illinoinensis by rapid amplification of cDNA ends. The gene was 1 077bp with a 768bp open reading frame encoding 255 amino acids. Multiple sequence comparisons revealed that CiMADS9 is a typical MIKC-type MADS-box gene with a MADS-box domain and a K semi-conserved region. Phylogenetic analysis indicated that CiMADS9 belongs to the AGL15 group of the MADS-box gene family. Quantitative reverse transcription polymerase chain reaction analysis indicated that the expression levels in reproductive organs (i.e., flowers and young fruits) were considerably higher than in vegetative tissues (i.e., leaves and branches). The highest expression levels were observed in male flowers. An overexpression vector for CiMADS9 was constructed and the gene was inserted into the Arabidopsis thaliana genome. CiMADS9 expression was confirmed in all transgenic lines. Compared with wild-type plants, transgenic A. thaliana plants overexpressing CiMADS9 exhibited delayed flowering and an increased number of leaves

Transcriptome analysis reveals salt-stress-regulated biological processes and key pathways in roots of cotton (Gossypium hirsutum L.)

AbstractHigh salinity is one of the main factors limiting cotton growth and productivity. The genes that regulate salt stress in TM-1 upland cotton were monitored using microarray and real-time PCR (RT-PCR) with samples taken from roots. Microarray analysis showed that 1503 probe sets were up-regulated and 1490 probe sets were down-regulated in plants exposed for 3h to 100mM NaCl, and RT-PCR analysis validated 42 relevant/related genes. The distribution of enriched gene ontology terms showed such important processes as the response to water stress and pathways of hormone metabolism and signal transduction were induced by the NaCl treatment. Some key regulatory gene families involved in abiotic and biotic sources of stress such as WRKY, ERF, and JAZ were differentially expressed. Our transcriptome analysis might provide some useful insights into salt-mediated signal transduction pathways in cotton and offer a number of candidate genes as potential markers of tolerance to salt stress

Shape Determination for Deformed Electromagnetic Cavities

The measured physical parameters of a superconducting cavity differ from those of the designed ideal cavity. This is due to shape deviations caused by both loose machine tolerances during fabrication and by the tuning process for the accelerating mode. We present a shape determination algorithm to solve for the unknown deviations from the ideal cavity using experimentally measured cavity data. The objective is to match the results of the deformed cavity model to experimental data through least-squares minimization. The inversion variables are unknown shape deformation parameters that describe perturbations of the ideal cavity. The constraint is the Maxwell eigenvalue problem. We solve the nonlinear optimization problem using a line-search based reduced space Gauss-Newton method where we compute shape sensitivities with a discrete adjoint approach. We present two shape determination examples, one from synthetic and the other from experimental data. The results demonstrate that the proposed algorithm is very effective in determining the deformed cavity shape

Disrupted Asymmetry of Inter- and Intra-Hemispheric Functional Connectivity at Rest in Medication-Free Obsessive-Compulsive Disorder

Disrupted functional asymmetry of cerebral hemispheres may be altered in patients with obsessive-compulsive disorder (OCD). However, little is known about whether anomalous brain asymmetries originate from inter- and/or intra-hemispheric functional connectivity (FC) at rest in OCD. In this study, resting-state functional magnetic resonance imaging was applied to 40 medication-free patients with OCD and 38 gender-, age-, and education-matched healthy controls (HCs). Data were analyzed using the parameter of asymmetry (PAS) and support vector machine methods. Patients with OCD showed significantly increased PAS in the left posterior cingulate cortex, left precentral gyrus/postcentral gyrus, and right inferior occipital gyrus and decreased PAS in the left dorsolateral prefrontal cortex (DLPFC), bilateral middle cingulate cortex (MCC), left inferior parietal lobule, and left cerebellum Crus I. A negative correlation was found between decreased PAS in the left DLPFC and Yale–Brown Obsessive-compulsive Scale compulsive behavior scores in the patients. Furthermore, decreased PAS in the bilateral MCC could be used to distinguish OCD from HCs with a sensitivity of 87.50%, an accuracy of 88.46%, and a specificity of 89.47%. These results highlighted the contribution of disrupted asymmetry of intra-hemispheric FC within and outside the cortico-striato-thalamocortical circuits at rest in the pathophysiology of OCD, and reduced intra-hemispheric FC in the bilateral MCC may serve as a potential biomarker to classify individuals with OCD from HCs

A Proposed High-Power UV Industrial Demonstration Laser at CEBAF

The Laser Processing Consortium, a collaboration of industries, universities, and the Continuous Electron Beam Accelerator Facility (CEBAF) in Newport News, Virginia, has proposed building a demonstration industrial processing laser for surface treatment and micro-machining. The laser is a free-electron laser (FEL) with average power output exceeding 1 kW in the ultraviolet (UV). The design calls for a novel driver accelerator that recovers most of the energy of the exhaust electron beam to produce laser light with good wall-plug efficiency. The laser and accelerator design use technologies that are scalable to much higher power. The authors will describe the critical design issues in the laser such as the stability, power handling, and losses of the optical resonator, and the quality, power, and reliability of the electron beam. They will also describe the calculated laser performance. Finally progress to date on accelerator development and resonator modeling will be reported