Coordinate Ascent for Off-Policy RL with Global Convergence Guarantees

We revisit the domain of off-policy policy optimization in RL from the perspective of coordinate ascent. One commonly-used approach is to leverage the off-policy policy gradient to optimize a surrogate objective -- the total discounted in expectation return of the target policy with respect to the state distribution of the behavior policy. However, this approach has been shown to suffer from the distribution mismatch issue, and therefore significant efforts are needed for correcting this mismatch either via state distribution correction or a counterfactual method. In this paper, we rethink off-policy learning via Coordinate Ascent Policy Optimization (CAPO), an off-policy actor-critic algorithm that decouples policy improvement from the state distribution of the behavior policy without using the policy gradient. This design obviates the need for distribution correction or importance sampling in the policy improvement step of off-policy policy gradient. We establish the global convergence of CAPO with general coordinate selection and then further quantify the convergence rates of several instances of CAPO with popular coordinate selection rules, including the cyclic and the randomized variants of CAPO. We then extend CAPO to neural policies for a more practical implementation. Through experiments, we demonstrate that CAPO provides a competitive approach to RL in practice.Comment: 47 pages, 4 figure

Comparative study of an externship program versus a corporate-academic cooperation program for enhancing nursing competence of graduating students

BACKGROUND: New graduates report intense stress during the transition from school to their first work settings. Managing this transition is important to reduce turnover rates. This study compared the effects of an externship program and a corporate-academic cooperation program on enhancing junior college students’ nursing competence and retention rates in the first 3 months and 1 year of initial employment. METHODS: This two-phase study adopted a pretest and posttest quasi-experimental design. All participants were graduating students drawn from a 5-year junior nursing college in Taiwan. There were 19 and 24 students who participated in the phase I externship program and phase II corporate-academic cooperation program, respectively. The nursing competence of the students had to be evaluated by mentors within 48 hours of practicum training and after practicum training. The retention rate was also surveyed at 3 months and 1 year after beginning employment. RESULTS: Students who participated in the corporate-academic cooperation program achieved a statistically significant improvement in nursing competence and retention rates relative to those who participated in the externship program (p < 0.01 and p < 0.05, respectively). CONCLUSIONS: The corporate-academic cooperation program facilitates the transition of junior college nursing students into independent staff nurses, enhances their nursing competence, and boosts retention rates

Effect of Rhesus D incompatibility on schizophrenia depends on offspring sex.

Rhesus D incompatibility increases risk for schizophrenia, with some evidence that risk is limited to male offspring. The purpose of this study is to determine whether risk for schizophrenia due to Rhesus D incompatibility differs by offspring sex using a nuclear family-based candidate gene approach and a meta-analysis approach. The genetic study is based on a sample of 277 nuclear families with RHD genotype data on at least one parent and at least one child diagnosed with schizophrenia or related disorder. Meta-analysis inclusion criteria were (1) well-defined sample of schizophrenia patients with majority born before 1970, (2) Rhesus D incompatibility phenotype or genotype data available on mother and offspring, and by offspring sex. Two of ten studies, plus the current genetic study sample, fulfilled these criteria, for a total of 358 affected males and 226 affected females. The genetic study found that schizophrenia risk for incompatible males was significantly greater than for compatible offspring (p=0.03), while risk for incompatible and compatible females was not significantly different (p=.32). Relative risks for incompatible males and females were not significantly different from each other. Meta-analysis using a larger number of affected males and females supports their difference. Taken together, these results provide further support that risk of schizophrenia due to Rhesus D incompatibility is limited to incompatible males, although a weak female incompatibility effect cannot be excluded. Sex differences during fetal neurodevelopment should be investigated to fully elucidate the etiology of schizophrenia

Novel Codon-optimization Genes Encoded in Chlorella for Triacylglycerol Accumulation

AbstractMicroalgae have been recognized as one of the potential resources for biodiesel production based on its fast growth or its high total lipid content depending on species. Expression of Kennedy pathway genes, which encodes GPAT, LPAAT, PAP, and DGAT for increasing the metabolic flux towards the TAG storage in Chlorella sp. from 20 to 46 wt% and total lipid accumulation from 35 to 60wt.% corresponding to each specific gene combination under autotrophy, compare to the wild type (vector only). The highest TAG content was found in cells expressing a quadruple-gene construct (GPAT-LPAAT-PAP-DGAT) in the Kennedy pathway, corresponding to 46wt.% of TAG and 60wt.% of total lipid content. This work provides the optimization of TAG production in Chlorella sp. can be achieved by manipulating the selected genes, in turns making commercially producing biodiesel practical

Transforming growth factor-β1 induces matrix metalloproteinase-9 and cell migration in astrocytes: roles of ROS-dependent ERK- and JNK-NF-κB pathways

<p>Abstract</p> <p>Background</p> <p>Transforming growth factor-β (TGF-β) and matrix metalloproteinases (MMPs) are the multifunctional factors during diverse physiological and pathological processes including development, wound healing, proliferation, and cancer metastasis. Both TGF-β and MMPs have been shown to play crucial roles in brain pathological changes. Thus, we investigated the molecular mechanisms underlying TGF-β1-induced MMP-9 expression in brain astrocytes.</p> <p>Methods</p> <p>Rat brain astrocytes (RBA-1) were used. MMP-9 expression was analyzed by gelatin zymography and RT-PCR. The involvement of signaling molecules including MAPKs and NF-κB in the responses was investigated using pharmacological inhibitors and dominant negative mutants, determined by western blot and gene promoter assay. The functional activity of MMP-9 was evaluated by cell migration assay.</p> <p>Results</p> <p>Here we report that TGF-β1 induces MMP-9 expression and enzymatic activity via a TGF-β receptor-activated reactive oxygen species (ROS)-dependent signaling pathway. ROS production leads to activation of extracellular signal-regulated kinase 1/2 (ERK1/2) and c-Jun-N-terminal kinase (JNK) and then activation of the NF-κB transcription factor. Activated NF-κB turns on transcription of the MMP-9 gene. The rat MMP-9 promoter, containing a NF-κB <it>cis</it>-binding site, was identified as a crucial domain linking to TGF-β1 action.</p> <p>Conclusions</p> <p>Collectively, in RBA-1 cells, activation of ERK1/2- and JNK-NF-κB cascades by a ROS-dependent manner is essential for MMP-9 up-regulation/activation and cell migration induced by TGF-β1. These findings indicate a new regulatory pathway of TGF-β1 in regulating expression of MMP-9 in brain astrocytes, which is involved in physiological and pathological tissue remodeling of central nervous system.</p

Using the maternal-fetal genotype incompatibility test to assess non-inherited maternal HLA-DRB1 antigen coding alleles as rheumatoid arthritis risk factors

Non-inherited maternal antigens encoded by specific HLA-DRB1 alleles (NIMA) have been implicated as a rheumatoid arthritis (RA) risk factor. Using genotype data from North American Rheumatoid Arthritis Consortium study participants and the maternal-fetal genotype incompatibility (MFG) test, we find evidence for offspring allelic effects but no evidence for NIMA as a RA risk factor. We discuss possible reasons why our result conflicts with several previous studies (including one of our own) that used RA patients from northern Europe

Gradient static-strain stimulation in a microfluidic chip for 3D cellular alignment

This is the published version. Copyright 2014 Royal Society of ChemistryCell alignment is a critical factor to govern cellular behavior and function for various tissue engineering applications ranging from cardiac to neural regeneration. In addition to physical geometry, strain is a crucial parameter to manipulate cellular alignment for functional tissue formation. In this paper, we introduce a simple approach to generate a range of gradient static strains without external mechanical control for the stimulation of cellular behavior within 3D biomimetic hydrogel microenvironments. A glass-supported microfluidic chip with a convex flexible polydimethylsiloxane (PDMS) membrane on the top was employed for loading the cells suspended in a prepolymer solution. Following UV crosslinking through a photomask with a concentric circular pattern, the cell-laden hydrogels were formed in a height gradient from the center (maximum) to the boundary (minimum). When the convex PDMS membrane retracted back to a flat surface, it applied compressive gradient forces on the cell-laden hydrogels. The concentric circular hydrogel patterns confined the direction of hydrogel elongation, and the compressive strain on the hydrogel therefore resulted in elongation stretch in the radial direction to guide cell alignment. NIH3T3 cells were cultured in the chip for 3 days with compressive strains that varied from ~65% (center) to ~15% (boundary) on hydrogels. We found that the hydrogel geometry dominated the cell alignment near the outside boundary, where cells aligned along the circular direction, and the compressive strain dominated the cell alignment near the center, where cells aligned radially. This study developed a new and simple approach to facilitate cellular alignment based on hydrogel geometry and strain stimulation for tissue engineering applications. This platform offers unique advantages and is significantly different from the existing approaches owing to the fact that gradient generation was accomplished in a miniature device without using an external mechanical source