Accelerated Policy Gradient: On the Nesterov Momentum for Reinforcement Learning

Policy gradient methods have recently been shown to enjoy global convergence at a $\Theta(1/t)$ rate in the non-regularized tabular softmax setting. Accordingly, one important research question is whether this convergence rate can be further improved, with only first-order updates. In this paper, we answer the above question from the perspective of momentum by adapting the celebrated Nesterov's accelerated gradient (NAG) method to reinforcement learning (RL), termed \textit{Accelerated Policy Gradient} (APG). To demonstrate the potential of APG in achieving faster global convergence, we formally show that with the true gradient, APG with softmax policy parametrization converges to an optimal policy at a $\tilde{O}(1/t^2)$ rate. To the best of our knowledge, this is the first characterization of the global convergence rate of NAG in the context of RL. Notably, our analysis relies on one interesting finding: Regardless of the initialization, APG could end up reaching a locally nearly-concave regime, where APG could benefit significantly from the momentum, within finite iterations. By means of numerical validation, we confirm that APG exhibits $\tilde{O}(1/t^2)$ rate as well as show that APG could significantly improve the convergence behavior over the standard policy gradient.Comment: 51 pages, 8 figure

Novel Codon-optimization Genes Encoded in Chlorella for Triacylglycerol Accumulation

AbstractMicroalgae have been recognized as one of the potential resources for biodiesel production based on its fast growth or its high total lipid content depending on species. Expression of Kennedy pathway genes, which encodes GPAT, LPAAT, PAP, and DGAT for increasing the metabolic flux towards the TAG storage in Chlorella sp. from 20 to 46 wt% and total lipid accumulation from 35 to 60wt.% corresponding to each specific gene combination under autotrophy, compare to the wild type (vector only). The highest TAG content was found in cells expressing a quadruple-gene construct (GPAT-LPAAT-PAP-DGAT) in the Kennedy pathway, corresponding to 46wt.% of TAG and 60wt.% of total lipid content. This work provides the optimization of TAG production in Chlorella sp. can be achieved by manipulating the selected genes, in turns making commercially producing biodiesel practical

Synaptophysin Expression in Rat Retina Following Acute High Intraocular Pressure

In response to injury, synapse alteration may occur earlier than the changes in the cell body of neurons. Although retinal ganglion cell death and thinning of the inner part of retina were found after acute high intraocular pressure (HIOP), the structural and functional changes of synapses in the retina remain unknown. In the present study, we investigated the protein and mRNA expression of synaptophysin (SYN), an important molecule closely related to synaptic activities, synaptogenesis and synaptic plasticity. In addition, we also studied the ultrastructural changes of the retinal synapses. We found that (1) synaptophysin was upregulated transiently at both protein and mRNA level following HIOP; (2) broadened distribution of synaptophysin protein was present within the outer nuclear layer at the early stage following HIOP; (3) in the outer nuclear layer bouton-like vesicle-containing structures were observed by electron microscopy. This data suggested that, besides degeneration, synapses in rat retina may undergo regenerative events following HIOP

Outage and Capacity Performance Evaluation of Distributed MIMO Systems over a Composite Fading Channel

The exact closed-form expressions regarding the outage probability and capacity of distributed MIMO (DMIMO) systems over a composite fading channel are derived. This is achieved firstly by using a lognormal approximation to a gamma-lognormal distribution when a mobile station (MS) in the cell is in a fixed position, and the so-called maximum ratio transmission/selected combining (MRT-SC) and selected transmission/maximum ratio combining (ST-MRC) schemes are adopted in uplink and downlink, respectively. Then, based on a newly proposed nonuniform MS cell distribution model, which is more consistent with the MS cell hotspot distribution in an actual communication environment, the average outage probability and capacity formulas are further derived. Finally, the accuracy of the approximation method and the rationality of the corresponding theoretical analysis regarding the system performance are proven and illustrated by computer simulations

Lipopolysaccharide-stimulated Leukocytes Contribute to Platelet Aggregative Dysfunction, Which is Attenuated by Catalase in Rats

Endotoxemia causes several hematological dysfunctions, including platelet degranulation or disseminated intravascular coagulation, which lead to thrombotic and hemorrhagic events. Here, we tested the hypothesis that bacterial lipopolysaccharide (LPS)-stimulated leukocytes contribute to platelet aggregative dysfunction, and this function is attenuated by antioxidants. Plateletrich plasma (PRP) was prepared from whole blood of normal and endotoxemic rats. The ability of platelet aggregation was measured by an aggregometer. LPS (50–100 μg/mL) was incubated with PRP, whole blood and PRP with polymorphonuclear leukocytes (PMNs) for 30 minutes, 60 minutes and 90 minutes, and platelet aggregation was detected. LPS-induced platelet aggregative dysfunction was undetectable in intact PRP which was isolated from normal whole blood, whereas it was detected in PRP isolated from endotoxemic rats and LPS-treated whole blood. Moreover, the effect of LPS-induced platelet aggregative dysfunction on intact PRP was observed when the PMNs were added. LPS-induced platelet aggregative dysfunction was significantly attenuated by catalase alone and in combination with NG-nitro-L-arginine methyl ester, but not by NG-nitro-L-arginine methyl ester alone. These results indicate that LPS-stimulated PMNs modulate platelet aggregation during LPS treatment and the effects are reversed by antioxidants. PMNs serve as an approach to understand LPS-induced platelet aggregative dysfunction during endotoxemia. During this process, the generation of reactive oxygen species, hydrogen peroxide especially, from LPS-stimulated PMNs could be an important potential factor in LPS-induced platelet aggregative dysfunction. Catalase contributes to the prevention of platelet dysfunction during LPS-induced sepsis

Potent anti-tumor activity of telomerase-dependent and HSV-TK armed oncolytic adenovirus for non-small cell lung cancer in vitro and in vivo

<p>Abstract</p> <p>Background</p> <p>Non-small cell lung cancer (NSCLC) is the leading cause of cancer related mortality, any improvements in therapeutic strategies are urgently required. In this study we generated a novel 'suicide gene' armed oncolytic adenoviral vector and investigated its antitumor effect both in vitro and in vivo.</p> <p>Methods</p> <p>Since the up-regulated expression of human telomerase reverse transcriptase (hTERT) is a hallmark of alltypes of NSCLC, we chose hTERT promoter to transcriptionally control E1A gene expression to obtain adenoviral replication in NSCLC. In order to further enhance anti-tumor effect of this oncolytic adenoviral vector, we inserted a 'suicide gene' i.e. Herpes Simplex Virus Thymidine Kinase (HSV-TK) into oncolytic adenoviral vector to engineer a novel armed oncolytic adenoviral vector 'Ad.hTERT-E1A-TK'.</p> <p>Results</p> <p>Ad.hTERT-E1A-TK efficiently killed different types of tumor cells including two types of NSCLC cells <it>in vitro</it>, causing no damage to normal primary fibroblasts. Furthermore, Ad.hTERT-E1A-TK infection combined with administration of prodrug gancyclovir (GCV) resulted in more potent cytotoxicity on NSCLC cells, and synergistically suppressed human NSCLC tumor growth in nude mice.</p> <p>Conclusion</p> <p>The results from this study showed that Ad.hTERT-E1A-TK/GCV could be a potent but safe anti-tumor strategy for NSCLC biotherapy.</p