Accelerated Policy Gradient: On the Nesterov Momentum for Reinforcement Learning

Policy gradient methods have recently been shown to enjoy global convergence at a $\Theta(1/t)$ rate in the non-regularized tabular softmax setting. Accordingly, one important research question is whether this convergence rate can be further improved, with only first-order updates. In this paper, we answer the above question from the perspective of momentum by adapting the celebrated Nesterov's accelerated gradient (NAG) method to reinforcement learning (RL), termed \textit{Accelerated Policy Gradient} (APG). To demonstrate the potential of APG in achieving faster global convergence, we formally show that with the true gradient, APG with softmax policy parametrization converges to an optimal policy at a $\tilde{O}(1/t^2)$ rate. To the best of our knowledge, this is the first characterization of the global convergence rate of NAG in the context of RL. Notably, our analysis relies on one interesting finding: Regardless of the initialization, APG could end up reaching a locally nearly-concave regime, where APG could benefit significantly from the momentum, within finite iterations. By means of numerical validation, we confirm that APG exhibits $\tilde{O}(1/t^2)$ rate as well as show that APG could significantly improve the convergence behavior over the standard policy gradient.Comment: 51 pages, 8 figure

Molecular population genetics and gene expression analysis of duplicated CBF genes of Arabidopsis thaliana

<p>Abstract</p> <p>Background</p> <p><it>CBF/DREB </it>duplicate genes are widely distributed in higher plants and encode transcriptional factors, or CBFs, which bind a DNA regulatory element and impart responsiveness to low temperatures and dehydration.</p> <p>Results</p> <p>We explored patterns of genetic variations of <it>CBF1, -2</it>, and -<it>3 </it>from 34 accessions of <it>Arabidopsis thaliana</it>. Molecular population genetic analyses of these genes indicated that <it>CBF2 </it>has much reduced nucleotide diversity in the transcriptional unit and promoter, suggesting that <it>CBF2 </it>has been subjected to a recent adaptive sweep, which agrees with reports of a regulatory protein of <it>CBF2</it>. Investigating the ratios of K_a/K_sbetween all paired <it>CBF </it>paralogus genes, high conservation of the AP2 domain was observed, and the major divergence of proteins was the result of relaxation in two regions within the transcriptional activation domain which was under positive selection after <it>CBF </it>duplication. With respect to the level of <it>CBF </it>gene expression, several mutated nucleotides in the promoters of <it>CBF3 </it>and <it>-1 </it>of specific ecotypes might be responsible for its consistently low expression.</p> <p>Conclusion</p> <p>We concluded from our data that important evolutionary changes in <it>CBF1, -2</it>, and -<it>3 </it>may have primarily occurred at the level of gene regulation as well as in protein function.</p

Depression is a predictor for both smoking and quitting intentions among male coronary artery disease patients

Coronary artery disease (CAD) is the third most prominent cause of death globally, and smoking is the most common risk factors for CAD. However, few studies have examined both smoking and smoking cessation intentions in patients with CAD. The study aims to explore the predictors for smoking and quitting intentions among male CAD patients. This was a cross-sectional study. A total of 368 male CAD patients were recruited and classified into never smoked, quit smoking, and continuing to smoke three groups. Demographic information, level of nicotine dependence, carbon monoxide concentration, depression, and resilience were analyzed by using t-test, one- way analysis of variance (ANOVA), and least significant difference (LSD) post-hoc test and the multiple logistic regression analysis. The results revealed that among participants, 23.1% had never smoked, 40.5% had quit smoking, and 36.4% continued to smoke. Multiple logistic regression analysis revealed that age (OR=0.95, 95% CI=0.90–0.99), carbon monoxide (OR=1.74, 95% CI=1.51–2.01), and depression (OR=1.13, 95% CI=1.04–1.23) predicted participants who continued to smoke. Among the 134 participants who continued to smoke, 35.8% exhibited no intention to quit, and 64.2% planned to quit. Nicotine dependence (OR=0.79, 95% CI=0.66–0.94) and depression (OR=1.10, 95% CI=1.02–1.20) were significant predictors in participants who intended to quit smoking. The study demonstrates that depression is a significant predictor for both smoking and quitting intentions among male CAD patients

Shared decision-making for chronic obstructive pulmonary disease smoking cessation

Chronic obstructive pulmonary disease (COPD) is the main cause of death among people aged 65 years and above. Smoking cessation reduces the risk of morbidity and mortality. This study used the variables of smoking cessation behavior and psychological dependence to evaluate the effectiveness of smoking cessation shared decision-making (SDM) with traditional smoking cessation education in patients with COPD. This randomized controlled trial represents a significant positive correlation was observed among smoking duration (p&lt;.05), the number of cigarettes (p&lt;.05), smoking cessation behavior (p&lt;.05), and psychological cigarette dependence. The intervention group (n=44) underwent session of smoking cessation SDM, whereas the control group (n=44) underwent session of traditional smoking cessation education. After three months of the intervention, significant improvements in psychological cigarette dependence (p&lt;.05) and smoking cessation behavior (p&lt;.05) were observed in both groups. The study confirmed that the success rate of smoking cessation in the intervention group is higher than the control group

MetaSquare: An integrated metadatabase of 16S rRNA gene amplicon for microbiome taxonomic classification

MOTIVATION: Taxonomic classification of 16S ribosomal RNA gene amplicon is an efficient and economic approach in microbiome analysis. 16S rRNA sequence databases like SILVA, RDP, EzBioCloud and HOMD used in downstream bioinformatic pipelines have limitations on either the sequence redundancy or the delay on new sequence recruitment. To improve the 16S rRNA gene-based taxonomic classification, we merged these widely used databases and a collection of novel sequences systemically into an integrated resource. RESULTS: MetaSquare version 1.0 is an integrated 16S rRNA sequence database. It is composed of more than 6 million sequences and improves taxonomic classification resolution on both long-read and short-read methods. AVAILABILITY AND IMPLEMENTATION: Accessible at https://hub.docker.com/r/lsbnb/metasquare_db and https://github.com/lsbnb/MetaSquare. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online