Accelerated Policy Gradient: On the Nesterov Momentum for Reinforcement Learning

Policy gradient methods have recently been shown to enjoy global convergence at a $\Theta(1/t)$ rate in the non-regularized tabular softmax setting. Accordingly, one important research question is whether this convergence rate can be further improved, with only first-order updates. In this paper, we answer the above question from the perspective of momentum by adapting the celebrated Nesterov's accelerated gradient (NAG) method to reinforcement learning (RL), termed \textit{Accelerated Policy Gradient} (APG). To demonstrate the potential of APG in achieving faster global convergence, we formally show that with the true gradient, APG with softmax policy parametrization converges to an optimal policy at a $\tilde{O}(1/t^2)$ rate. To the best of our knowledge, this is the first characterization of the global convergence rate of NAG in the context of RL. Notably, our analysis relies on one interesting finding: Regardless of the initialization, APG could end up reaching a locally nearly-concave regime, where APG could benefit significantly from the momentum, within finite iterations. By means of numerical validation, we confirm that APG exhibits $\tilde{O}(1/t^2)$ rate as well as show that APG could significantly improve the convergence behavior over the standard policy gradient.Comment: 51 pages, 8 figure

Molecular population genetics and gene expression analysis of duplicated CBF genes of Arabidopsis thaliana

<p>Abstract</p> <p>Background</p> <p><it>CBF/DREB </it>duplicate genes are widely distributed in higher plants and encode transcriptional factors, or CBFs, which bind a DNA regulatory element and impart responsiveness to low temperatures and dehydration.</p> <p>Results</p> <p>We explored patterns of genetic variations of <it>CBF1, -2</it>, and -<it>3 </it>from 34 accessions of <it>Arabidopsis thaliana</it>. Molecular population genetic analyses of these genes indicated that <it>CBF2 </it>has much reduced nucleotide diversity in the transcriptional unit and promoter, suggesting that <it>CBF2 </it>has been subjected to a recent adaptive sweep, which agrees with reports of a regulatory protein of <it>CBF2</it>. Investigating the ratios of K_a/K_sbetween all paired <it>CBF </it>paralogus genes, high conservation of the AP2 domain was observed, and the major divergence of proteins was the result of relaxation in two regions within the transcriptional activation domain which was under positive selection after <it>CBF </it>duplication. With respect to the level of <it>CBF </it>gene expression, several mutated nucleotides in the promoters of <it>CBF3 </it>and <it>-1 </it>of specific ecotypes might be responsible for its consistently low expression.</p> <p>Conclusion</p> <p>We concluded from our data that important evolutionary changes in <it>CBF1, -2</it>, and -<it>3 </it>may have primarily occurred at the level of gene regulation as well as in protein function.</p

MetaSquare: An integrated metadatabase of 16S rRNA gene amplicon for microbiome taxonomic classification

MOTIVATION: Taxonomic classification of 16S ribosomal RNA gene amplicon is an efficient and economic approach in microbiome analysis. 16S rRNA sequence databases like SILVA, RDP, EzBioCloud and HOMD used in downstream bioinformatic pipelines have limitations on either the sequence redundancy or the delay on new sequence recruitment. To improve the 16S rRNA gene-based taxonomic classification, we merged these widely used databases and a collection of novel sequences systemically into an integrated resource. RESULTS: MetaSquare version 1.0 is an integrated 16S rRNA sequence database. It is composed of more than 6 million sequences and improves taxonomic classification resolution on both long-read and short-read methods. AVAILABILITY AND IMPLEMENTATION: Accessible at https://hub.docker.com/r/lsbnb/metasquare_db and https://github.com/lsbnb/MetaSquare. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online

The risk of false inclusion of a relative in parentage testing – an in silico population study

Aim To investigate the potential of false inclusion of a close genetic relative in paternity testing by using computer generated families. Methods 10 000 computer-simulated families over three generations were generated based on genotypes using 15 short tandem repeat loci. These data were used in assessing the probability of inclusion or exclusion of paternity when the father is actually a sibling, grandparent, uncle, half sibling, cousin, or a random male. Further, we considered a duo case where the mother’s DNA type was not available and a trio case including the mother’s profile. Results The data showed that the duo scenario had the highest and lowest false inclusion rates when considering a sibling (19.03 ± 0.77%) and a cousin (0.51 ± 0.14%) as the father, respectively; and the rate when considering a random male was much lower (0.04 ± 0.04%). The situation altered slightly with a trio case where the highest rate (0.56 ± 0.15%) occurred when a paternal uncle was considered as the father, and the lowest rate (0.03 ± 0.03%) occurred when a cousin was considered as the father. We also report on the distribution of the numbers for non-conformity (non-matching loci) where the father is a close genetic relative. Conclusions The results highlight the risk of false inclusion in parentage testing. These data provide a valuable reference when incorporating either a mutation in the father’s DNA type or if a close relative is included as being the father; particularly when there are varying numbers of non-matching loci

THE INFLUENCE OF ADJUSTABLE PUTTER HEAD WEIGHTING ON THE STROKE

The purpose of this study was to investigate the effects of putter head weighting towards the heel and the toe on the kinematic aspects of the putting stroke. Seven (n=7) male golfers (age 42.6 ±2.3 y) with high proficiency (handicap 9.5 ±1.4) were recruited for this study. The experiment was carried out in an indoor studio with artificial grass (Stimp 10). Two toe weight and two heel weight settings were tested and compared with the standard weighting. Results suggest that putter head weighing influences the characteristics of the putting stroke, and systematic differences were found between toe and heel weighting. It is concluded that fitting the weight and the balance of a putter head is critical for supporting each individual's stroke and putting performance