Accelerated Policy Gradient: On the Nesterov Momentum for Reinforcement Learning

Policy gradient methods have recently been shown to enjoy global convergence at a $\Theta(1/t)$ rate in the non-regularized tabular softmax setting. Accordingly, one important research question is whether this convergence rate can be further improved, with only first-order updates. In this paper, we answer the above question from the perspective of momentum by adapting the celebrated Nesterov's accelerated gradient (NAG) method to reinforcement learning (RL), termed \textit{Accelerated Policy Gradient} (APG). To demonstrate the potential of APG in achieving faster global convergence, we formally show that with the true gradient, APG with softmax policy parametrization converges to an optimal policy at a $\tilde{O}(1/t^2)$ rate. To the best of our knowledge, this is the first characterization of the global convergence rate of NAG in the context of RL. Notably, our analysis relies on one interesting finding: Regardless of the initialization, APG could end up reaching a locally nearly-concave regime, where APG could benefit significantly from the momentum, within finite iterations. By means of numerical validation, we confirm that APG exhibits $\tilde{O}(1/t^2)$ rate as well as show that APG could significantly improve the convergence behavior over the standard policy gradient.Comment: 51 pages, 8 figure

Full HD JPEG XR Encoder Design for Digital Photography Applications

Non

Development of Building Integrated Photovoltaic (BIPV) System with PV Ceramic Tile and its Application for Building Façade

AbstractIn this paper, the single-crystal silicon-based solar cells laminated between tempered glass and ceramic tile is developed to be utilized in the building's façade. Firstly, the electrical, optical, and thermal properties of the proposed PV module are evaluated. Then, the wind-resistance test is implanted to evaluate the installation feasibility in Taiwan where have typhoon frequently. The electrical and deflection characteristics of the proposed PV module have no obviously changed after a 50 thermal cycling test and a 200hour humidity-freeze test, based on IEC 61215, and a wind-resistance test, respectively. Finally, electrical power generation of the proposed BIPV system with 1 kWp electrical power capacity installed in a demonstration house is performed. The experimental results indicate that accumulative power generation is 185 kWh during 6-month monitoring period. And that, the exterior temperature of the demonstration house is lower than that on the surface of the BIPV system about 10°C. The proposed BIPV system not only provides the passive energy for its power loading, but also improves the indoor thermal environment by fluent natural ventilation

MetaSquare: An integrated metadatabase of 16S rRNA gene amplicon for microbiome taxonomic classification

MOTIVATION: Taxonomic classification of 16S ribosomal RNA gene amplicon is an efficient and economic approach in microbiome analysis. 16S rRNA sequence databases like SILVA, RDP, EzBioCloud and HOMD used in downstream bioinformatic pipelines have limitations on either the sequence redundancy or the delay on new sequence recruitment. To improve the 16S rRNA gene-based taxonomic classification, we merged these widely used databases and a collection of novel sequences systemically into an integrated resource. RESULTS: MetaSquare version 1.0 is an integrated 16S rRNA sequence database. It is composed of more than 6 million sequences and improves taxonomic classification resolution on both long-read and short-read methods. AVAILABILITY AND IMPLEMENTATION: Accessible at https://hub.docker.com/r/lsbnb/metasquare_db and https://github.com/lsbnb/MetaSquare. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online

THE EFFECT OF COUNTER MOVEMENT JUMP PERFORMANCE IN MIDDLEAGED ELDERLY PRACTICING TAI-CHI EXERCISE

The purpose of this study was to investigate biomechanical effects of Tal-Chi exercise on the lower-extremity in middle-aged elders during counter-movement jump. Twelve middle-aged elders with regular Tai Chi exercise experience and twelve healthy middle-aged elders participated in this study. Ten Vicon Motion System cameras, two Kistler force plates were used simultaneously to capture the kinematic and dynamlc parameters of standing vertical jumps. Independent samples &test was performed for statistical analysis ( u = .05 ). Since the jump height of Tai Chi group was significantly higher ( p c .05 ). It showed that practicing Tai Chi exercise could effectively slow down the degeneration of the moment and power at the hip Joint. Therefore, middle-aged elders were recommended to engage in long-term Tai Chi exercise

Efficiency of gratings for silica fiber-coupled internal Smith-Purcell radiation and Cherenkov diffraction radiation -- a quantitative numerical study

We propose a setup for measuring visible and near-visible internal Smith-Purcell radiation and Cherenkov Diffraction Radiation, based on silica and silicon, and perform quantitative numerical analysis of its radiation efficiency. We calculate the total radiated energy per electron and the spectral distribution of different radiation orders, taking into account material dispersion and absorption. For an optimized silica grating of 200 micrometer length, the total radiated energy reaches 2~eV per electron for 2 MeV electrons. Above the Cherenkov threshold, in most cases the energy of Cherenkov diffraction radiation is several times higher than the energy of internal Smith-Purcell radiation, but for some geometries and frequency ranges first or second order radiation may dominate over Cherenkov radiation (zeroth order). Radiation up to the 4th order is detected in the simulation. The spectrum of Cherenkov radiation is highly resonant, with local minima for frequencies where maxima of the other radiation orders occur. The spectrum of Cherenkov radiation takes a frequency comb-like shape for a uniform layer of Si on SiO2 substrate, due to a Fabry-Perot effect occurring for the evanescent field of the moving electron. The proposed setup could become a prototype of a non-invasive particle beam monitor for both conventional and laser particle accelerators

Decreased Risk of Osteoporosis Incident in Subjects Receiving Chinese Herbal Medicine for Sjögren Syndrome Treatment: A Retrospective Cohort Study with a Nested Case-Control Analysis

Sjögren syndrome (SS) is a long-lasting inflammatory autoimmune disease that may cause diverse manifestations, particularly osteoporosis. Though usage of Chinese herbal medicine (CHM) can safely manage autoimmune disease and treatment-related symptoms, the relation between CHM use and osteoporosis risk in SS persons is not yet recognized. With that in mind, this population-level nested case-control study aimed to compare the risk of osteoporosis with and without CHM use. Potential subjects aged 20–70 years, diagnosed with SS between 2001 and 2010, were retrieved from a national health claims database. Those diagnosed with osteoporosis after SS were identified and randomly matched to those without osteoporosis. We capitalize on the conditional logistic regression to estimate osteoporosis risk following CHM use. A total of 1240 osteoporosis cases were detected and randomly matched to 1240 controls at a ratio of 1:1. Those receiving conventional care plus CHM had a substantially lower chance of osteoporosis than those without CHM. Prolonged use of CHM, especially for one year or more, markedly dwindled sequent osteoporosis risk by 71%. Integrating CHM into standard care may favor the improvement of bone function, but further well-designed randomized controlled trials to investigate the possible mechanism are needed