A Graph-Based Reinforcement Learning Method with Converged State Exploration and Exploitation

In any classical value-based reinforcement learning method, an agent, despite of its continuous interactions with the environment, is yet unable to quickly generate a complete and independent description of the entire environment, leaving the learning method to struggle with a difficult dilemma of choosing between the two tasks, namely exploration and exploitation. This problem becomes more pronounced when the agent has to deal with a dynamic environment, of which the configuration and/or parameters are constantly changing. In this paper, this problem is approached by first mapping a reinforcement learning scheme to a directed graph, and the set that contains all the states already explored shall continue to be exploited in the context of such a graph. We have proved that the two tasks of exploration and exploitation eventually converge in the decision-making process, and thus, there is no need to face the exploration vs. exploitation tradeoff as all the existing reinforcement learning methods do. Rather this observation indicates that a reinforcement learning scheme is essentially the same as searching for the shortest path in a dynamic environment, which is readily tackled by a modified Floyd-Warshall algorithm as proposed in the paper. The experimental results have confirmed that the proposed graph-based reinforcement learning algorithm has significantly higher performance than both standard Q-learning algorithm and improved Q-learning algorithm in solving mazes, rendering it an algorithm of choice in applications involving dynamic environments

Analytical Models for Distribution of the Envelope and Phase of Linearly Modulated Signals in AWGN Channel

In this paper, analytical expressions for the distribution of the envelope and phase of linearly modulated signals such as BPSK, M-PSK, and M-QAM in AWGN are presented. We perform numerical simulations for different orders of signal constellations. The results show that the proposed theoretical models are in excellent agreement with the estimated distributions from various numerical experiments

Evolved Massive Stars at Low-metallicity V. Mass-Loss Rate of Red Supergiant Stars in the Small Magellanic Cloud

We assemble the most complete and clean red supergiant (RSG) sample (2,121 targets) so far in the Small Magellanic Cloud (SMC) with 53 different bands of data to study the MLR of RSGs. In order to match the observed spectral energy distributions (SEDs), a theoretical grid of 17,820 Oxygen-rich models (``normal'' and ``dusty'' grids are half-and-half) is created by the radiatively-driven wind model of the DUSTY code, covering a wide range of dust parameters. We select the best model for each target by calculating the minimal modified chi-square and visual inspection. The resulting MLRs from DUSTY are converted to real MLRs based on the scaling relation, for which a total MLR of $6.16\times10^{-3}$ $M_\odot$ yr$^{-1}$ is measured (corresponding to a dust-production rate of $\sim6\times10^{-6}$ $M_\odot$ yr$^{-1}$), with a typical MLR of $\sim10^{-6}$ $M_\odot$ yr$^{-1}$ for the general population of the RSGs. The complexity of mass-loss estimation based on the SED is fully discussed for the first time, indicating large uncertainties based on the photometric data (potentially up to one order of magnitude or more). The Hertzsprung-Russell and luminosity versus median absolute deviation diagrams of the sample indicate the positive relation between luminosity and MLR. Meanwhile, the luminosity versus MLR diagrams show a ``knee-like'' shape with enhanced mass-loss occurring above $\log_{10}(L/L_\odot)\approx4.6$, which may be due to the degeneracy of luminosity, pulsation, low surface gravity, convection, and other factors. We derive our MLR relation by using a third-order polynomial to fit the sample and compare our result with previous empirical MLR prescriptions. Given that our MLR prescription is based on a much larger sample than previous determinations, it provides a more accurate relation at the cool and luminous region of the H-R diagram at low-metallicity compared to previous studies.Comment: 16 pages, 19 figures, accepted by A&

Ciliary parathyroid hormone signaling activates transforming growth factor-β to maintain intervertebral disc homeostasis during aging

© 2018 The Author(s). Degenerative disc disease (DDD) is associated with intervertebral disc degeneration of spinal instability. Here, we report that the cilia of nucleus pulposus (NP) cells mediate mechanotransduction to maintain anabolic activity in the discs. We found that mechanical stress promotes transport of parathyroid hormone 1 receptor (PTH1R) to the cilia and enhances parathyroid hormone (PTH) signaling in NP cells. PTH induces transcription of integrin αvβ6 to activate the transforming growth factor (TGF)-β-connective tissue growth factor (CCN2)-matrix proteins signaling cascade. Intermittent injection of PTH (iPTH) effectively attenuates disc degeneration of aged mice by direct signaling through NP cells, specifically improving intervertebral disc height and volume by increasing levels of TGF-β activity, CCN2, and aggrecan. PTH1R is expressed in both mouse and human NP cells. Importantly, knockout PTH1R or cilia in the NP cells results in significant disc degeneration and blunts the effect of PTH on attenuation of aged discs. Thus, mechanical stress-induced transport of PTH1R to the cilia enhances PTH signaling, which helps maintain intervertebral disc homeostasis, particularly during aging, indicating therapeutic potential of iPTH for DDD

Industrial Relations Experiments in China: Balancing Equity and Efficiency the Chinese Way

China should build socialism by "crossing the river by feeling for stones" (Deng Xiaoping). Chinese industrial relations are changing accordingly. Local union experiments have implemented local-level changes experimenting with institutional reforms that address efficiency and equity imbalances. Local union leaders have exercised autonomy to develop multi-employer “community unions” in Changchun’s Chaoyang District to represent peasant migrant workers employed by small firms by targeting small geographic zones and hiring union presidents as organizers, aggregating union members into amalgamated units. While the union’s role still includes social harmonization, unions have undertaken an additional representative role. Similar efforts elsewhere have given the union representation experience. Unions have organized multi-employer federations across industries. Unions also have collaborated with local governments on innovative structures to ensure that companies in some industries, such as construction, post a “bond” to guarantee end-of-year compensation. Finally, this paper discusses the role of the new Labor Contract Law in institutionalizing these changes. The LCL defines more precisely employment relationships and workers’ legal rights and seems to increase unions’ legal authority to ensure that employers respect individual workers’ rights, supports the extension of collective contracts to more enterprises, and appears to give unions greater authority to represent workers within the employment relationship and before legal authorities. These changes may provide a material basis for balancing efficiency with equity. We think these experiments have political foundations, whether it is “harmonious society” or simply to extend the union’s organizing maintain political status. Further research will determine whether these experiments are successful

HKUST-1/ZIF-67 Nanocomposites as Heterogeneous Cu–Co-Bimetallic Fenton-like Catalysts for Efficient Removal of Methylene Blue

Fenton-like reactions with Fe-based metal–organic framework (MOF) catalysts have been extensively explored in the field of environmental remediation. However, easy precipitation of Fe2+/Fe3+ and the production of sludge under basic conditions caused catalyst loss and greatly limited their large-scale application in industry. The development of an Fe-free Fenton-like reaction is of extreme importance and remains in its infant stage. Herein, a series of Fe-free dual MOF nanoparticles (HKUST-1/ZIF-67-X) were fabricated by in situ coating of ZIF-67 on HKUST-1 and were systematically analyzed by various characterization techniques, such as scanning electron microscopy (SEM), transmission electron microscopy (TEM), X-ray diffraction (XRD), Fourier transform infrared (FT-IR), X-ray photoelectron spectroscopy (XPS), and N2 adsorption–desorption isotherms. Subsequently, these materials were applied in catalyzing methylene blue (MB) degradation. The effects of several operation parameters, i.e., pH, H2O2 dosage, catalyst dosage, and reaction temperature, on MB degradation were investigated. It was unveiled that HKUST-1/ZIF-67-7% exhibited an outstanding catalytic activity without the production of any sludge, which could reach as high as 93.29% MB degradation efficiency within 40 min. This was attributed to the unique core–satellite histoarchitecture of HKUST-1/ZIF-67-7% and the synergistic effect between HKUST-1 and ZIF-67. The HKUST-1/ZIF-67-7% composite still achieved up to 80.17% MB degradation efficiency at the fifth catalysis cycle. Importantly, HKUST-1/ZIF-67-7% exhibited significant catalytic efficiency under a wide pH range (4.2–10.1) and top catalytic efficiency at the near neutral pH value. The low cost, environment benignancy, satisfactory degradation efficiency, wide pH application range, and excellent reusability emphasize its great application potential in Fenton-like degradation of pollutants. This contribution could provide a paradigm investigation for designing non-iron-based MOF catalysts to solve the increasingly pressing pollution issues