An immediate-return reinforcement learning for the atypical Markov decision processes

The atypical Markov decision processes (MDPs) are decision-making for maximizing the immediate returns in only one state transition. Many complex dynamic problems can be regarded as the atypical MDPs, e.g., football trajectory control, approximations of the compound Poincaré maps, and parameter identification. However, existing deep reinforcement learning (RL) algorithms are designed to maximize long-term returns, causing a waste of computing resources when applied in the atypical MDPs. These existing algorithms are also limited by the estimation error of the value function, leading to a poor policy. To solve such limitations, this paper proposes an immediate-return algorithm for the atypical MDPs with continuous action space by designing an unbiased and low variance target Q-value and a simplified network framework. Then, two examples of atypical MDPs considering the uncertainty are presented to illustrate the performance of the proposed algorithm, i.e., passing the football to a moving player and chipping the football over the human wall. Compared with the existing deep RL algorithms, such as deep deterministic policy gradient and proximal policy optimization, the proposed algorithm shows significant advantages in learning efficiency, the effective rate of control, and computing resource usage

Design and operation experience of zero-carbon campus

Shandong Normal University - Lishan College is a zero-carbon campus, using the technical scheme “multiple sources of energy complementing each other, 100% utilization of renewable energy, combining concentrated demonstration with popular extension”. It has built 5 distributed energy stations, including roof PV power station, solar heating water system, biomass vacuum hot water unit, natural gas (straw pyrolysis gas, biomethane) CCHP demonstration project etc. Renewable energy can provide power, heating, air conditioning and hot water for 10,000 teachers and students. The zero-carbon campus save 3,650 tce/a, and the CO2 emission reduction is 9,490 t/a. It offers great experience and is a model for the regional clean and low carbon energy usage and the realization of sustainable development