A Graph-Based Reinforcement Learning Method with Converged State Exploration and Exploitation

In any classical value-based reinforcement learning method, an agent, despite of its continuous interactions with the environment, is yet unable to quickly generate a complete and independent description of the entire environment, leaving the learning method to struggle with a difficult dilemma of choosing between the two tasks, namely exploration and exploitation. This problem becomes more pronounced when the agent has to deal with a dynamic environment, of which the configuration and/or parameters are constantly changing. In this paper, this problem is approached by first mapping a reinforcement learning scheme to a directed graph, and the set that contains all the states already explored shall continue to be exploited in the context of such a graph. We have proved that the two tasks of exploration and exploitation eventually converge in the decision-making process, and thus, there is no need to face the exploration vs. exploitation tradeoff as all the existing reinforcement learning methods do. Rather this observation indicates that a reinforcement learning scheme is essentially the same as searching for the shortest path in a dynamic environment, which is readily tackled by a modified Floyd-Warshall algorithm as proposed in the paper. The experimental results have confirmed that the proposed graph-based reinforcement learning algorithm has significantly higher performance than both standard Q-learning algorithm and improved Q-learning algorithm in solving mazes, rendering it an algorithm of choice in applications involving dynamic environments

Spatiotemporal Variation of nutrient concentrations in the Upper Shule River Basin, the Qinghai-Tibetan Plateau, China

This study aimed to understand the spatiotemporal variation of nutrient concentrations in stream water, using analyses of total nitrogen (TN), NO3−-N, NH4+-N and total phosphorus (TP) concentrations from April to October in 2016 in the upper Shule River basin in the northeastern margin of the Qinghai-Tibetan Plateau. Dissolved inorganic nitrogen (DIN; NO3−-N + NH4+-N) concentrations in both groundwater and stream water were the predominant form of TN, accounting for about 88%. Median concentrations in stream water were 815 μg L−1 for TN, 459 μg L−1 for NO3−-N, 253 μg L−1 for NH4+-N, and 25 μg L−1 for TP. Great spatiotemporal variations of nitrogen and phosphorus concentrations in stream water were observed, which likely resulted from varying recharge sources (snow and groundwater) and biogeochemical processes. Monthly variations in nutrient concentrations in stream water had a reciprocal pattern between nitrogen and phosphorus concentrations. Our results demonstrated that the quality of steam water in the upper reach of Shule River basin was basically good for drinking purposes according to the environmental quality standards for surface water, China. The results also improved our understanding of how nutrient concentrations varied in stream water and provided baseline information for future studies on nutrients in the Qinghai-Tibetan Plateau

Identification of the Factors Influencing the Baseflow in the Permafrost Region of the Northeastern Qinghai-Tibet Plateau

Baseflow is an essential component of river runoff. Accurate measurements and analyses of baseflow change are challenging in permafrost-covered regions. In this paper, the upper reaches of the Shule River were selected as the study area, in which to study the baseflow change regulation and causes. The variable infiltration capacity (VIC) model, based on the ARNO baseflow formulation, was used to simulate the baseflow. Simulated baseflow was validated by the isotopic baseflow separation results and measured runoff in the recession periods throughout an entire year. It was found that approximately 63.1% of the river runoff was sourced by baseflow in the study region; the baseflow change was relatively smooth throughout the year, and it lagged a few days behind the river runoff. Approximately 80% of the total baseflow was generated in the 3500–4500 m alpine regions, with mainly low-temperature and mid-temperature permafrost. Based on the climate, runoff, land use, soil temperature and moisture data of the permafrost active layer, the mechanism of baseflow change in the permafrost zone was analysed. Precipitation and temperature positively enhanced the baseflow in the permafrost region throughout a year, but the baseflow was more influenced by the temperature than precipitation. In the study area, the cold desert and alpine grassland had the largest regulation capacity for baseflow. Affected by the permafrost freeze-thaw process, a baseflow peak occurred in the spring and the baseflow recession slowed in the autumn. This lead to a more uniform distribution of baseflow and runoff throughout the year

Hydrochemical Changes and Influencing Factors in the Dongkemadi Region, Tanggula Range, China

In order to detect the source and controlling factors of hydrochemical ions in glacier meltwater-recharged rivers, the chemical characteristics of the river water, precipitation, and meltwater of the Dongkemadi River Basin, China, in 2014 (from May to October) were systematically analyzed, and combined with the hydrological and meteorological data. The results show that the hydrochemical pattern of the typical river was HCO3−-Ca2+. The most cations were Ca2+ and Mg2+, and the predominant anions were HCO3− and SO42−, in the river. The concentration of major ions and total dissolved solids (TDS) in the river water were much larger than that in the precipitation and meltwater. The TDS concentration was ordered: River water > precipitation > meltwater. The water-rock interaction and the dilution effect of the precipitation and meltwater on the runoff ions resulted in a negative correlation between the ion concentration of the river water and the river flow. The chemical ions of the river runoff mainly originated from rock weathering and the erosion (abrasion) caused by glacier movement. In addition, the contributions of different sources to the dissolved components of the Dongkemadi River were ordered: Carbonate (75.8%) > silicate (15.5%) > hydatogenic rock (5.7%) > atmospheric precipitation (3%), calculated by a forward geochemical model. And the hydrochemical weathering rates of carbonate and silicate minerals were 12.30 t·km−2·a−1 and 1.98 t·km−2·a−1, respectively. The CO2 fluxes, consumed by the chemical weathering of carbonate and silicate, were 3.28 × 105 mol·km−2·a−1 and 0.91 × 105 mol·km−2·a−1, respectively

Intense Chemical Weathering at Glacial Meltwater-Dominated Hailuogou Basin in the Southeastern Tibetan Plateau

Climate warming has caused rapid shrinkage of glaciers in the Tibetan Plateau (TP), but the impact of glacier retreat on the chemical denudation rate remains largely unknown at the temperate glacial basins. The chemical weathering processes were examined at a temperate glacial basin (HLG) in the southeastern TP based on comprehensive data from the supraglacial meltwater, proglacial river water, precipitation and groundwater over two glacier melt seasons in 2008 and 2013. The concentrations of major ions and suspended sediments in river water exhibit a pronounced seasonality and display a close relationship with river discharge, suggesting a strong hydrological control on the chemical and physical weathering processes. Runoff chemistry is dominated by carbonate weathering and sulfide oxidation. HCO3−, Ca2+, and/or SO42− are the dominant ions in meltwater, river water, precipitation and groundwater. For river water, HCO3− and Ca2+ primarily come from calcite weathering, and SO42− is mainly derived from pyrite oxidation. Both solute and sediment fluxes are positively related to river discharge (r = 0.69, p < 0.01 for sediments). The solute flux and yields are 18,095–19,435 t·year−1 and 225–241 t·km−2·year−1, and the sediment load and yields are 126,390 t·year−1 and 1570 t·km−2·year−1, respectively. The solute yields, cationic denudation rate (CDR; 2850–3108 Σ*meq+ m−2·year−1) and chemical weathering intensity (CWI; 616–711 Σ*meq+ m−3·year−1) at HLG are higher than those at most basins irrespective of the lithology, suggesting more intense weathering in the TP in comparison to other glacial basins worldwide

Erosion rates deduced from seasonal mass balance along the upper Urumqi River in Tianshan

International audienceWe report measurements performed during two complete flow seasons on the Urumqi River, a proglacial mountain stream in the northeastern flank of the Tianshan, an active mountain range in Central Asia. This survey of flow dynamics and sediment transport (dissolved, suspended and bed loads), together with a 25-year record of daily discharge , enables the assessment of secular denudation rates on this high mountain catchment of Central Asia. Our results show that chemical weathering accounts for more than one-third of the total denudation rate. Sediment transported as bed load cannot be neglected in the balance, given that sand and gravel transport accounts for one third of the solid load of the river. Overall, the mean denudation rates are low, averaging 46 t × km −2 × yr −1 (17–18 m Myr −1). We furthermore analyse the hydrologic record to show that the long-term sediment budget is not dominated by extreme and rare events but by the total amount of rainfall or annual runoff. The rates we obtain are in agreement with rates obtained from the mass balance reconstruction of the Plio-Quaternary gravely deposits of the foreland but signicantly lower than the rates recently obtained from cosmogenic dating of the Kuitun River sands, west of the Urumqi River. We show that the resolution of this incompatibility may have an important consquence for our understanding of the interplay between erosion and tectonics in the semi-humid ranges of Central Asia