Activations and Gradients Compression for Model-Parallel Training

Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a popular approach for training modern models. Information compression can be applied to decrease workers communication time, as it is often a bottleneck in such systems. This work explores how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error feedback approach. We conduct experiments on image classification and language model fine-tuning tasks. Our findings demonstrate that gradients require milder compression rates than activations. We observe that $K=10\%$ is the lowest TopK compression level, which does not harm model convergence severely. Experiments also show that models trained with TopK perform well only when compression is also applied during inference. We find that error feedback techniques do not improve model-parallel training compared to plain compression, but allow model inference without compression with almost no quality drop. Finally, when applied with the AQ-SGD approach, TopK stronger than with $K=30\%$ worsens model performance significantly.Comment: 17 pages, 6 figures, 5 table

Influences of Hillslope Biogeochemistry on Anaerobic Soil Organic Matter Decomposition in a Tundra Watershed

We investigated rates and controls on greenhouse gas (CO2 and CH4) production in two contrasting water‐saturated tundra soils within a permafrost‐affected watershed near Nome, Alaska, United States. Three years of field sample analysis have shown that soil from a fen‐like area in the toeslope of the watershed had higher pH and higher porewater ion concentrations than soil collected from a bog‐like peat plateau at the top of the hillslope. The influence of these contrasting geochemical and topographic environments on CO2 and CH4 production was tested in soil microcosms by incubating both the organic‐ and mineral‐layer soils anaerobically for 55 days. Nitrogen (as NH4Cl) was added to half of the microcosms to test potential effects of N limitation on microbial greenhouse gas production. We found that the organic toeslope soils produced more CO2 and CH4, fueled by higher pH and higher concentrations of water‐extractable organic C (WEOC). Our results also indicate N limitation on CO2 production in the peat plateau soils but not the toeslope soils. Together these results suggest that the weathering and leaching of ions and nutrients from tundra hillslopes can increase the rate of anaerobic soil organic matter decomposition in downslope soils by (1) increasing the pH of soil porewater; (2) providing bioavailable WEOC and fermentation products such as acetate; and (3) relieving microbial N limitation through nutrient runoff. We conclude that the soil geochemistry as mediated by landscape position is an important factor influencing the rate and magnitude of greenhouse gas production in tundra soils

Cryogenic Displacement and Accumulation of Biogenic Methane in Frozen Soils

Evidences of highly localized methane fluxes are reported from the Arctic shelf, hot spots of methane emissions in thermokarst lakes, and are believed to evolve to features like Yamal crater on land. The origin of large methane outbursts is problematic. Here we show, that the biogenic methane (δ13C ≤ −71‰) which formed before and during soil freezing is presently held in the permafrost. Field and experimental observations show that methane tends to accumulate at the permafrost table or in the coarse-grained lithological pockets surrounded by the sediments less-permeable for gas. Our field observations, radiocarbon dating, laboratory tests and theory all suggest that depending on the soil structure and freezing dynamics, this methane may have been displaced downwards tens of meters during freezing and has accumulated in the lithological pockets. The initial flux of methane from the one pocket disclosed by drilling was at a rate of more than 2.5 kg C(CH4) m−2 h−1. The age of the methane was 8–18 thousand years younger than the age of the sediments, suggesting that it was displaced tens of meters during freezing. The theoretical background provided the insight on the cryogenic displacement of methane in support of the field and experimental data. Upon freezing of sediments, methane follows water migration and either dissipates in the freezing soils or concentrates at certain places controlled by the freezing rate, initial methane distribution and soil structure

Permafrost degradation and its consequences for carbon storage in soils of Interior Alaska

Permafrost soils in the northern hemisphere are known to harbor large amounts of soil organic matter (SOM). Global climate warming endangers this stable soil organic carbon (SOC) pool by triggering permafrost thaw and deepening the active layer, while at the same time progressing soil formation. But depending, e.g., on ice content or drainage, conditions in the degraded permafrost can range from water-saturated/anoxic to dry/oxic, with concomitant shifts in SOM stabilizing mechanisms. In this field study in Interior Alaska, we investigated two sites featuring degraded permafrost, one water-saturated and the other well-drained, alongside a third site with intact permafrost. Soil aggregate- and density fractions highlighted that permafrost thaw promoted macroaggregate formation, amplified by the incorporation of particulate organic matter, in topsoils of both degradation sites, thus potentially counteracting a decrease in topsoil SOC induced by the permafrost thawing. However, the subsoils were found to store notably less SOC than the intact permafrost in all fractions of both degradation sites. Our investigations revealed up to net 75% smaller SOC storage in the upper 100 cm of degraded permafrost soils as compared to the intact one, predominantly related to the subsoils, while differences between soils of wet and dry degraded landscapes were minor. This study provides evidence that the consideration of different permafrost degradation landscapes and the employment of soil fractionation techniques is a useful combination to investigate soil development and SOM stabilization processes in this sensitive ecosystem

Variability in above- and belowground Carbon Stocks in a Siberian Larch Watershed

Permafrost soils store between 1330 and 1580Pg carbon (C), which is 3 times the amount of C in global vegetation, almost twice the amount of C in the atmosphere, and half of the global soil organic C pool. Despite the massive amount of C in permafrost, estimates of soil C storage in the high-latitude permafrost region are highly uncertain, primarily due to undersampling at all spatial scales; circumpolar soil C estimates lack sufficient continental spatial diversity, regional intensity, and replication at the field-site level. Siberian forests are particularly undersampled, yet the larch forests that dominate this region may store more than twice as much soil C as all other boreal forest types in the continuous permafrost zone combined. Here we present above- and belowground C stocks from 20 sites representing a gradient of stand age and structure in a larch watershed of the Kolyma River, near Chersky, Sakha Republic, Russia. We found that the majority of C stored in the top 1m of the watershed was stored belowground (92%), with 19% in the top 10cm of soil and 40% in the top 30cm. Carbon was more variable in surface soils (10cm; coefficient of variation (CV) = 0.35 between stands) than in the top 30cm (CV = 0.14) or soil profile to 1m (CV = 0.20). Combined active-layer and deep frozen deposits (surface – 15m) contained 205kgCm−2 (yedoma, non-ice wedge) and 331kgCm−2 (alas), which, even when accounting for landscape-level ice content, is an order of magnitude more C than that stored in the top meter of soil and 2 orders of magnitude more C than in aboveground biomass. Aboveground biomass was composed of primarily larch (53%) but also included understory vegetation (30%), woody debris (11%) and snag (6%) biomass. While aboveground biomass contained relatively little (8%) of the C stocks in the watershed, aboveground processes were linked to thaw depth and belowground C storage. Thaw depth was negatively related to stand age, and soil C density (top 10cm) was positively related to soil moisture and negatively related to moss and lichen cover. These results suggest that, as the climate warms, changes in stand age and structure may be as important as direct climate effects on belowground environmental conditions and permafrost C vulnerability

Northern Hemisphere permafrost map based on TTOP modelling for 2000-2016 at 1 km²scale

Permafrost is a key element of the cryosphere and an essential climate variable in the Global Climate Observing System. There is no remote-sensing method available to reliably monitor the permafrost thermal state. To estimate permafrost distribution at a hemispheric scale, we employ an equilibrium state model for the temperature at the top of the permafrost (TTOP model) for the 2000–2016 period, driven by remotely-sensed land surface temperatures, down-scaled ERA-Interim climate reanalysis data, tundra wetness classes and landcover map from the ESA Landcover Climate Change Initiative (CCI) project. Subgrid variability of ground temperatures due to snow and landcover variability is represented in the model using subpixel statistics. The results are validated against borehole measurements and reviewed regionally. The accuracy of the modelled mean annual ground temperature (MAGT) at the top of the permafrost is ±2 °C when compared to permafrost borehole data. The modelled permafrost area (MAGT 0) is around 21 × 106 km2 (22% of exposed land area), which is approximately 2 × 106 km2 less than estimated previously. Detailed comparisons at a regional scale show that the model performs well in sparsely vegetated tundra regions and mountains, but is less accurate in densely vegetated boreal spruce and larch forests

Permafrost is warming at a global scale

Permafrost warming has the potential to amplify global climate change, because when frozen sediments thaw it unlocks soil organic carbon. Yet to date, no globally consistent assessment of permafrost temperature change has been compiled. Here we use a global data set of permafrost temperature time series from the Global Terrestrial Network for Permafrost to evaluate temperature change across permafrost regions for the period since the International Polar Year (2007-2009). During the reference decade between 2007 and 2016, ground temperature near the depth of zero annual amplitude in the continuous permafrost zone increased by 0.39 ± 0.15 °C. Over the same period, discontinuous permafrost warmed by 0.20 ± 0.10 °C. Permafrost in mountains warmed by 0.19 ± 0.05 °C and in Antarctica by 0.37 ± 0.10 °C. Globally, permafrost temperature increased by 0.29 ± 0.12 °C. The observed trend follows the Arctic amplification of air temperature increase in the Northern Hemisphere. In the discontinuous zone, however, ground warming occurred due to increased snow thickness while air temperature remained statistically unchanged

The state of the Martian climate

60°N was +2.0°C, relative to the 1981–2010 average value (Fig. 5.1). This marks a new high for the record. The average annual surface air temperature (SAT) anomaly for 2016 for land stations north of starting in 1900, and is a significant increase over the previous highest value of +1.2°C, which was observed in 2007, 2011, and 2015. Average global annual temperatures also showed record values in 2015 and 2016. Currently, the Arctic is warming at more than twice the rate of lower latitudes