DiffRate : Differentiable Compression Rate for Efficient Vision Transformers

Token compression aims to speed up large-scale vision transformers (e.g. ViTs) by pruning (dropping) or merging tokens. It is an important but challenging task. Although recent advanced approaches achieved great success, they need to carefully handcraft a compression rate (i.e. number of tokens to remove), which is tedious and leads to sub-optimal performance. To tackle this problem, we propose Differentiable Compression Rate (DiffRate), a novel token compression method that has several appealing properties prior arts do not have. First, DiffRate enables propagating the loss function's gradient onto the compression ratio, which is considered as a non-differentiable hyperparameter in previous work. In this case, different layers can automatically learn different compression rates layer-wisely without extra overhead. Second, token pruning and merging can be naturally performed simultaneously in DiffRate, while they were isolated in previous works. Third, extensive experiments demonstrate that DiffRate achieves state-of-the-art performance. For example, by applying the learned layer-wise compression rates to an off-the-shelf ViT-H (MAE) model, we achieve a 40% FLOPs reduction and a 1.5x throughput improvement, with a minor accuracy drop of 0.16% on ImageNet without fine-tuning, even outperforming previous methods with fine-tuning. Codes and models are available at https://github.com/OpenGVLab/DiffRate.Comment: 16 pages, 8 figures, 13 table

Anisotropic shock responses of nanoporous Al by molecular dynamics simulations.

Mechanical responses of nanoporous aluminum samples under shock in different crystallographic orientations (, , , and ) are investigated by molecular dynamics simulations. The shape evolution of void during collapse is found to have no relationship with the shock orientation. Void collapse rate and dislocation activities at the void surface are found to strongly dependent on the shock orientation. For a relatively weaker shock, void collapses fastest when shocked along the orientation; while for a relatively stronger shock, void collapses fastest in the orientation. The dislocation nucleation position is strongly depended on the impacting crystallographic orientation. A theory based on resolved shear stress is used to explain which slip planes the earliest-appearing dislocations prefer to nucleate on under different shock orientations

3D Characterization of Pore Structure and Pore Scale Seepage Simulation of Sandstone Based on Computational Tomography

The microscopic pore structure of sandstone determines its macroscopic permeability. Based on computer tomography (CT) technology, CT scans were performed on three different types of sandstone pore structures, namely coarse sandstone, medium sandstone, and fine sandstone. And the three-dimensional microscopic structure of sandstone pores was reconstructed. Furthermore, based on the Navier–Stokes equations, the fluid flow process in the pore structure of sandstone was simulated, and the effective permeability of sandstone was obtained. By extracting the pore structure from sandstone CT images, the average porosity of coarse sandstone, medium sandstone, and fine sandstone was 16.43%, 12.03%, and 11.64%, respectively. And the porosity of unconnected pores is less than 0.5%. The porosity and permeability of coarse sandstone are higher than those of medium sandstone and fine sandstone with an average value of 1.7 D. The porosity of medium sandstone and fine sandstone is relatively similar. However, the average pore radius and pore throat radius of medium sandstone are larger than those of fine sandstone. More importantly, although the permeability and porosity of sandstone are generally linearly related, when the porosity is low, the data show a large dispersion, and auxiliary indicators such as pore structure characteristic parameters such as pore throat radius should be adopted to evaluate the permeability of sandstone. The flow trajectory of fluid in the pore structure of sandstone is revealed through the streamline of fluid in the pore structure, revealing the mechanism of fluid flow

Clay mineral evolution and formation of intermediate phases during pedogenesis on picrite basalt bedrock under temperate conditions (Yunnan, southwestern China)

International audienceIn order to better understanding clay mineral evolution coupled with geochemical changes during weathering of igneous rocks under temperate conditions, the picrite basalt-derived soil in Dali (South China) was investigated using X-ray diffraction (XRD), X-ray fluorescence (XRF), scanning electron microscope (SEM), and high-resolution transmission electron microscopy (HRTEM) methods. Our results show that smectite and mixed-layer clays occur throughout the weathered soil profile, while discrete kaolinite and illite phases are absent in the soil. From saprolite to topsoil, smectite decreases sharply, while mixed-layer kaolinite/smectite (K/S) and illite/smectite (I/S) clays increase markedly. Clay minerals in the saprolite consist of both dioctahedral and trioctahedral species, while those of the middle to upper profile display a uniform dioctahedral structure. I/S phases are characterized by interstratification of 12-Å smectite and 10-Å illite layers, and K/S phases by interstratification of 12-Å smectite and 7-Å kaolinite layers, with I/S/K clays containing all three layer types. Kaolinization of smectite occurs at the initial stage of weathering, earlier than that of smectite illitization. Desilication and K-fixation of smectite take place simultaneously during advanced weathering, resulting in formation of illitic and kaolinitic phases continuously throughout the developmental history of the soil. Notably high K concentrations in the weathering profile may be related to K-fixing in the interlayer of illite due to smectite illitization in response to more advanced weathering and pedogenic processes, while the increasing K content in the topsoil may be ascribed to fertilizer in land use and uptake of K by plants from deeper soil horizons. Fe2O3 and TiO2 accumulation in the topsoil is probably mediated by microorganisms

Association of maternal folate status in the second trimester of pregnancy with the risk of gestational diabetes mellitus

Interest in the high folate status of pregnant women has increased due to its role in the prevention of neural tube defects (NTDs). The effect of increased red blood cell (RBC) folate status during the second trimester of pregnancy on gestational diabetes mellitus (GDM) remains unclear. We measured RBC folate concentrations by competitive protein‐binding assay and obtained clinical information from electronic medical records. Logistic regression analysis was used to explore the associations of RBC folate concentrations with risks of gestational diabetes mellitus (GDM). We further assessed the potential nonlinear relations between continuous log‐transformed RBC folate concentrations and GDM risk by using the restricted cubic splines. We observed high RBC folate concentrations in GDM patients compared to control group [median (interquartile range, IQR), GDM vs. controls: 1,554.03 (1,240.54–1,949.99) vs. 1,478.83 (1,124.60–1,865.71) nmol/L, p = .001]. Notably, high folate concentrations were significantly associated with an increased risk of GDM [RR per 1‐SD increase: 1.16 (1.03, 1.30), p = .012] after adjustment for maternal age, parity, and body mass index (BMI) at enrollment. In the restricted cubic spline model, a test of the null hypothesis of the linear relationship was rejected (p = .001). Our study firstly showed that maternal RBC folate concentrations during the second trimester of pregnancy increase the risk of GDM in a Chinese population. Further randomized clinical trials (RCTs) are warranted to confirm the adverse effect

Association of maternal folate status in the second trimester of pregnancy with the risk of gestational diabetes mellitus

Interest in the high folate status of pregnant women has increased due to its role in the prevention of neural tube defects (NTDs). The effect of increased red blood cell (RBC) folate status during the second trimester of pregnancy on gestational diabetes mellitus (GDM) remains unclear. We measured RBC folate concentrations by competitive protein‐binding assay and obtained clinical information from electronic medical records. Logistic regression analysis was used to explore the associations of RBC folate concentrations with risks of gestational diabetes mellitus (GDM). We further assessed the potential nonlinear relations between continuous log‐transformed RBC folate concentrations and GDM risk by using the restricted cubic splines. We observed high RBC folate concentrations in GDM patients compared to control group [median (interquartile range, IQR), GDM vs. controls: 1,554.03 (1,240.54–1,949.99) vs. 1,478.83 (1,124.60–1,865.71) nmol/L, p = .001]. Notably, high folate concentrations were significantly associated with an increased risk of GDM [RR per 1‐SD increase: 1.16 (1.03, 1.30), p = .012] after adjustment for maternal age, parity, and body mass index (BMI) at enrollment. In the restricted cubic spline model, a test of the null hypothesis of the linear relationship was rejected (p = .001). Our study firstly showed that maternal RBC folate concentrations during the second trimester of pregnancy increase the risk of GDM in a Chinese population. Further randomized clinical trials (RCTs) are warranted to confirm the adverse effect

Elevated physical weathering exceeds chemical weathering of clays during the Paleocene-Eocene Thermal Maximum in the continental Bighorn Basin (Wyoming, USA)

The Paleocene-Eocene Thermal Maximum (PETM) global warming event at ∼56 million years before present changed catchment weathering and erosion. Increased chemical weathering of silicate minerals is thought to be an important process removing CO2 from the atmosphere. However, changes in clay mineralogy can often be explained by enhanced erosion of catchment laterites during the event. Here, we investigate chemical and physical weathering and erosive flux changes through the PETM interval in the Bighorn Basin, Wyoming, a Laramide foreland basin, in a proximal continental-interior alluvial setting. These show an increase of detrital smectite with a lag time of 20-kyr after the main onset the PETM. The smectite increase continued for at least 50-kyr after the event. In-situ, post-depositional pedogenic clay mineral formation is similar between pre-PETM and PETM soil profiles, despite large macroscopic differences between soils that formed before and during the event. Drier, hotter summers during the PETM probably caused decreased vegetation cover that, in concert with more frequent and heavier rainstorms, intensified the erosion of smectite-rich Cretaceous bentonites on the margins of the catchment, which exceeded changes in chemical weathering within the catchment. The lagged response in reaching full PETM clay mineral values can be explained by the time required for upstream sediment to reach the catchment basin floodplain. The prolonged nature of smectite enhancement after the PETM event may again relate to signal propagation times that are now even longer due to lower fluvial recycling rates. Our results indicate that chemical weathering changes were probably superceded by enhanced physical weathering and clay-mineral transport from basin margins at this continental-interior study site

Elevated physical weathering exceeds chemical weathering of clays during the Paleocene-Eocene Thermal Maximum in the continental Bighorn Basin (Wyoming, USA)

The Paleocene-Eocene Thermal Maximum (PETM) global warming event at ∼56 million years before present changed catchment weathering and erosion. Increased chemical weathering of silicate minerals is thought to be an important process removing CO2 from the atmosphere. However, changes in clay mineralogy can often be explained by enhanced erosion of catchment laterites during the event. Here, we investigate chemical and physical weathering and erosive flux changes through the PETM interval in the Bighorn Basin, Wyoming, a Laramide foreland basin, in a proximal continental-interior alluvial setting. These show an increase of detrital smectite with a lag time of 20-kyr after the main onset the PETM. The smectite increase continued for at least 50-kyr after the event. In-situ, post-depositional pedogenic clay mineral formation is similar between pre-PETM and PETM soil profiles, despite large macroscopic differences between soils that formed before and during the event. Drier, hotter summers during the PETM probably caused decreased vegetation cover that, in concert with more frequent and heavier rainstorms, intensified the erosion of smectite-rich Cretaceous bentonites on the margins of the catchment, which exceeded changes in chemical weathering within the catchment. The lagged response in reaching full PETM clay mineral values can be explained by the time required for upstream sediment to reach the catchment basin floodplain. The prolonged nature of smectite enhancement after the PETM event may again relate to signal propagation times that are now even longer due to lower fluvial recycling rates. Our results indicate that chemical weathering changes were probably superceded by enhanced physical weathering and clay-mineral transport from basin margins at this continental-interior study site.Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Applied Geolog