NTK-approximating MLP Fusion for Efficient Language Model Fine-tuning

Fine-tuning a pre-trained language model (PLM) emerges as the predominant strategy in many natural language processing applications. However, even fine-tuning the PLMs and doing inference are expensive, especially on edge devices with low computing power. Some general approaches (e.g. quantization and distillation) have been widely studied to reduce the compute/memory of PLM fine-tuning, while very few one-shot compression techniques are explored. In this paper, we investigate the neural tangent kernel (NTK)--which reveals the gradient descent dynamics of neural networks--of the multilayer perceptrons (MLP) modules in a PLM and propose to coin a lightweight PLM through NTK-approximating MLP fusion. To achieve this, we reconsider the MLP as a bundle of sub-MLPs, and cluster them into a given number of centroids, which can then be restored as a compressed MLP and surprisingly shown to well approximate the NTK of the original PLM. Extensive experiments of PLM fine-tuning on both natural language understanding (NLU) and generation (NLG) tasks are provided to verify the effectiveness of the proposed method MLP fusion. Our code is available at https://github.com/weitianxin/MLP_Fusion.Comment: ICML 202

A combined SEM, CV and EIS study of multi-layered porous ceramic reactors for flue gas purification

The effect of sintering temperature of 12-layered porous ceramic reactors (comprising 5 cells) was studied using scanning electron microscopy (SEM), cyclic voltammetry (CV) and electrochemical impedance spectroscopy (EIS). The difference in microstructures of the reactors was evaluated by SEM. Additional information on the influence of sintering temperature on the properties of the reactors could be gained by the use of EIS. The present work has provided the first set of fundamental electrochemical data and their interpretation in terms of fabrication conditions, for the multi-layered porous ceramic reactors

Optimizing the performance of porous electrochemical cells for flue gas purification using the DOE method

The DOE model was used to improve the performance of cells for electrochemical gas purification. Three factors were chosen: the amount of graphite, the Lanthanum Strontium Manganate/Gadolinium-doped Cerium oxide weight % ratio, and the Lanthanum Strontium Manganate pre-calcination temperature (with or without Lanthanum Strontium Manganate calcinated at 1000 °C). The effects of the following physical properties were measured: porosity, pore size, shrinkage, and conductivity. The sintered tapes were also characterized with scanning electron microscopy. Graphite was added as a pore former. The work shows, that a change in a factor not only changes the performance property that one would expect, but also influence other properties

Particle growth with photochemical age from new particle formation to haze in the winter of Beijing, China

Secondary aerosol formation in the aging process of primary emission is the main reason for haze pollution in eastern China. Pollution evolution with photochemical age was studied for the first time at a comprehensive field observation station during winter in Beijing. The photochemical age was used as an estimate of the time scale attributed to the aging process and was estimated from the ratio of toluene to benzene in this study. A low photochemical age indicates a fresh emission. The photochemical age of air masses during new particle formation (NPF) days was lower than that on haze days. In general, the strongest NPF events, along with a peak of the formation rate of 1.5 nm(J(1.5)) and 3 nmparticles (J(3)), were observed when the photochemical age was between 12 and 24 h while rarely took place with photochemical ages less than 12 h. When photochemical age was larger than 48 h, haze occurred and NPF was suppressed. The sources and sinks of nanoparticles had distinct relation with the photochemical age. Our results show that the condensation sink (CS) showed a valley with photochemical ages ranging from 12 to 24 h, while H2SO4 concentration showed no obvious trend with the photochemical age. The high concentrations of precursor vapours within an air mass lead to persistent nucleation with photochemical age ranging from 12 to 48 h in winter. Coincidently, the fast increase of PM2.5 mass was also observed during this range of photochemical age. Noteworthy, CS increased with the photochemical age on NPF days only, which is the likely reason for the observation that the PM2.5 mass increased faster with photochemical age on NPF days compared with other days. The evolution of particles with the photochemical age provides new insights into understanding how particles originating from NPF transform to haze pollution. (C) 2020 Elsevier B.V. All rights reserved.Peer reviewe