Training Transformers with 4-bit Integers

Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by up to 35.1%.Comment: 9 pages, 8 figure

A meta learning scheme for fast accent domain expansion in Mandarin speech recognition

Spoken languages show significant variation across mandarin and accent. Despite the high performance of mandarin automatic speech recognition (ASR), accent ASR is still a challenge task. In this paper, we introduce meta-learning techniques for fast accent domain expansion in mandarin speech recognition, which expands the field of accents without deteriorating the performance of mandarin ASR. Meta-learning or learn-to-learn can learn general relation in multi domains not only for over-fitting a specific domain. So we select meta-learning in the domain expansion task. This more essential learning will cause improved performance on accent domain extension tasks. We combine the methods of meta learning and freeze of model parameters, which makes the recognition performance more stable in different cases and the training faster about 20%. Our approach significantly outperforms other methods about 3% relatively in the accent domain expansion task. Compared to the baseline model, it improves relatively 37% under the condition that the mandarin test set remains unchanged. In addition, it also proved this method to be effective on a large amount of data with a relative performance improvement of 4% on the accent test set

Experimental characterization of the 4D tensor monopole and topological nodal rings

Quantum mechanics predicts the existence of the Dirac and the Yang monopoles. Although their direct experimental observation in high-energy physics is still lacking, these monopoles, together with their associated vector gauge fields, have been demonstrated in synthetic matter. On the other hand, monopoles in even-dimensional spaces have proven more elusive. A potential unifying framework--string theory--that encompasses quantum mechanics promotes the vector gauge fields to tensor gauge fields, and predicts the existence of more exotic tensor monopole in 4D space. Here we report the first experimental observation of a tensor monopole in a 4D parameter space synthesized by the spin degrees of freedom of a single solid-state defect in diamond. Using two complementary methods, we reveal the existence of the tensor monopole through measurements of its quantized topological invariant. By introducing a fictitious external field that breaks chiral symmetry, we further observe a novel phase transition to a topological nodal ring semimetal phase that is protected by mirror symmetries.Comment: main: 10 pages, 4 figures + SI: 22 pages, 27 figure

IP3R-dependent mitochondrial dysfunction mediates C5b-9-induced ferroptosis in trichloroethylene-caused immune kidney injury

Patients with occupational medicamentose-like dermatitis due to trichloroethylene often suffer from immune kidney injury. Our previous study reveals that C5b-9-dependent cytosolic Ca2+ overload-induced ferroptosis is involved in trichloroethylene sensitized kidney injury. However, how C5b-9 causes cytosolic Ca2+ rise and the specific mechanism whereby overloaded Ca2+ induces ferroptosis remain unknown. The purpose of our study was to explore the role of IP3R-dependent mitochondrial dysfunction in C5b-9 mediated ferroptosis in trichloroethylene sensitized kidney. Our results showed that IP3R was activated, and mitochondrial membrane potential was decreased in the renal epithelial cells of trichloroethylene-sensitized mice, and these changes were antagonized by CD59, a C5b-9 inhibitory protein. Moreover, this phenomenon was reproduced in a C5b-9-attacked HK-2 cell model. Further investigation showed that RNA interference with IP3R not only alleviated C5b-9-induced cytosolic Ca2+ overload and mitochondrial membrane potential loss but also attenuated C5b-9-induced ferroptosis in HK-2 cells. Mechanistically, IP3R-dependent cytosolic Ca2+ overload activated the mitochondrial permeability transition pore, resulting in the loss of mitochondrial membrane potential and ferroptosis of HK-2 cells. Finally, cyclosporin A, a mitochondrial permeability transition pore inhibitor, not only ameliorated IP3R-dependent mitochondrial dysfunction but also blocked C5b-9-induced ferroptosis. Taken together, these results suggest that IP3R-dependent mitochondrial dysfunction plays an important role in trichloroethylene sensitized renal tubular ferroptosis

The deubiquitinating enzyme MINDY2 promotes pancreatic cancer proliferation and metastasis by stabilizing ACTN4 expression and activating the PI3K/AKT/mTOR signaling pathway

The pathogenic mechanisms of pancreatic cancer (PC) are still not fully understood. Ubiquitination modifications have a crucial role in tumorigenesis and progression. Yet, the role of MINDY2, a member of the motif interacting with Ub-containing novel DUB family (MINDY), as a newly identified deubiquitinating enzyme, in PC is still unclear. In this study, we found that MINDY2 expression is elevated in PC tissue (clinical samples) and was associated with poor prognosis. We also found that MINDY2 is associated with pro-carcinogenic factors such as epithelial-mesenchymal transition (EMT), inflammatory response, and angiogenesis; the ROC curve suggested that MINDY2 has a high diagnostic value in PC. Immunological correlation analysis suggested that MINDY2 is deeply involved in immune cell infiltration in PC and is associated with immune checkpoint-related genes. In vivo and in vitro experiments further suggested that elevated MINDY2 promotes PC proliferation, invasive metastasis, and EMT. Meanwhile, actinin alpha 4 (ACTN4) was identified as a MINDY2-interacting protein by mass spectrometry and other experiments, and ACTN4 protein levels were significantly correlated with MINDY2 expression. The ubiquitination assay confirmed that MINDY2 stabilizes the ACTN4 protein level by deubiquitination. The pro-oncogenic effect of MINDY2 was significantly inhibited by silencing ACTN4. Bioinformatics Analysis and Western blot experiments further confirmed that MINDY2 stabilizes ACTN4 through deubiquitination and thus activates the PI3K/AKT/mTOR signaling pathway. In conclusion, we identified the oncogenic role and mechanism of MINDY2 in PC, suggesting that MINDY2 is a viable candidate gene for PC and may be a therapeutic target and critical prognostic indicator

Forecasting method of clean energy development potential considering sector coupling

Sector coupling includes not only the coupling within energy sectors such as electricity, heat and gas, but also the coupling between energy sector and transportation sector, construction sector and industrial sector. This article introduces a method for measuring the development potential of clean energy based on sector coupling. First, the analysis model of electric energy substitution potential is constructed, the analysis object is determined, and the analysis object is quantified. On the basis of the definition of clean energy development potential, an IPAT model for electric energy substitution is constructed to realize the comprehensive evaluation of clean energy. Secondly, based on Markov theory to realize the sub-path calculation of the development potential of clean energy. Finally, based on the IPAT model and the decoupling theoretical model, this paper sets up three different alternative scenarios to make a more comprehensive prediction and analysis of the medium and long-term clean energy development potential, and applies the introduced measurement methods to the calculation of China’s clean energy development potential. The results show the effectiveness of the algorithm in the calculation of the development potential of clean energy based on sector coupling

Porous Ni0.5Zn0.5Fe2O4 nanospheres: synthesis, characterization, and application for lithium storage

Monodisperse porous Ni0.5Zn0.5Fe2O4 nanospheres have been successfully synthesized by the solvothermal method. The diameter of the nanospheres can be tuned by controlling the reactant concentration. Lower reactant concentration is favoured for the synthesis of mesoporous Ni0.5Zn0.5Fe2O4 nanospheres with higher surface area. The electrochemical results show that mesoporous Ni0.5Zn0.5Fe2O4 nanospheres exhibit high reversible specific capacity (1110 mAh g-1) for Li storage and high capacity retention, with 700 mAh g-1 retained up to 50 cycles. The excellent electrochemical properties could be attributed to the large surface area and mesoporous structure. The results suggest that Ni0.5Zn0.5Fe2O4 could be a promising high capacity anode material for lithium ion batteries

Strength Analysis and Structure Optimization of the Crankshaft of an Opposed-Power Reciprocating Pump

The opposed-power reciprocating pump has the characteristics of high pressure, large flow, and high efficiency and energy saving. However, due to the special structure of the opposed-power reciprocating pump, existing theoretical methods cannot analyze its dynamic performance. Therefore, this paper proposes a method of analyzing the power end of the opposed-power reciprocating pump. Firstly, according to the working principle and structural characteristics of the traditional plunger pump, the novel and complex structure of the opposed-power reciprocating pump is analyzed by analogy, and the force analysis model of the crankshaft is established. The dynamic analysis model of the Matlab program is used to solve the dynamic load and section stress in the working process, and the variation law of crankshaft load is obtained. The 25 most critical working conditions are selected for analysis, and the most critical station and section of the crankshaft are obtained. With the connection between ANSYS Workbench and Solidworks, the model is imported into ANSYS Workbench, the load on the crank pin is loaded by APDL command flow, and the static analysis of the crankshaft is carried out to obtain the stress and strain of the crankshaft. Finally, the static and fatigue strength of the dangerous section is checked, and it is proven that the strength and stiffness of the crankshaft meet the design requirements. The results show that the dynamic analysis results of the crankshaft under critical working conditions are consistent with the finite element analysis, verifying the rationality of the method and providing a reference for the improvement and optimized design of the crankshaft of the opposed-power reciprocating pump