ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer

Vision Transformers (ViTs) have shown impressive performance and have become a unified backbone for multiple vision tasks. But both attention and multi-layer perceptions (MLPs) in ViTs are not efficient enough due to dense multiplications, resulting in costly training and inference. To this end, we propose to reparameterize the pre-trained ViT with a mixture of multiplication primitives, e.g., bitwise shifts and additions, towards a new type of multiplication-reduced model, dubbed $\textbf{ShiftAddViT}$, which aims for end-to-end inference speedups on GPUs without the need of training from scratch. Specifically, all $\texttt{MatMuls}$ among queries, keys, and values are reparameterized by additive kernels, after mapping queries and keys to binary codes in Hamming space. The remaining MLPs or linear layers are then reparameterized by shift kernels. We utilize TVM to implement and optimize those customized kernels for practical hardware deployment on GPUs. We find that such a reparameterization on (quadratic or linear) attention maintains model accuracy, while inevitably leading to accuracy drops when being applied to MLPs. To marry the best of both worlds, we further propose a new mixture of experts (MoE) framework to reparameterize MLPs by taking multiplication or its primitives as experts, e.g., multiplication and shift, and designing a new latency-aware load-balancing loss. Such a loss helps to train a generic router for assigning a dynamic amount of input tokens to different experts according to their latency. In principle, the faster experts run, the larger amount of input tokens are assigned. Extensive experiments consistently validate the effectiveness of our proposed ShiftAddViT, achieving up to \textbf{5.18\times} latency reductions on GPUs and \textbf{42.9%} energy savings, while maintaining comparable accuracy as original or efficient ViTs.Comment: Accepted by NeurIPS 202

4-[4-(Piperidin-1-yl)piperidin-1-yl]benzonitrile

In the title compound, C17H23N3, both piperidine rings adopt chair conformations. In the crystal packing, intermolecular C—H⋯N hydrogen bonds and C—H⋯π interactions are present

Intelligent ZHENG Classification of Hypertension Depending on ML-kNN and Information Fusion

Hypertension is one of the major causes of heart cerebrovascular diseases. With a good accumulation of hypertension clinical data on hand, research on hypertension's ZHENG differentiation is an important and attractive topic, as Traditional Chinese Medicine (TCM) lies primarily in “treatment based on ZHENG differentiation.” From the view of data mining, ZHENG differentiation is modeled as a classification problem. In this paper, ML-kNN—a multilabel learning model—is used as the classification model for hypertension. Feature-level information fusion is also used for further utilization of all information. Experiment results show that ML-kNN can model the hypertension's ZHENG differentiation well. Information fusion helps improve models' performance

Associated Z0H0Z^0H^0 production with leptonic decays at LHC in next-to-leading order QCD

In this work we investigate the effects of the littlest Higgs model (LHM) up to the QCD next-to-leading order (NLO) on the $Z^0H^0$ associated production at the CERN Large Hadron Collider (LHC). We study the dependences of the leading order and NLO QCD corrected integrated cross sections for this process on the factorization/renormalization scale and the LHM parameters. We also provide the distributions of the transverse momenta of final decay products $\mu^-$ and $\tau^-$. Our results show that the heavy neutral gauge bosons $Z_H$ and $A_H$ could induce significant discrepancies from the standard model predictions. It is found that when the LHM parameters are taken as $c=0.5$, $c^{\prime}=0.22$, $f=4 TeV$ and $\mu=(M_H+M_Z)/2$, the effects at the $\sqrt{s}=14 TeV$ LHC from the heavy neutral gauge boson are about 12.83% and 10.37% to the leading order and NLO QCD corrected integrated cross sections, respectively. We also conclude that the NLO QCD corrections at the $\sqrt{s}=14 TeV$ LHC can obviously reduce the scale uncertainty of the integrated cross section, and significantly enhance the differential cross sections of $p_T^{\mu^-}$ and $p_T^{\tau^-}$. It demonstrates that the precision measurement of the $Z^0H^0$ associated production process at the LHC could provide the clue of the LHM physics.Comment: 26 pages, 11 figure

Anxiolytic-Like Effects of Compound Zhi Zhu Xiang in Rats

The purpose of this study was to determine whether compound zhi zhu xiang (CZZX) exerts anxiolytic-like effects in rats. The animals were orally administered CZZX (0.75, 1.5, and 3 g/kg daily) for 10 days and tested in the elevated plus maze (EPM), Vogel conflict test (VCT), and open field. Repeated treatment with CZZX (3 g/kg/day, p.o.) significantly increased the percentage of both entries into and time spent on the open arms of the EPM compared with saline controls. In the VCT, repeated treatment with CZZX (1.5 and 3 g/kg/day, p.o.) significantly increased the number of punished licks. The drug did not change the total entries into the open arms of the EPM or interfere with water consumption or nociceptive threshold, discarding potential confounding factors in the two tests. In the open field, locomotion was not reduced, discarding the possible sedative effect of CZZX. In the binding assay, the binding of [3H] Ro 15-1788 (flumazenil) to the benzodiazepine binding site in washed crude synaptosomal membranes from rat cerebral cortex was affected by CZZX. These data indicate an anxiolytic-like profile of action for CZZX without sedative side effects, and this activity may be mediated by benzodiazepine binding site modulation at γ-aminobutyric acid-A receptors

Genome-wide investigation and expression analyses of the pentatricopeptide repeat protein gene family in foxtail millet

Orthologous relationships of the PPR genes between foxtail millet and those of other grass species. (TIF 5719 kb