ConvFormer: Closing the Gap Between CNN and Vision Transformers

Vision transformers have shown excellent performance in computer vision tasks. However, the computation cost of their (local) self-attention mechanism is expensive. Comparatively, CNN is more efficient with built-in inductive bias. Recent works show that CNN is promising to compete with vision transformers by learning their architecture design and training protocols. Nevertheless, existing methods either ignore multi-level features or lack dynamic prosperity, leading to sub-optimal performance. In this paper, we propose a novel attention mechanism named MCA, which captures different patterns of input images by multiple kernel sizes and enables input-adaptive weights with a gating mechanism. Based on MCA, we present a neural network named ConvFormer. ConvFormer adopts the general architecture of vision transformers, while replacing the (local) self-attention mechanism with our proposed MCA. Extensive experimental results demonstrated that ConvFormer achieves state-of-the-art performance on ImageNet classification, which outperforms similar-sized vision transformers(ViTs) and convolutional neural networks (CNNs). Moreover, for object detection on COCO and semantic segmentation tasks on ADE20K, ConvFormer also shows excellent performance compared with recently advanced methods. Code and models will be available

OVO: One-shot Vision Transformer Search with Online distillation

Pure transformers have shown great potential for vision tasks recently. However, their accuracy in small or medium datasets is not satisfactory. Although some existing methods introduce a CNN as a teacher to guide the training process by distillation, the gap between teacher and student networks would lead to sub-optimal performance. In this work, we propose a new One-shot Vision transformer search framework with Online distillation, namely OVO. OVO samples sub-nets for both teacher and student networks for better distillation results. Benefiting from the online distillation, thousands of subnets in the supernet are well-trained without extra finetuning or retraining. In experiments, OVO-Ti achieves 73.32% top-1 accuracy on ImageNet and 75.2% on CIFAR-100, respectively.Comment: arXiv admin note: substantial text overlap with arXiv:2107.00651 by other author

Learning Convolutional Neural Networks in the Frequency Domain

Convolutional neural network (CNN) has achieved impressive success in computer vision during the past few decades. The image convolution operation helps CNNs to get good performance on image-related tasks. However, the image convolution has high computation complexity and hard to be implemented. This paper proposes the CEMNet, which can be trained in the frequency domain. The most important motivation of this research is that we can use the straightforward element-wise multiplication operation to replace the image convolution in the frequency domain based on the Cross-Correlation Theorem, which obviously reduces the computation complexity. We further introduce a Weight Fixation mechanism to alleviate the problem of over-fitting, and analyze the working behavior of Batch Normalization, Leaky ReLU, and Dropout in the frequency domain to design their counterparts for CEMNet. Also, to deal with complex inputs brought by Discrete Fourier Transform, we design a two-branches network structure for CEMNet. Experimental results imply that CEMNet achieves good performance on MNIST and CIFAR-10 databases.Comment: Submitted to NeurIPS 202

Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks

Text classification tasks often encounter few shot scenarios with limited labeled data, and addressing data scarcity is crucial. Data augmentation with mixup has shown to be effective on various text classification tasks. However, most of the mixup methods do not consider the varying degree of learning difficulty in different stages of training and generate new samples with one hot labels, resulting in the model over confidence. In this paper, we propose a self evolution learning (SE) based mixup approach for data augmentation in text classification, which can generate more adaptive and model friendly pesudo samples for the model training. SE focuses on the variation of the model's learning ability. To alleviate the model confidence, we introduce a novel instance specific label smoothing approach, which linearly interpolates the model's output and one hot labels of the original samples to generate new soft for label mixing up. Through experimental analysis, in addition to improving classification accuracy, we demonstrate that SE also enhances the model's generalize ability

The expression and antigenicity of a truncated spike-nucleocapsid fusion protein of severe acute respiratory syndrome-associated coronavirus

<p>Abstract</p> <p>Background</p> <p>In the absence of effective drugs, controlling SARS relies on the rapid identification of cases and appropriate management of the close contacts, or effective vaccines for SARS. Therefore, developing specific and sensitive laboratory tests for SARS as well as effective vaccines are necessary for national authorities.</p> <p>Results</p> <p>Genes encoding truncated nucleocapsid (N) and spike (S) proteins of <it>SARSCoV </it>were cloned into the expression vector <it>pQE30 </it>and fusionally expressed in <it>Escherichia coli </it>M15. The fusion protein was analyzed for reactivity with SARS patients' sera and with anti-sera against the two human coronaviruses <it>HCoV </it>229E and <it>HCoV </it>OC43 by ELISA, IFA and immunoblot assays. Furthermore, to evaluate the antigen-specific humoral antibody and T-cell responses in mice, the fusion protein was injected into 6-week-old BALB/c mice and a neutralization test as well as a T-cell analysis was performed. To evaluate the antiviral efficacy of immunization, BALB/c mice were challenged intranasally with <it>SARSCoV </it>at day 33 post injection and viral loads were determined by fluorescent quantitative RT-PCR. Serological results showed that the diagnostic sensitivity and specificity of the truncated S-N fusion protein derived the SARS virus were > 99% (457/460) and 100.00% (650/650), respectively. Furthermore there was no cross-reactivity with other two human coronaviruses. High titers of antibodies to <it>SRASCoV </it>appeared in the immunized mice and the neutralization test showed that antibodies to the fusion protein could inhibit <it>SARSCoV</it>. The T cell proliferation showed that the fusion protein could induce an antigen-specific T-cell response. Fluorescent quantitative RT-PCR showed that BALB/c mice challenged intranasally with <it>SARSCoV </it>at day 33 post injection were completely protected from virus replication.</p> <p>Conclusion</p> <p>The truncated S-N fusion protein is a suitable immunodiagnostic antigen and vaccine candidate.</p

Analysis on the technical detection capacity of radioactive substances in food in China

To analyze the detection capacity of radioactive substances in food in China， and improve the radioactive contamination monitoring system. By studying the distribution of certified institutions and testing items and the results of proficiency assessment， the current situation and deficiencies of the detection capacities were analyzed， and corresponding countermeasures were put forward. The capacity of radioactive material detection in China can better support the operation of the monitoring system， however， the effectiveness and sustainability of testing capacity， the layout of capacity network and the construction of food radioactive pollution monitoring system need to be further improved and strengthened， so as to meet the needs of normal circumstances and rapid response in case of nuclear or radiological emergencies in China

Neuroprotective Effects of Pre-Treament with l-Carnitine and Acetyl-l-Carnitine on Ischemic Injury In Vivo and In Vitro

The therapeutic effect of stroke is hampered by the lack of neuroprotective drugs against ischemic insults beyond the acute phase. Carnitine plays important roles in mitochondrial metabolism and in modulating the ratio of coenzyme A (CoA)/acyl-CoA. Here, we investigate the neuroprotective effects of l-carnitine (LC) and Acetyl-l-carnitine (ALC) pre-treatment on ischemic insults under the same experimental conditions. We used a transient middle cerebral artery occlusion (MCAO) model to evaluate the protective roles of LC and ALC in acute focal cerebral ischemia in vivo and to understand the possible mechanisms using model of PC12 cell cultures in vitro. Results showed that ALC, but not LC, decreased infarction size in SD rats after MCAO in vivo. However, both LC and ALC pretreatment reduced oxygen-glucose deprivation (OGD)-induced cell injury and decreased OGD-induced cell apoptosis and death in vitro; at the same time, both of them increased the activities of super oxide dismutase (SOD) and ATPase, and decreased the concentration of malondialdehyde (MDA) in vitro. Thus, our findings suggested that LC and ALC pre-treatment are highly effective in the prevention of neuronal cell against ischemic injury in vitro, however, only ALC has the protective effect on neuronal cell injury after ischemia in vivo

IL28B Genetic Variation Is Associated with Spontaneous Clearance of Hepatitis C Virus, Treatment Response, Serum IL-28B Levels in Chinese Population

&lt;p&gt;&lt;b&gt;Background:&lt;/b&gt; The interleukin-28B gene (IL28B) locus has been associated with host resistance to hepatitis C virus (HCV) infection and response to PEG-IFN/RBV treatment in western populations. This study was to determine whether this gene variant is also associated with spontaneous clearance of HCV infection, treatment response and IL-28B protein production in Chinese patients.&lt;/p&gt; &lt;p&gt;&lt;b&gt;Methods:&lt;/b&gt; We genotyped IL28B genetic variations (rs12980275, rs8103142, rs8099917 and rs12979860) by pyrosequencing DNA samples from cohorts consisting of 529 subjects with persistent HCV infection, 196 subjects who cleared the infection, 171 healthy individuals and 235 chronic HCV patients underwent IFN/RBV treatment. The expression of IL-28B were measured by ELISA and RT-PCR.&lt;/p&gt; &lt;p&gt;&lt;b&gt;Results:&lt;/b&gt; We found that the four IL28B variants were in complete linkage disequilibrium (r2 = 0.97–0.98). The rs12979860 CC genotype was strongly associated with spontaneously HCV clearance and successful IFN/RBV treatment compared to the CT/TT. IL-28B levels in persistent HCV patients were significantly lower than subjects who spontaneously resolved HCV and healthy controls and were also associated with high levels of ALT (alanine aminotransferase) and AST (aspartate aminotransferase). IL-28B levels were also significantly lower in individuals carrying T alleles than CC homozygous.&lt;/p&gt; &lt;p&gt;&lt;b&gt;Conclusions:&lt;/b&gt; Thus, the rs12979860-CC variant upstream of IL28B gene is associated with spontaneous clearance of HCV, susceptible to IFN/RBV treatment and increased IL-28B levels in this Chinese population.&lt;/p&gt