Controller design for synchronization of an array of delayed neural networks using a controllable

This is the post-print version of the Article - Copyright @ 2011 ElsevierIn this paper, a controllable probabilistic particle swarm optimization (CPPSO) algorithm is introduced based on Bernoulli stochastic variables and a competitive penalized method. The CPPSO algorithm is proposed to solve optimization problems and is then applied to design the memoryless feedback controller, which is used in the synchronization of an array of delayed neural networks (DNNs). The learning strategies occur in a random way governed by Bernoulli stochastic variables. The expectations of Bernoulli stochastic variables are automatically updated by the search environment. The proposed method not only keeps the diversity of the swarm, but also maintains the rapid convergence of the CPPSO algorithm according to the competitive penalized mechanism. In addition, the convergence rate is improved because the inertia weight of each particle is automatically computed according to the feedback of fitness value. The efficiency of the proposed CPPSO algorithm is demonstrated by comparing it with some well-known PSO algorithms on benchmark test functions with and without rotations. In the end, the proposed CPPSO algorithm is used to design the controller for the synchronization of an array of continuous-time delayed neural networks.This research was partially supported by the National Natural Science Foundation of PR China (Grant No 60874113), the Research Fund for the Doctoral Program of Higher Education (Grant No 200802550007), the Key Creative Project of Shanghai Education Community (Grant No 09ZZ66), the Key Foundation Project of Shanghai(Grant No 09JC1400700), the Engineering and Physical Sciences Research Council EPSRC of the U.K. under Grant No. GR/S27658/01, an International Joint Project sponsored by the Royal Society of the U.K., and the Alexander von Humboldt Foundation of Germany

Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers

Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.Comment: Accepted to CVPR202

Masked Images Are Counterfactual Samples for Robust Fine-tuning

Deep learning models are challenged by the distribution shift between the training data and test data. Recently, the large models pre-trained on diverse data demonstrate unprecedented robustness to various distribution shifts. However, fine-tuning on these models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. Existing methods for tackling this trade-off do not explicitly address the OOD robustness problem. In this paper, based on causal analysis on the aforementioned problems, we propose a novel fine-tuning method, which use masked images as counterfactual samples that help improving the robustness of the fine-tuning model. Specifically, we mask either the semantics-related or semantics-unrelated patches of the images based on class activation map to break the spurious correlation, and refill the masked patches with patches from other images. The resulting counterfactual samples are used in feature-based distillation with the pre-trained model. Extensive experiments verify that regularizing the fine-tuning with the proposed masked images can achieve a better trade-off between ID and OOD performance, surpassing previous methods on the OOD performance. Our code will be publicly available.Comment: Accepted by CVPR 2023 (v2: improve the clarity

Sensitivity of Space-based Gravitational-Wave Interferometers to Ultralight Bosonic Fields and Dark Matter

Ultralight bosonic fields (ULBFs) are predicted by various theories beyond the standard model of particle physics and are viable candidates of cold dark matter. There have been increasing interests to search for the ULBFs in physical and astronomical experiments. In this paper, we investigate the sensitivity of several planned space-based gravitational-wave interferometers to ultralight scalar and vector fields. Using time-delay interferometry (TDI) to suppress the overwhelming laser frequency noise, we derive the averaged transfer functions of different TDI combinations to scalar and vector fields, and estimate the impacts of bosonic field's velocities. We obtain the sensitivity curves for LISA, Taiji and TianQin, and explore their projected constraints on the couplings between ULBFs and standard model particles, illustrating with the ULBFs as dark matter.Comment: 33 pages, 8 figure

Boosting Generalization with Adaptive Style Techniques for Fingerprint Liveness Detection

We introduce a high-performance fingerprint liveness feature extraction technique that secured first place in LivDet 2023 Fingerprint Representation Challenge. Additionally, we developed a practical fingerprint recognition system with 94.68% accuracy, earning second place in LivDet 2023 Liveness Detection in Action. By investigating various methods, particularly style transfer, we demonstrate improvements in accuracy and generalization when faced with limited training data. As a result, our approach achieved state-of-the-art performance in LivDet 2023 Challenges.Comment: 1st Place in LivDet2023 Fingerprint Representation Challeng

Brain-inspired Graph Spiking Neural Networks for Commonsense Knowledge Representation and Reasoning

How neural networks in the human brain represent commonsense knowledge, and complete related reasoning tasks is an important research topic in neuroscience, cognitive science, psychology, and artificial intelligence. Although the traditional artificial neural network using fixed-length vectors to represent symbols has gained good performance in some specific tasks, it is still a black box that lacks interpretability, far from how humans perceive the world. Inspired by the grandmother-cell hypothesis in neuroscience, this work investigates how population encoding and spiking timing-dependent plasticity (STDP) mechanisms can be integrated into the learning of spiking neural networks, and how a population of neurons can represent a symbol via guiding the completion of sequential firing between different neuron populations. The neuron populations of different communities together constitute the entire commonsense knowledge graph, forming a giant graph spiking neural network. Moreover, we introduced the Reward-modulated spiking timing-dependent plasticity (R-STDP) mechanism to simulate the biological reinforcement learning process and completed the related reasoning tasks accordingly, achieving comparable accuracy and faster convergence speed than the graph convolutional artificial neural networks. For the fields of neuroscience and cognitive science, the work in this paper provided the foundation of computational modeling for further exploration of the way the human brain represents commonsense knowledge. For the field of artificial intelligence, this paper indicated the exploration direction for realizing a more robust and interpretable neural network by constructing a commonsense knowledge representation and reasoning spiking neural networks with solid biological plausibility

Neutrino decay as a possible interpretation to the MiniBooNE observation with unparticle scenario

In a new measurement on neutrino oscillation $\nu_{\mu}\to\nu_e$, the MiniBooNE Collaboration observes an excess of electron-like events at low energy and the phenomenon may demand an explanation which obviously is beyond the oscillation picuture. We propose that heavier neutrino $\nu_2$ decaying into a lighter one $\nu_1$ via the transition process $\nu_{\mu}\to \nu_e+X$ where $X$ denotes any light products, could be a natural mechanism. The theoretical model we employ here is the unparticle scenario established by Georgi. We have studied two particular modes \nu_\mu\to \nu_e+\Un and $\nu_\mu\to \nu_e+\bar\nu_e+\nu_e$. Unfortunately, the number coming out from the computation is too small to explain the observation. Moreover, our results are consistent with the cosmology constraint on the neutrino lifetime and the theoretical estimation made by other groups, therefore we can conclude that even though neutrino decay seems plausible in this case, it indeed cannot be the source of the peak at lower energy observed by the MiniBooNE collaboration and there should be other mechanisms responsible for the phenomenon.Comment: 14 pages, conclusions are changed; published version for EPJ

Natural IAP inhibitor Embelin enhances therapeutic efficacy of ionizing radiation in prostate cancer

This is the published version, also available here: http://www.ncbi.nlm.nih.gov/pubmed/21804946.Embelin is an active ingredient of traditional herbal medicine that exhibits anti-tumor effects in human prostate cancer cells. However, therapeutic effect of embelin in combination with conventional radiation therapy is not yet determined. In this study, we evaluate the sensitizing potential of embelin on ionizing radiation (IR) in a human prostate cancer model. In vitro, embelin combined with radiation potently suppressed prostate cancer PC-3 cell proliferation that was associated with S and G2/M arrest in cell cycle. Moreover, the combination treatment promoted caspase-independent apoptosis, as evidenced by the increased apoptotic cell death without caspase-3 activation, but not autophagy. Clonogenic survival assay showed that S-phase arrest was required for embelin-mediated radiosensitization. In vivo, embelin significantly improved tumor response to X-ray radiation in the PC-3 xenograft model. Combination therapy produced enhanced tumor growth delay and prolonged time to progression, with minimal systemic toxicity. Immunohistochemistry studies showed that embelin plus IR significantly inhibited cell proliferation, induced apoptosis, and decreased microvessel density in tumors as compared with either treatment alone, suggesting an enhanced combinatory inhibition on tumor suppression and angiogenesis. Our results demonstrate that embelin significantly facilitates tumor suppression by radiation therapy both in vitro and in vivo in the prostate cancer model. This finding warrants embelin as a novel adjuvant therapeutic candidate for the treatment of hormone-refractory prostate cancer that is resistant to radiation therapy