Cross-Inferential Networks for Source-free Unsupervised Domain Adaptation

One central challenge in source-free unsupervised domain adaptation (UDA) is the lack of an effective approach to evaluate the prediction results of the adapted network model in the target domain. To address this challenge, we propose to explore a new method called cross-inferential networks (CIN). Our main idea is that, when we adapt the network model to predict the sample labels from encoded features, we use these prediction results to construct new training samples with derived labels to learn a new examiner network that performs a different but compatible task in the target domain. Specifically, in this work, the base network model is performing image classification while the examiner network is tasked to perform relative ordering of triplets of samples whose training labels are carefully constructed from the prediction results of the base network model. Two similarity measures, cross-network correlation matrix similarity and attention consistency, are then developed to provide important guidance for the UDA process. Our experimental results on benchmark datasets demonstrate that our proposed CIN approach can significantly improve the performance of source-free UDA.Comment: ICIP2023 accepte

Short-Term Plasticity Neurons Learning to Learn and Forget

Short-term plasticity (STP) is a mechanism that stores decaying memories in synapses of the cerebral cortex. In computing practice, STP has been used, but mostly in the niche of spiking neurons, even though theory predicts that it is the optimal solution to certain dynamic tasks. Here we present a new type of recurrent neural unit, the STP Neuron (STPN), which indeed turns out strikingly powerful. Its key mechanism is that synapses have a state, propagated through time by a self-recurrent connection-within-the-synapse. This formulation enables training the plasticity with backpropagation through time, resulting in a form of learning to learn and forget in the short term. The STPN outperforms all tested alternatives, i.e. RNNs, LSTMs, other models with fast weights, and differentiable plasticity. We confirm this in both supervised and reinforcement learning (RL), and in tasks such as Associative Retrieval, Maze Exploration, Atari video games, and MuJoCo robotics. Moreover, we calculate that, in neuromorphic or biological circuits, the STPN minimizes energy consumption across models, as it depresses individual synapses dynamically. Based on these, biological STP may have been a strong evolutionary attractor that maximizes both efficiency and computational power. The STPN now brings these neuromorphic advantages also to a broad spectrum of machine learning practice. Code is available at https://github.com/NeuromorphicComputing/stpnComment: Accepted at ICML 202

Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network

Leveraging the low-power, event-driven computation and the inherent temporal dynamics, spiking neural networks (SNNs) are potentially ideal solutions for processing dynamic and asynchronous signals from event-based sensors. However, due to the challenges in training and the restrictions in architectural design, there are limited examples of competitive SNNs in the realm of event-based dense prediction when compared to artificial neural networks (ANNs). In this paper, we present an efficient spiking encoder-decoder network designed for large-scale event-based semantic segmentation tasks. This is achieved by optimizing the encoder using a hierarchical search method. To enhance learning from dynamic event streams, we harness the inherent adaptive threshold of spiking neurons to modulate network activation. Moreover, we introduce a dual-path Spiking Spatially-Adaptive Modulation (SSAM) block, specifically designed to enhance the representation of sparse events, thereby considerably improving network performance. Our proposed network achieves a 72.57% mean intersection over union (MIoU) on the DDD17 dataset and a 57.22% MIoU on the recently introduced, larger DSEC-Semantic dataset. This performance surpasses the current state-of-the-art ANNs by 4%, whilst consuming significantly less computational resources. To the best of our knowledge, this is the first study demonstrating SNNs outperforming ANNs in demanding event-based semantic segmentation tasks, thereby establishing the vast potential of SNNs in the field of event-based vision. Our source code will be made publicly accessible

Automotive Object Detection via Learning Sparse Events by Temporal Dynamics of Spiking Neurons

Event-based sensors, with their high temporal resolution (1us) and dynamical range (120dB), have the potential to be deployed in high-speed platforms such as vehicles and drones. However, the highly sparse and fluctuating nature of events poses challenges for conventional object detection techniques based on Artificial Neural Networks (ANNs). In contrast, Spiking Neural Networks (SNNs) are well-suited for representing event-based data due to their inherent temporal dynamics. In particular, we demonstrate that the membrane potential dynamics can modulate network activity upon fluctuating events and strengthen features of sparse input. In addition, the spike-triggered adaptive threshold can stabilize training which further improves network performance. Based on this, we develop an efficient spiking feature pyramid network for event-based object detection. Our proposed SNN outperforms previous SNNs and sophisticated ANNs with attention mechanisms, achieving a mean average precision (map50) of 47.7% on the Gen1 benchmark dataset. This result significantly surpasses the previous best SNN by 9.7% and demonstrates the potential of SNNs for event-based vision. Our model has a concise architecture while maintaining high accuracy and much lower computation cost as a result of sparse computation. Our code will be publicly available

Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation

Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. We take inspiration from the biological plausibility learning where the neuron responses are tuned based on a local synapse-change procedure and activated by competitive lateral inhibition rules. Based on these feed-forward learning rules, we design a soft Hebbian learning process which provides an unsupervised and effective mechanism for online adaptation. We observe that the performance of this feed-forward Hebbian learning for fully test-time adaptation can be significantly improved by incorporating a feedback neuro-modulation layer. It is able to fine-tune the neuron responses based on the external feedback generated by the error back-propagation from the top inference layers. This leads to our proposed neuro-modulated Hebbian learning (NHL) method for fully test-time adaptation. With the unsupervised feed-forward soft Hebbian learning being combined with a learned neuro-modulator to capture feedback from external responses, the source model can be effectively adapted during the testing process. Experimental results on benchmark datasets demonstrate that our proposed method can significantly improve the adaptation performance of network models and outperforms existing state-of-the-art methods.Comment: CVPR2023 accepte

Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling

Weakly-supervised action localization aims to recognize and localize action instancese in untrimmed videos with only video-level labels. Most existing models rely on multiple instance learning(MIL), where the predictions of unlabeled instances are supervised by classifying labeled bags. The MIL-based methods are relatively well studied with cogent performance achieved on classification but not on localization. Generally, they locate temporal regions by the video-level classification but overlook the temporal variations of feature semantics. To address this problem, we propose a novel attention-based hierarchically-structured latent model to learn the temporal variations of feature semantics. Specifically, our model entails two components, the first is an unsupervised change-points detection module that detects change-points by learning the latent representations of video features in a temporal hierarchy based on their rates of change, and the second is an attention-based classification model that selects the change-points of the foreground as the boundaries. To evaluate the effectiveness of our model, we conduct extensive experiments on two benchmark datasets, THUMOS-14 and ActivityNet-v1.3. The experiments show that our method outperforms current state-of-the-art methods, and even achieves comparable performance with fully-supervised methods.Comment: Accepted to ICCV 2023. arXiv admin note: text overlap with arXiv:2203.15187, arXiv:2003.12424, arXiv:2104.02967 by other author

Defining key metabolic roles in osmotic adjustment and ROS homeostasis in the recretohalophyte Karelinia caspia under salt stress

The recretohalophyte Karelinia caspia is of forage and medical value and can remediate saline soils. We here assess the contribution of primary/secondary metabolism to osmotic adjustment and ROS homeostasis in Karelinia caspia under salt stress using multi‐omic approaches. Computerized phenomic assessments, tests for cellular osmotic changes and lipid peroxidation indicated that salt treatment had no detectable physical effect on K. caspia. Metabolomic analysis indicated that amino acids, saccharides, organic acids, polyamine, phenolic acids, and vitamins accumulated significantly with salt treatment. Transcriptomic assessment identified differentially expressed genes closely linked to the changes in above primary/secondary metabolites under salt stress. In particular, shifts in carbohydrate metabolism (TCA cycle, starch and sucrose metabolism, glycolysis) as well as arginine and proline metabolism were observed to maintain a low osmotic potential. Chlorogenic acid/vitamin E biosynthesis was also enhanced, which would aid in ROS scavenging in the response of K. caspia to salt. Overall, our findings define key changes in primary/secondary metabolism that are coordinated to modulate the osmotic balance and ROS homeostasis to contribute to the salt tolerance of K. caspia

ROS scavenging and ion homeostasis is required for the adaptation of halophyte Karelinia caspia to high salinity

The halophyte Karelinia caspia has not only fodder and medical value but also can remediate saline-alkali soils. Our previous study showed that salt-secreting by salt glands is one of main adaptive strategies of K. caspia under high salinity. However, ROS scavenging, ion homeostasis, and photosynthetic characteristics responses to high salinity remain unclear in K. caspia. Here, physio-biochemical responses and gene expression associated with ROS scavenging and ions transport were tested in K. caspia subjected to 100–400 mM NaCl for 7 days. Results showed that both antioxidant enzymes (SOD, APX) activities and non-enzymatic antioxidants (chlorogenic acid, α-tocopherol, flavonoids, polyamines) contents were significantly enhanced, accompanied by up-regulating the related enzyme and non-enzymatic antioxidant synthesis gene (KcCu/Zn-SOD, KcAPX6, KcHCT, KcHPT1, Kcγ-TMT, KcF3H, KcSAMS and KcSMS) expression with increasing concentrations of NaCl. These responses are beneficial for removing excess ROS to maintain a stable level of H(2)O(2) and O(2)(−) without lipid peroxidation in the K. caspia response to high salt. Meanwhile, up-regulating expression of KcSOS1/2/3, KcNHX1, and KcAVP was linked to Na(+) compartmentalization into vacuoles or excretion through salt glands in K. caspia. Notably, salt can improve the function of PSII that facilitate net photosynthetic rates, which is helpful to growing normally in high saline. Overall, the findings suggested that ROS scavenging systems and Na(+)/K(+) transport synergistically contributed to redox equilibrium, ion homeostasis, and the enhancement of PSII function, thereby conferring high salt tolerance

Structure, morphology and magnetic properties of flowerlike gamma-Fe2O3@NiO core/shell nanocomposites synthesized from different precursor concentrations

The flowerlike gamma-Fe2O3@NiO core/shell nanocomposites are synthesized by the two-step method. Their structure and morphology can be controlled by tuning the precursor concentration. Microstructural analysis reveals that all the samples have distinct core/shell structure without impurities, and the NiO shells are built of many irregular nanosheets which enclose the surface of gamma-Fe2O3 core. As the precursor concentration decreases (i.e., more NiO content), the NiO grain grows significantly, and the thickness of NiO shells increases. Magnetic experiments are performed to analyze the influences of different microstructures on magnetic properties of samples and we have the following two results. First, at 5 K, along with increasing thickness of NiO shell, the saturation magnetization increases, while the residual magnetization decreases slightly. Second, the hysteresis loops under cooling field demonstrate that the value of exchange bias effect fluctuates between 13 Oe and 17 Oe. This is mainly because of the NiO shell that (i) is composed of irregular nanosheets with disordered orientations, and (ii) does not form a complete coating around gamma-Fe2O3 core

Attentive Decision-making and Dynamic Resetting of Continual Running SRNNs for End-to-End Streaming Keyword Spotting

Efficient end-to-end processing of continuous and streaming signals is one of the key challenges for Artificial Intelligence (AI) in particular for Edge applications that are energy-constrained. Spiking neural networks are explored to achieve efficient edge AI, employing low-latency, sparse processing, and small network size resulting in low-energy operation. Spiking Recurrent Neural Networks (SRNNs) achieve good performance on sample data at excellent network size and energy. When applied to continual streaming data, like a series of concatenated keyword samples, SRNNs, like traditional RNNs, recognize successive information increasingly poorly as the network dynamics become saturated. SRNNs process concatenated streams of data in three steps: i) Relevant signals have to be localized. ii) Evidence then needs to be integrated to classify the signal, and finally, iii) the neural dynamics must be combined with network state resetting events to remedy network saturation. Here we show how a streaming form of attention can aid SRNNs in localizing events in a continuous stream of signals, where a brain-inspired decision-making circuit then integrates evidence to determine the correct classification. This decision then leads to a delayed network reset, remedying network state saturation. We demonstrate the effectiveness of this approach on streams of concatenated keywords, reporting high accuracy combined with low average network activity as the attention signal effectively gates network activity in the absence of signals. We also show that the dynamic normalization effected by the attention mechanism enables a degree of environmental transfer learning, where the same keywords obtained in different circumstances are still correctly classified. The principles presented here also carry over to similar applications of classical RNNs and thus may be of general interest for continual running applications.</p