Cross-Inferential Networks for Source-free Unsupervised Domain Adaptation

One central challenge in source-free unsupervised domain adaptation (UDA) is the lack of an effective approach to evaluate the prediction results of the adapted network model in the target domain. To address this challenge, we propose to explore a new method called cross-inferential networks (CIN). Our main idea is that, when we adapt the network model to predict the sample labels from encoded features, we use these prediction results to construct new training samples with derived labels to learn a new examiner network that performs a different but compatible task in the target domain. Specifically, in this work, the base network model is performing image classification while the examiner network is tasked to perform relative ordering of triplets of samples whose training labels are carefully constructed from the prediction results of the base network model. Two similarity measures, cross-network correlation matrix similarity and attention consistency, are then developed to provide important guidance for the UDA process. Our experimental results on benchmark datasets demonstrate that our proposed CIN approach can significantly improve the performance of source-free UDA.Comment: ICIP2023 accepte

Sample as You Infer: Predictive Coding With Langevin Dynamics

We present a novel algorithm for parameter learning in generic deep generative models that builds upon the predictive coding (PC) framework of computational neuroscience. Our approach modifies the standard PC algorithm to bring performance on-par and exceeding that obtained from standard variational auto-encoder (VAE) training. By injecting Gaussian noise into the PC inference procedure we re-envision it as an overdamped Langevin sampling, which facilitates optimisation with respect to a tight evidence lower bound (ELBO). We improve the resultant encoder-free training method by incorporating an encoder network to provide an amortised warm-start to our Langevin sampling and test three different objectives for doing so. Finally, to increase robustness to the sampling step size and reduce sensitivity to curvature, we validate a lightweight and easily computable form of preconditioning, inspired by Riemann Manifold Langevin and adaptive optimizers from the SGD literature. We compare against VAEs by training like-for-like generative models using our technique against those trained with standard reparameterisation-trick-based ELBOs. We observe our method out-performs or matches performance across a number of metrics, including sample quality, while converging in a fraction of the number of SGD training iterations.Comment: FID values updated to use a fixed 50,000 samples for all experiments - Jeffrey's divergence now consistently best performing. Dynov2 based metrics removed due to inconsistency of results - and since not industry standard. Multiple beta values tested in Fig 4. Theta LR for VAEs; beta and inf LR for LPC now tuned for results. Figure 5B updated; curves now correspond to results in Table

Short-Term Plasticity Neurons Learning to Learn and Forget

Short-term plasticity (STP) is a mechanism that stores decaying memories in synapses of the cerebral cortex. In computing practice, STP has been used, but mostly in the niche of spiking neurons, even though theory predicts that it is the optimal solution to certain dynamic tasks. Here we present a new type of recurrent neural unit, the STP Neuron (STPN), which indeed turns out strikingly powerful. Its key mechanism is that synapses have a state, propagated through time by a self-recurrent connection-within-the-synapse. This formulation enables training the plasticity with backpropagation through time, resulting in a form of learning to learn and forget in the short term. The STPN outperforms all tested alternatives, i.e. RNNs, LSTMs, other models with fast weights, and differentiable plasticity. We confirm this in both supervised and reinforcement learning (RL), and in tasks such as Associative Retrieval, Maze Exploration, Atari video games, and MuJoCo robotics. Moreover, we calculate that, in neuromorphic or biological circuits, the STPN minimizes energy consumption across models, as it depresses individual synapses dynamically. Based on these, biological STP may have been a strong evolutionary attractor that maximizes both efficiency and computational power. The STPN now brings these neuromorphic advantages also to a broad spectrum of machine learning practice. Code is available at https://github.com/NeuromorphicComputing/stpnComment: Accepted at ICML 202

When in Doubt, Think Slow: Iterative Reasoning with Latent Imagination

In an unfamiliar setting, a model-based reinforcement learning agent can be limited by the accuracy of its world model. In this work, we present a novel, training-free approach to improving the performance of such agents separately from planning and learning. We do so by applying iterative inference at decision-time, to fine-tune the inferred agent states based on the coherence of future state representations. Our approach achieves a consistent improvement in both reconstruction accuracy and task performance when applied to visual 3D navigation tasks. We go on to show that considering more future states further improves the performance of the agent in partially-observable environments, but not in a fully-observable one. Finally, we demonstrate that agents with less training pre-evaluation benefit most from our approach

Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network

Leveraging the low-power, event-driven computation and the inherent temporal dynamics, spiking neural networks (SNNs) are potentially ideal solutions for processing dynamic and asynchronous signals from event-based sensors. However, due to the challenges in training and the restrictions in architectural design, there are limited examples of competitive SNNs in the realm of event-based dense prediction when compared to artificial neural networks (ANNs). In this paper, we present an efficient spiking encoder-decoder network designed for large-scale event-based semantic segmentation tasks. This is achieved by optimizing the encoder using a hierarchical search method. To enhance learning from dynamic event streams, we harness the inherent adaptive threshold of spiking neurons to modulate network activation. Moreover, we introduce a dual-path Spiking Spatially-Adaptive Modulation (SSAM) block, specifically designed to enhance the representation of sparse events, thereby considerably improving network performance. Our proposed network achieves a 72.57% mean intersection over union (MIoU) on the DDD17 dataset and a 57.22% MIoU on the recently introduced, larger DSEC-Semantic dataset. This performance surpasses the current state-of-the-art ANNs by 4%, whilst consuming significantly less computational resources. To the best of our knowledge, this is the first study demonstrating SNNs outperforming ANNs in demanding event-based semantic segmentation tasks, thereby establishing the vast potential of SNNs in the field of event-based vision. Our source code will be made publicly accessible

Automotive Object Detection via Learning Sparse Events by Temporal Dynamics of Spiking Neurons

Event-based sensors, with their high temporal resolution (1us) and dynamical range (120dB), have the potential to be deployed in high-speed platforms such as vehicles and drones. However, the highly sparse and fluctuating nature of events poses challenges for conventional object detection techniques based on Artificial Neural Networks (ANNs). In contrast, Spiking Neural Networks (SNNs) are well-suited for representing event-based data due to their inherent temporal dynamics. In particular, we demonstrate that the membrane potential dynamics can modulate network activity upon fluctuating events and strengthen features of sparse input. In addition, the spike-triggered adaptive threshold can stabilize training which further improves network performance. Based on this, we develop an efficient spiking feature pyramid network for event-based object detection. Our proposed SNN outperforms previous SNNs and sophisticated ANNs with attention mechanisms, achieving a mean average precision (map50) of 47.7% on the Gen1 benchmark dataset. This result significantly surpasses the previous best SNN by 9.7% and demonstrates the potential of SNNs for event-based vision. Our model has a concise architecture while maintaining high accuracy and much lower computation cost as a result of sparse computation. Our code will be publicly available

Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation

Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. We take inspiration from the biological plausibility learning where the neuron responses are tuned based on a local synapse-change procedure and activated by competitive lateral inhibition rules. Based on these feed-forward learning rules, we design a soft Hebbian learning process which provides an unsupervised and effective mechanism for online adaptation. We observe that the performance of this feed-forward Hebbian learning for fully test-time adaptation can be significantly improved by incorporating a feedback neuro-modulation layer. It is able to fine-tune the neuron responses based on the external feedback generated by the error back-propagation from the top inference layers. This leads to our proposed neuro-modulated Hebbian learning (NHL) method for fully test-time adaptation. With the unsupervised feed-forward soft Hebbian learning being combined with a learned neuro-modulator to capture feedback from external responses, the source model can be effectively adapted during the testing process. Experimental results on benchmark datasets demonstrate that our proposed method can significantly improve the adaptation performance of network models and outperforms existing state-of-the-art methods.Comment: CVPR2023 accepte

Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling

Weakly-supervised action localization aims to recognize and localize action instancese in untrimmed videos with only video-level labels. Most existing models rely on multiple instance learning(MIL), where the predictions of unlabeled instances are supervised by classifying labeled bags. The MIL-based methods are relatively well studied with cogent performance achieved on classification but not on localization. Generally, they locate temporal regions by the video-level classification but overlook the temporal variations of feature semantics. To address this problem, we propose a novel attention-based hierarchically-structured latent model to learn the temporal variations of feature semantics. Specifically, our model entails two components, the first is an unsupervised change-points detection module that detects change-points by learning the latent representations of video features in a temporal hierarchy based on their rates of change, and the second is an attention-based classification model that selects the change-points of the foreground as the boundaries. To evaluate the effectiveness of our model, we conduct extensive experiments on two benchmark datasets, THUMOS-14 and ActivityNet-v1.3. The experiments show that our method outperforms current state-of-the-art methods, and even achieves comparable performance with fully-supervised methods.Comment: Accepted to ICCV 2023. arXiv admin note: text overlap with arXiv:2203.15187, arXiv:2003.12424, arXiv:2104.02967 by other author

Defining key metabolic roles in osmotic adjustment and ROS homeostasis in the recretohalophyte Karelinia caspia under salt stress

The recretohalophyte Karelinia caspia is of forage and medical value and can remediate saline soils. We here assess the contribution of primary/secondary metabolism to osmotic adjustment and ROS homeostasis in Karelinia caspia under salt stress using multi‐omic approaches. Computerized phenomic assessments, tests for cellular osmotic changes and lipid peroxidation indicated that salt treatment had no detectable physical effect on K. caspia. Metabolomic analysis indicated that amino acids, saccharides, organic acids, polyamine, phenolic acids, and vitamins accumulated significantly with salt treatment. Transcriptomic assessment identified differentially expressed genes closely linked to the changes in above primary/secondary metabolites under salt stress. In particular, shifts in carbohydrate metabolism (TCA cycle, starch and sucrose metabolism, glycolysis) as well as arginine and proline metabolism were observed to maintain a low osmotic potential. Chlorogenic acid/vitamin E biosynthesis was also enhanced, which would aid in ROS scavenging in the response of K. caspia to salt. Overall, our findings define key changes in primary/secondary metabolism that are coordinated to modulate the osmotic balance and ROS homeostasis to contribute to the salt tolerance of K. caspia

ROS scavenging and ion homeostasis is required for the adaptation of halophyte Karelinia caspia to high salinity

The halophyte Karelinia caspia has not only fodder and medical value but also can remediate saline-alkali soils. Our previous study showed that salt-secreting by salt glands is one of main adaptive strategies of K. caspia under high salinity. However, ROS scavenging, ion homeostasis, and photosynthetic characteristics responses to high salinity remain unclear in K. caspia. Here, physio-biochemical responses and gene expression associated with ROS scavenging and ions transport were tested in K. caspia subjected to 100–400 mM NaCl for 7 days. Results showed that both antioxidant enzymes (SOD, APX) activities and non-enzymatic antioxidants (chlorogenic acid, α-tocopherol, flavonoids, polyamines) contents were significantly enhanced, accompanied by up-regulating the related enzyme and non-enzymatic antioxidant synthesis gene (KcCu/Zn-SOD, KcAPX6, KcHCT, KcHPT1, Kcγ-TMT, KcF3H, KcSAMS and KcSMS) expression with increasing concentrations of NaCl. These responses are beneficial for removing excess ROS to maintain a stable level of H(2)O(2) and O(2)(−) without lipid peroxidation in the K. caspia response to high salt. Meanwhile, up-regulating expression of KcSOS1/2/3, KcNHX1, and KcAVP was linked to Na(+) compartmentalization into vacuoles or excretion through salt glands in K. caspia. Notably, salt can improve the function of PSII that facilitate net photosynthetic rates, which is helpful to growing normally in high saline. Overall, the findings suggested that ROS scavenging systems and Na(+)/K(+) transport synergistically contributed to redox equilibrium, ion homeostasis, and the enhancement of PSII function, thereby conferring high salt tolerance