FE-Fusion-VPR: Attention-based Multi-Scale Network Architecture for Visual Place Recognition by Fusing Frames and Events

Traditional visual place recognition (VPR), usually using standard cameras, is easy to fail due to glare or high-speed motion. By contrast, event cameras have the advantages of low latency, high temporal resolution, and high dynamic range, which can deal with the above issues. Nevertheless, event cameras are prone to failure in weakly textured or motionless scenes, while standard cameras can still provide appearance information in this case. Thus, exploiting the complementarity of standard cameras and event cameras can effectively improve the performance of VPR algorithms. In the paper, we propose FE-Fusion-VPR, an attention-based multi-scale network architecture for VPR by fusing frames and events. First, the intensity frame and event volume are fed into the two-stream feature extraction network for shallow feature fusion. Next, the three-scale features are obtained through the multi-scale fusion network and aggregated into three sub-descriptors using the VLAD layer. Finally, the weight of each sub-descriptor is learned through the descriptor re-weighting network to obtain the final refined descriptor. Experimental results show that on the Brisbane-Event-VPR and DDD20 datasets, the Recall@1 of our FE-Fusion-VPR is 29.26% and 33.59% higher than Event-VPR and Ensemble-EventVPR, and is 7.00% and 14.15% higher than MultiRes-NetVLAD and NetVLAD. To our knowledge, this is the first end-to-end network that goes beyond the existing event-based and frame-based SOTA methods to fuse frame and events directly for VPR

Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks

Wang X, Jin Y, Hao K. Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks. IEEE Transactions on Neural Networks and Learning Systems. 2020;31(4):1363-1374.Existing synaptic plasticity rules for optimizing the connections between neurons within the reservoir of echo state networks (ESNs) remain to be global in that the same type of plasticity rule with the same parameters is applied to all neurons. However, this is biologically implausible and practically inflexible for learning the structures in the input signals, thereby limiting the learning performance of ESNs. In this paper, we propose to use local plasticity rules that allow different neurons to use different types of plasticity rules and different parameters, which are achieved by optimizing the parameters of the local plasticity rules using the evolution strategy (ES) with covariance matrix adaptation (CMA-ES). We show that evolving neural plasticity will result in a synergistic learning of different plasticity rules, which plays an important role in improving the learning performance. Meanwhile, we show that the local plasticity rules can effectively alleviate synaptic interferences in learning the structure in sensory inputs. The proposed local plasticity rules are compared with a number of the state-of-the-art ESN models and the canonical ESN using a global plasticity rule on a set of widely used prediction and classification benchmark problems to demonstrate its competitive learning performance

Computational Modeling of Structural Synaptic Plasticity in Echo State Networks

Wang X, Jin Y, Hao K. Computational Modeling of Structural Synaptic Plasticity in Echo State Networks. IEEE Transactions on Cybernetics. 2022;52(10):11254-11266.Most existing studies on computational modeling of neural plasticity have focused on synaptic plasticity. However, regulation of the internal weights in the reservoir based on synaptic plasticity often results in unstable learning dynamics. In this article, a structural synaptic plasticity learning rule is proposed to train the weights and add or remove neurons within the reservoir, which is shown to be able to alleviate the instability of the synaptic plasticity, and to contribute to increase the memory capacity of the network as well. Our experimental results also reveal that a few stronger connections may last for a longer period of time in a constantly changing network structure, and are relatively resistant to decay or disruptions in the learning process. These results are consistent with the evidence observed in biological systems. Finally, we show that an echo state network (ESN) using the proposed structural plasticity rule outperforms an ESN using synaptic plasticity and three state-of-the-art ESNs on four benchmark tasks

Echo state networks regulated by local intrinsic plasticity rules for regression

Wang X, Jin Y, Hao K. Echo state networks regulated by local intrinsic plasticity rules for regression. Neurocomputing. 2019;351:111-122.Intrinsic plasticity, as a biologically inspired unsupervised learning rule, is used for adapting the intrinsic excitability of the reservoir neurons. Existing intrinsic plasticity rules can only select a set of fixed rule parameters for the whole reservoir neurons, which affects the learning performance due to the lack of flexibility in providing intrinsic plasticity. In this paper, we present an echo state network (ESN) with local intrinsic plasticity rule built by different reservoir neurons which can adopt the intrinsic plasticity rule with different rule parameters to adjust its intrinsic excitability. And the covariance matrix adaptation evolution strategy is used to search and select the rule parameters corresponding to different reservoir neurons. Compared with several state-of-the-art ESN models and an ESN with the global plasticity rule, the proposed local intrinsic plasticity rule is able to achieve much better performance in some benchmark prediction tasks

Computational Modeling of Structural Synaptic Plasticity in Echo State Networks

Most existing studies on computational modeling of neural plasticity have focused on synaptic plasticity. However, regulation of the internal weights in the reservoir based on synaptic plasticity often results in unstable learning dynamics. In this article, a structural synaptic plasticity learning rule is proposed to train the weights and add or remove neurons within the reservoir, which is shown to be able to alleviate the instability of the synaptic plasticity, and to contribute to increase the memory capacity of the network as well. Our experimental results also reveal that a few stronger connections may last for a longer period of time in a constantly changing network structure, and are relatively resistant to decay or disruptions in the learning process. These results are consistent with the evidence observed in biological systems. Finally, we show that an echo state network (ESN) using the proposed structural plasticity rule outperforms an ESN using synaptic plasticity and three state-of-the-art ESNs on four benchmark tasks

Synergies between synaptic and intrinsic plasticity in echo state networks

Wang X, Jin Y, Hao K. Synergies between synaptic and intrinsic plasticity in echo state networks. Neurocomputing. 2021;432:32-43.Synaptic plasticity and intrinsic plasticity, as two of the most common neural plasticity mechanisms, occur in all neural circuits throughout life. Neurobiological studies indicated that the interplay between synaptic and intrinsic plasticity contributes to the adaptation of the nervous system to different synaptic input signals. However, most existing computational models of neural plasticity consider these two plasticity mechanisms separately, which is biologically implausible. In this paper, a synergistic plasticity learning rule is proposed to adapt the reservoir connections in echo state networks (ESNs), which not only takes into account the regulation of synaptic weights, but also considers the adjustment of neuronal intrinsic excitability. The proposed synergetic plasticity rule is verified on a number of prediction and classification benchmark problems and our empirical results demonstrate that the ESN with synergistic plasticity learning rule performs much better than the state-of-the-art ESN models, and an ESN with a single neural plasticity rule

Organizational Geosocial Network: A Graph Machine Learning Approach Integrating Geographic and Public Policy Information for Studying the Development of Social Organizations in China

This study aims to give an insight into the development trends and patterns of social organizations (SOs) in China from the perspective of network science integrating geography and public policy information embedded in the network structure. Firstly, we constructed a first-of-its-kind database which encompasses almost all social organizations established in China throughout the past decade. Secondly, we proposed four basic structures to represent the homogeneous and heterogeneous networks between social organizations and related social entities, such as government administrations and community members. Then, we pioneered the application of graph models to the field of organizations and embedded the Organizational Geosocial Network (OGN) into a low-dimensional representation of the social entities and relations while preserving their semantic meaning. Finally, we applied advanced graph deep learning methods, such as graph attention networks (GAT) and graph convolutional networks (GCN), to perform exploratory classification tasks by training models with county-level OGNs dataset and make predictions of which geographic region the county-level OGN belongs to. The experiment proves that different regions possess a variety of development patterns and economic structures where local social organizations are embedded, thus forming differential OGN structures, which can be sensed by graph machine learning algorithms and make relatively accurate predictions. To the best of our knowledge, this is the first application of graph deep learning to the construction and representation learning of geosocial network models of social organizations, which has certain reference significance for research in related fields

A Gated Recurrent Unit based Echo State Network

Wang X, Jin Y, Hao K. A Gated Recurrent Unit based Echo State Network. In: 2020 International Joint Conference on Neural Networks (IJCNN). IEEE; 2020: 1-7

Outage performance of a cognitive amplify-and-forward relay network with primary user interference over Nakagami-m fading channels

In this study, primary user (PU) interference in a cognitive amplify-and-forward relay network is considering over a Nakagami-m fading channel, and the outage probability is derived for the secondary users. This PU interference has not previously been considered in the literature. Results are presented which show that PU interference can significantly degrade secondary user performance, and thus should be considered in the design of a cognitive radio network