Easy attention: A simple self-attention mechanism for Transformers

To improve the robustness of transformer neural networks used for temporal-dynamics prediction of chaotic systems, we propose a novel attention mechanism called easy attention. Due to the fact that self attention only makes usage of the inner product of queries and keys, it is demonstrated that the keys, queries and softmax are not necessary for obtaining the attention score required to capture long-term dependencies in temporal sequences. Through implementing singular-value decomposition (SVD) on the softmax attention score, we further observe that the self attention compresses contribution from both queries and keys in the spanned space of the attention score. Therefore, our proposed easy-attention method directly treats the attention scores as learnable parameters. This approach produces excellent results when reconstructing and predicting the temporal dynamics of chaotic systems exhibiting more robustness and less complexity than the self attention or the widely-used long short-term memory (LSTM) network. Our results show great potential for applications in more complex high-dimensional dynamical systems.Comment: 12 pages and 8 figure

Chimera states: Coexistence of coherence and incoherence in networks of coupled oscillators

A chimera state is a spatio-temporal pattern in a network of identical coupled oscillators in which synchronous and asynchronous oscillation coexist. This state of broken symmetry, which usually coexists with a stable spatially symmetric state, has intrigued the nonlinear dynamics community since its discovery in the early 2000s. Recent experiments have led to increasing interest in the origin and dynamics of these states. Here we review the history of research on chimera states and highlight major advances in understanding their behaviour.Comment: 26 pages, 3 figure

A Compact CMOS Memristor Emulator Circuit and its Applications

Conceptual memristors have recently gathered wider interest due to their diverse application in non-von Neumann computing, machine learning, neuromorphic computing, and chaotic circuits. We introduce a compact CMOS circuit that emulates idealized memristor characteristics and can bridge the gap between concepts to chip-scale realization by transcending device challenges. The CMOS memristor circuit embodies a two-terminal variable resistor whose resistance is controlled by the voltage applied across its terminals. The memristor 'state' is held in a capacitor that controls the resistor value. This work presents the design and simulation of the memristor emulation circuit, and applies it to a memcomputing application of maze solving using analog parallelism. Furthermore, the memristor emulator circuit can be designed and fabricated using standard commercial CMOS technologies and opens doors to interesting applications in neuromorphic and machine learning circuits.Comment: Submitted to International Symposium of Circuits and Systems (ISCAS) 201

A chaotic spread spectrum system for underwater acoustic communication

The work is supported in part by NSFC (Grant no. 61172070), IRT of Shaanxi Province (2013KCT-04), EPSRC (Grant no.Ep/1032606/1).Peer reviewedPostprin

Wave-based extreme deep learning based on non-linear time-Floquet entanglement

Wave-based analog signal processing holds the promise of extremely fast, on-the-fly, power-efficient data processing, occurring as a wave propagates through an artificially engineered medium. Yet, due to the fundamentally weak non-linearities of traditional wave materials, such analog processors have been so far largely confined to simple linear projections such as image edge detection or matrix multiplications. Complex neuromorphic computing tasks, which inherently require strong non-linearities, have so far remained out-of-reach of wave-based solutions, with a few attempts that implemented non-linearities on the digital front, or used weak and inflexible non-linear sensors, restraining the learning performance. Here, we tackle this issue by demonstrating the relevance of Time-Floquet physics to induce a strong non-linear entanglement between signal inputs at different frequencies, enabling a power-efficient and versatile wave platform for analog extreme deep learning involving a single, uniformly modulated dielectric layer and a scattering medium. We prove the efficiency of the method for extreme learning machines and reservoir computing to solve a range of challenging learning tasks, from forecasting chaotic time series to the simultaneous classification of distinct datasets. Our results open the way for wave-based machine learning with high energy efficiency, speed, and scalability.Comment: 23 pages, 9 figure

Spontaneous and stimulus-induced coherent states of critically balanced neuronal networks

How the information microscopically processed by individual neurons is integrated and used in organizing the behavior of an animal is a central question in neuroscience. The coherence of neuronal dynamics over different scales has been suggested as a clue to the mechanisms underlying this integration. Balanced excitation and inhibition may amplify microscopic fluctuations to a macroscopic level, thus providing a mechanism for generating coherent multiscale dynamics. Previous theories of brain dynamics, however, were restricted to cases in which inhibition dominated excitation and suppressed fluctuations in the macroscopic population activity. In the present study, we investigate the dynamics of neuronal networks at a critical point between excitation-dominant and inhibition-dominant states. In these networks, the microscopic fluctuations are amplified by the strong excitation and inhibition to drive the macroscopic dynamics, while the macroscopic dynamics determine the statistics of the microscopic fluctuations. Developing a novel type of mean-field theory applicable to this class of interscale interactions, we show that the amplification mechanism generates spontaneous, irregular macroscopic rhythms similar to those observed in the brain. Through the same mechanism, microscopic inputs to a small number of neurons effectively entrain the dynamics of the whole network. These network dynamics undergo a probabilistic transition to a coherent state, as the magnitude of either the balanced excitation and inhibition or the external inputs is increased. Our mean-field theory successfully predicts the behavior of this model. Furthermore, we numerically demonstrate that the coherent dynamics can be used for state-dependent read-out of information from the network. These results show a novel form of neuronal information processing that connects neuronal dynamics on different scales.Comment: 20 pages 12 figures (main text) + 23 pages 6 figures (Appendix); Some of the results have been removed in the revision in order to reduce the volume. See the previous version for more result

Robust chimera states in SQUID metamaterials with local interactions

We report on the emergence of robust multi-clustered chimera states in a dissipative-driven system of symmetrically and locally coupled identical SQUID oscillators. The "snake-like" resonance curve of the single SQUID (Superconducting QUantum Interference Device) is the key to the formation of the chimera states and is responsible for the extreme multistability exhibited by the coupled system that leads to attractor crowding at the geometrical resonance (inductive-capacitive) frequency. Until now, chimera states were mostly believed to exist for nonlocal coupling. Our findings provide theoretical evidence that nearest neighbor interactions are indeed capable of supporting such states in a wide parameter range. SQUID metamaterials are the subject of intense experimental investigations and we are highly confident that the complex dynamics demonstrated in this manuscript can be confirmed in the laboratory