A Neural ODE Interpretation of Transformer Layers

Transformer layers, which use an alternating pattern of multi-head attention and multi-layer perceptron (MLP) layers, provide an effective tool for a variety of machine learning problems. As the transformer layers use residual connections to avoid the problem of vanishing gradients, they can be viewed as the numerical integration of a differential equation. In this extended abstract, we build upon this connection and propose a modification of the internal architecture of a transformer layer. The proposed model places the multi-head attention sublayer and the MLP sublayer parallel to each other. Our experiments show that this simple modification improves the performance of transformer networks in multiple tasks. Moreover, for the image classification task, we show that using neural ODE solvers with a sophisticated integration scheme further improves performance

Specific instrumentation and diagnostics for high-intensity hadron beams

An overview of various typical instruments used for high-intensity hadron beams is given. In addition, a few important diagnostic methods are discussed which are quite special for these kinds of beams.Comment: 58 pages, contribution to the CAS - CERN Accelerator School: Course on High Power Hadron Machines; 24 May - 2 Jun 2011, Bilbao, Spai

Transformer Network for Multi-Person Tracking and Re-Identification in Unconstrained Environment

Multi-object tracking (MOT) has profound applications in a variety of fields, including surveillance, sports analytics, self-driving, and cooperative robotics. Despite considerable advancements, existing MOT methodologies tend to falter when faced with non-uniform movements, occlusions, and appearance-reappearance scenarios of the objects. Recognizing this inadequacy, we put forward an integrated MOT method that not only marries object detection and identity linkage within a singular, end-to-end trainable framework but also equips the model with the ability to maintain object identity links over long periods of time. Our proposed model, named STMMOT, is built around four key modules: 1) candidate proposal generation, which generates object proposals via a vision-transformer encoder-decoder architecture that detects the object from each frame in the video; 2) scale variant pyramid, a progressive pyramid structure to learn the self-scale and cross-scale similarities in multi-scale feature maps; 3) spatio-temporal memory encoder, extracting the essential information from the memory associated with each object under tracking; and 4) spatio-temporal memory decoder, simultaneously resolving the tasks of object detection and identity association for MOT. Our system leverages a robust spatio-temporal memory module that retains extensive historical observations and effectively encodes them using an attention-based aggregator. The uniqueness of STMMOT lies in representing objects as dynamic query embeddings that are updated continuously, which enables the prediction of object states with attention mechanisms and eradicates the need for post-processing

Flux Pumping for No-Insulation High Temperature Superconducting REBCO Magnets

The high temperature superconducting (HTS) REBCO magnet has many advantages compared to the low temperature superconducting (LTS) magnet, including high operating temperature, high critical current under high magnetic fields, and strong mechanical properties. Due to these excellent properties, HTS REBCO magnets have huge potential in high-field high-current applications, such as accelerator magnets, tokamak fusion magnets, magnetic resonance imaging (MRI) systems, ultra-high-field magnets. It can pave the way for a more compact and economical superconducting magnet system, including. Currently, a novel no-insulation (NI) type HTS REBCO magnet has a self-protection ability, which has solved a long-lasting “thermal quench” problem. The self-protection ability has significantly improved the reliability of HTS REBCO magnets and makes the NI HTS REBCO magnet very advantageous in practical use. However, lossless superconducting joints for HTS REBCO coated conductors are not available. To maintain a stable magnetic field for the HTS REBCO magnet, an external power supply is required via a bulky no-superconducting copper current leads. The copper current leads consume a huge amount of electricity and increase the size of the magnet system. In order to reduce energy loss and the size of the magnet system, flux pumping technology was invented, which is a contactless charging technology and removes the no-superconducting resistive current leads. Up to now, however, flux pumping technology only focuses on conventional insulated type HTS REBCO magnets. The novel NI HTS REBCO magnet has different characteristics from the conventional insulated HTS REBCO magnet, such as bypass current and characteristic resistance (Rc). This thesis studies the flux pumping technology for the no-insulation type HTS REBCO magnet, figures out the technical challenges, and improves the flux pumping performance for NI HTS REBCO magnet with a very low Rc. Chapter 2 introduces the basic theory of superconductivity, the development of superconductors, and various superconducting applications. Chapter 3 reviews the flux pumping technology, including LTS and HTS flux pumps. Chapter 4 presents an active-switching HTS transformer rectifier flux pump (TRFP), including the fundamental physics, key components, and overall system. Chapter 5 presents the design and fabrication of the NI HTS REBCO magnet and proposed a novel solder impregnated NI HTS REBCO magnet. The characteristics of this novel NI HTS magnet are discussed. Chapter 6 studies the flux pumping performance of the HTS TRFP for different NI HTS REBCO magnets, analyses the unique flux pumping characteristics, and improves the flux pumping performance for NI REBCO magnet with a very low Rc. Chapter 7 designs high-performance switches for active-switching HTS TRFP via a multi-physics FEM COMSOL model. This thesis will help promote the use of the flux pumping technology for the NI HTS REBCO magnets systems, and it will be of interest to physicists and engineers who want to build an energy-efficient, compact, and reliable superconducting magnet system.Cambridge Trust-CSC International Scholarshi

A new hybrid algorithm for multi‐objective reactive power planning via facts devices and renewable wind resources

The power system planning problem considering system loss function, voltage profile function, the cost function of FACTS (flexible alternating current transmission system) devices, and stability function are investigated in this paper. With the growth of electronic technologies, FACTS devices have improved stability and more reliable planning in reactive power (RP) planning. In addition, in modern power systems, renewable resources have an inevitable effect on power system planning. Therefore, wind resources make a complicated problem of planning due to conflicting functions and non-linear constraints. This confliction is the stochastic nature of the cost, loss, and voltage functions that cannot be summarized in function. A multi-objective hybrid algorithm is proposed to solve this problem by considering the linear and non-linear constraints that combine particle swarm optimization (PSO) and the virus colony search (VCS). VCS is a new optimization method based on viruses’ search function to destroy host cells and cause the penetration of the best virus into a cell for reproduction. In the proposed model, the PSO is used to enhance local and global search. In addition, the non-dominated sort of the Pareto criterion is used to sort the data. The optimization results on different scenarios reveal that the combined method of the proposed hybrid algorithm can improve the parameters such as convergence time, index of voltage stability, and absolute magnitude of voltage deviation, and this method can reduce the total transmission line losses. In addition, the presence of wind resources has a positive effect on the mentioned issue

EENED: End-to-End Neural Epilepsy Detection based on Convolutional Transformer

Recently Transformer and Convolution neural network (CNN) based models have shown promising results in EEG signal processing. Transformer models can capture the global dependencies in EEG signals through a self-attention mechanism, while CNN models can capture local features such as sawtooth waves. In this work, we propose an end-to-end neural epilepsy detection model, EENED, that combines CNN and Transformer. Specifically, by introducing the convolution module into the Transformer encoder, EENED can learn the time-dependent relationship of the patient's EEG signal features and notice local EEG abnormal mutations closely related to epilepsy, such as the appearance of spikes and the sprinkling of sharp and slow waves. Our proposed framework combines the ability of Transformer and CNN to capture different scale features of EEG signals and holds promise for improving the accuracy and reliability of epilepsy detection. Our source code will be released soon on GitHub.Comment: Accepted by IEEE CAI 202