Improving Negative Sampling for Word Representation using Self-embedded Features

Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skipgram model without a proper negative sampler. By performing an insightful analysis from the stochastic gradient descent (SGD) learning perspective, we demonstrate that, both theoretically and intuitively, negative samples with larger inner product scores are more informative than those with lower scores for the SGD learner in terms of both convergence rate and accuracy. Understanding this, we propose an alternative sampling algorithm that dynamically selects informative negative samples during each SGD update. More importantly, the proposed sampler accounts for multi-dimensional self-embedded features during the sampling process, which essentially makes it more effective than the original popularity-based (one-dimensional) sampler. Empirical experiments further verify our observations, and show that our fine-grained samplers gain significant improvement over the existing ones without increasing computational complexity.Comment: Accepted in WSDM 201

Gated Class-Attention with Cascaded Feature Drift Compensation for Exemplar-free Continual Learning of Vision Transformers

In this paper we propose a new method for exemplar-free class incremental training of ViTs. The main challenge of exemplar-free continual learning is maintaining plasticity of the learner without causing catastrophic forgetting of previously learned tasks. This is often achieved via exemplar replay which can help recalibrate previous task classifiers to the feature drift which occurs when learning new tasks. Exemplar replay, however, comes at the cost of retaining samples from previous tasks which for some applications may not be possible. To address the problem of continual ViT training, we first propose gated class-attention to minimize the drift in the final ViT transformer block. This mask-based gating is applied to class-attention mechanism of the last transformer block and strongly regulates the weights crucial for previous tasks. Secondly, we propose a new method of feature drift compensation that accommodates feature drift in the backbone when learning new tasks. The combination of gated class-attention and cascaded feature drift compensation allows for plasticity towards new tasks while limiting forgetting of previous ones. Extensive experiments performed on CIFAR-100, Tiny-ImageNet and ImageNet100 demonstrate that our method outperforms existing exemplar-free state-of-the-art methods without the need to store any representative exemplars of past tasks

Characterising Alzheimer's Disease with EEG-based Energy Landscape Analysis

Alzheimer's disease (AD) is one of the most common neurodegenerative diseases, with around 50 million patients worldwide. Accessible and non-invasive methods of diagnosing and characterising AD are therefore urgently required. Electroencephalography (EEG) fulfils these criteria and is often used when studying AD. Several features derived from EEG were shown to predict AD with high accuracy, e.g. signal complexity and synchronisation. However, the dynamics of how the brain transitions between stable states have not been properly studied in the case of AD and EEG data. Energy landscape analysis is a method that can be used to quantify these dynamics. This work presents the first application of this method to both AD and EEG. Energy landscape assigns energy value to each possible state, i.e. pattern of activations across brain regions. The energy is inversely proportional to the probability of occurrence. By studying the features of energy landscapes of 20 AD patients and 20 healthy age-matched counterparts, significant differences were found. The dynamics of AD patients' brain networks were shown to be more constrained - with more local minima, less variation in basin size, and smaller basins. We show that energy landscapes can predict AD with high accuracy, performing significantly better than baseline models.Comment: 11 pages, 7 figure

A MEMS Energy Harvesting Device for Vibration with Low Acceleration

n/

Modeling realistic multiphase flows using a non-orthogonal multiple-relaxation-time lattice Boltzmann method

In this paper, we develop a three-dimensional multiple-relaxation-time lattice Boltzmann method (MRT-LBM) based on a set of non-orthogonal basis vectors. Compared with the classical MRT-LBM based on a set of orthogonal basis vectors, the present non-orthogonal MRT-LBM simplifies the transformation between the discrete velocity space and the moment space, and exhibits better portability across different lattices. The proposed method is then extended to multiphase flows at large density ratio with tunable surface tension, and its numerical stability and accuracy are well demonstrated by some benchmark cases. Using the proposed method, a practical case of a fuel droplet impacting on a dry surface at high Reynolds and Weber numbers is simulated and the evolution of the spreading film diameter agrees well with the experimental data. Furthermore, another realistic case of a droplet impacting on a super-hydrophobic wall with a cylindrical obstacle is reproduced, which confirms the experimental finding of Liu \textit{et al.} [``Symmetry breaking in drop bouncing on curved surfaces," Nature communications 6, 10034 (2015)] that the contact time is minimized when the cylinder radius is comparable with the droplet cylinder.Comment: 19 pages, 11 figure

A Study of background for IXPE

Focal plane X-ray polarimetry is intended for relatively bright sources with a negligible impact of background. However this might not be always possible for IXPE (Imaging X-ray Polarimetry Explorer) when observing faint extended sources like supernova remnants. We present for the first time the expected background of IXPE by Monte Carlo simulation and its impact on real observations of point and extended X-ray sources. The simulation of background has been performed by Monte Carlo based on GEANT4 framework. The spacecraft and the detector units have been modeled, and the expected background components in IXPE orbital environment have been evaluated. We studied different background rejection techniques based on the analysis of the tracks collected by the Gas Pixel Detectors on board IXPE. The estimated background is about 2.9 times larger than the requirement, yet it is still negligible when observing point like sources. Albeit small, the impact on supernova remnants indicates the need for a background subtraction for the observation of the extended sources.Comment: 16 pages, 16 figure

Dielectric Breakdown in Chemical Vapor Deposited Hexagonal Boron Nitride

Insulating films are essential in multiple electronic devices because they can provide essential functionalities, such as capacitance effects and electrical fields. Two-dimensional (2D) layered materials have superb electronic, physical, chemical, thermal, and optical properties, and they can be effectively used to provide additional performances, such as flexibility and transparency. 2D layered insulators are called to be essential in future electronic devices, but their reliability, degradation kinetics, and dielectric breakdown (BD) process are still not understood. In this work, the dielectric breakdown process of multilayer hexagonal boron nitride (h-BN) is analyzed on the nanoscale and on the device level, and the experimental results are studied via theoretical models. It is found that under electrical stress, local charge accumulation and charge trapping/detrapping are the onset mechanisms for dielectric BD formation. By means of conductive atomic force microscopy, the BD event was triggered at several locations on the surface of different dielectrics (SiO2, HfO2, Al2O3, multilayer h-BN, and monolayer h-BN); BD-induced hillocks rapidly appeared on the surface of all of them when the BD was reached, except in monolayer h-BN. The high thermal conductivity of h-BN combined with the one-atom-thick nature are genuine factors contributing to heat dissipation at the BD spot, which avoids self-accelerated and thermally driven catastrophic BD. These results point to monolayer h-BN as a sublime dielectric in terms of reliability, which may have important implications in future digital electronic devices.Fil: Jiang, Lanlan. Soochow University; ChinaFil: Shi, Yuanyuan. Soochow University; China. University of Stanford; Estados UnidosFil: Hui, Fei. Soochow University; China. Massachusetts Institute of Technology; Estados UnidosFil: Tang, Kechao. University of Stanford; Estados UnidosFil: Wu, Qian. Soochow University; ChinaFil: Pan, Chengbin. Soochow University; ChinaFil: Jing, Xu. Soochow University; China. University of Texas at Austin; Estados UnidosFil: Uppal, Hasan. University of Manchester; Reino UnidoFil: Palumbo, Félix Roberto Mario. Comisión Nacional de Energía Atómica; Argentina. Universidad Tecnológica Nacional; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas; ArgentinaFil: Lu, Guangyuan. Chinese Academy of Sciences; República de ChinaFil: Wu, Tianru. Chinese Academy of Sciences; República de ChinaFil: Wang, Haomin. Chinese Academy of Sciences; República de ChinaFil: Villena, Marco A.. Soochow University; ChinaFil: Xie, Xiaoming. Chinese Academy of Sciences; República de China. ShanghaiTech University; ChinaFil: McIntyre, Paul C.. University of Stanford; Estados UnidosFil: Lanza, Mario. Soochow University; Chin

Understanding and controlling the efficiency of Au24M(SR)18 nanoclusters as singlet-oxygen photosensitizers

Atomically precise Au24M(SR)18 clusters were used as singlet-oxygen photosensitizers. Comprehensive kinetic analysis provided insights into the mechanism and driving-force dependence of the quenching of 1O2 by gold nanoclusters