Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply

Traditional end-to-end (E2E) training of deep networks necessitates storing intermediate activations for back-propagation, resulting in a large memory footprint on GPUs and restricted model parallelization. As an alternative, greedy local learning partitions the network into gradient-isolated modules and trains supervisely based on local preliminary losses, thereby providing asynchronous and parallel training methods that substantially reduce memory cost. However, empirical experiments reveal that as the number of segmentations of the gradient-isolated module increases, the performance of the local learning scheme degrades substantially, severely limiting its expansibility. To avoid this issue, we theoretically analyze the greedy local learning from the standpoint of information theory and propose a ContSup scheme, which incorporates context supply between isolated modules to compensate for information loss. Experiments on benchmark datasets (i.e. CIFAR, SVHN, STL-10) achieve SOTA results and indicate that our proposed method can significantly improve the performance of greedy local learning with minimal memory and computational overhead, allowing for the boost of the number of isolated modules. Our codes are available at https://github.com/Tab-ct/ContSup.Comment: 9 figures, 12 table

SDiT: Spiking Diffusion Model with Transformer

Spiking neural networks (SNNs) have low power consumption and bio-interpretable characteristics, and are considered to have tremendous potential for energy-efficient computing. However, the exploration of SNNs on image generation tasks remains very limited, and a unified and effective structure for SNN-based generative models has yet to be proposed. In this paper, we explore a novel diffusion model architecture within spiking neural networks. We utilize transformer to replace the commonly used U-net structure in mainstream diffusion models. It can generate higher quality images with relatively lower computational cost and shorter sampling time. It aims to provide an empirical baseline for research of generative models based on SNNs. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 datasets demonstrate that our work is highly competitive compared to existing SNN generative models

On the Effectiveness of Speech Self-supervised Learning for Music

Self-supervised learning (SSL) has shown promising results in various speech and natural language processing applications. However, its efficacy in music information retrieval (MIR) still remains largely unexplored. While previous SSL models pre-trained on music recordings may have been mostly closed-sourced, recent speech models such as wav2vec2.0 have shown promise in music modelling. Nevertheless, research exploring the effectiveness of applying speech SSL models to music recordings has been limited. We explore the music adaption of SSL with two distinctive speech-related models, data2vec1.0 and Hubert, and refer to them as music2vec and musicHuBERT, respectively. We train $12$ SSL models with 95M parameters under various pre-training configurations and systematically evaluate the MIR task performances with 13 different MIR tasks. Our findings suggest that training with music data can generally improve performance on MIR tasks, even when models are trained using paradigms designed for speech. However, we identify the limitations of such existing speech-oriented designs, especially in modelling polyphonic information. Based on the experimental results, empirical suggestions are also given for designing future musical SSL strategies and paradigms

MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is primarily due to the distinctive challenges associated with modelling musical knowledge, particularly its tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified a superior combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). These teachers effectively guide our student model, a BERT-style transformer encoder, to better model music audio. In addition, we introduce an in-batch noise mixture augmentation to enhance the representation robustness. Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attains state-of-the-art (SOTA) overall scores. The code and models are online: https://github.com/yizhilll/MERT

ICDP workshop on scientific drilling of Nam Co on the Tibetan Plateau: 1 million years of paleoenvironmental history, geomicrobiology, tectonics and paleomagnetism derived from sediments of a high-altitude lake

The Tibetan Plateau is of peculiar societal relevance as it provides freshwater from the so-called “Water Tower of Asia” to a large portion of the Asian population. However, future climate change will affect the hydrological cycle in this area. To define parameters for future climate change scenarios it is necessary to improve the knowledge about thresholds, timing, pace and intensity of past climatic changes and associated environmental impacts. Sedimentary archives reaching far back in time and spanning several glacial–interglacial cycles such as Nam Co provide the unique possibility to extract such information. In order to explore the scientific opportunities that an ICDP drilling effort at Nam Co would provide, 40 scientists from 13 countries representing various scientific disciplines met in Beijing from 22 to 24 May 2018. Besides paleoclimatic investigations, opportunities for paleomagnetic, deep biosphere, tectonic and paleobiological studies were discussed. After having explored the technical and logistical challenges and the scientific opportunities all participants agreed on the great value and need to drill this extraordinary archive, which has a sediment thickness of more than 1 km, likely covering more than 1 Ma

Thermoeconomic evaluation and optimization of LiBr-H2O double absorption heat transformer driven by flat plate collector

In this paper, a solar double absorption heat transformer (SDAHT) operating with the LiBr-H2O solution is proposed to provide high temperature energy. The flat plate collector (FPC) is used to supply the demanded heat input in the present configuration, which will greatly broaden the application scope of the FPCs. The thermo-economic concept is applied to the evaluation and optimization of the SDAHT, aimed at minimizing its annual capital cost per kilowatt heat capacity (CPK) and payback period (PP). A dedicated computer model in the software Engineering Equation Solver was developed to conduct the study by means of a parametric analysis. The results show that there exists an optimum absorber/evaporator temperature at which CPK and PP can obtain the minimum values. With solar radiation intensity at 600 W/m(2), generation (evaporation) temperature at 75 degrees C, condensation temperature at 38 degrees C, absorption temperature at 135 degrees C and economizer efficiency set to 0.8, the optimum absorber/evaporator temperature and corresponding CPIC and PP are 104.0 degrees C, 928.1 euros/kW and 4.0 years, respectively. The cost of the FPCs takes the major part of CPK. Moreover, the effects of the operating parameters such as the generation (evaporation), condensation, absorption temperatures and design parameters including the first and second economizer efficiencies on the optimum absorber/evaporator temperature and corresponding CPK and PP have been analyzed in detail. Besides, some suggestions derived from the results are also given to assist the engineers in estimation of the economic performance

Support Vector Regression-Based Active Subspace (SVR-AS) Modeling of High-Speed Links for Fast and Accurate Sensitivity Analysis

A methodology based on the joint usage of support vector regression and active subspace is introduced in this paper for accelerated sensitivity analysis of high-speed links through parameter space dimensionality reduction. The proposed methodology uses the gradient directly obtained by support vector regression with Gaussian kernel to generate an active subspace with its application to the high-speed link model. Active subspace generated by this method is defined by the directions that are most influential on the desirable output measure. The resulting reduced-dimensional model is shown to perform well in sensitivity analysis of high-speed links including IBIS-AMI equalization, and is computationally more efficient than Sobol's method

\u27Dog-Bone\u27 Geometry Modeling based on PEEC for Package PDN

Dog-bone structure has been widely used in package PDN, yet it has not been thoroughly investigated. In this paper, the associated inductance for the dog-bone structure is studied with the partial element equivalent circuit (PEEC) method and CST. PEEC method is based on the electrical field integral equation (EFIE) and it serves as a bridge between electromagnetic problems and circuit ones. In this paper, the detailed geometry of the dog-bone structure in packages is discussed, the geometrical features of the inductance associated with the dog-bone structure are described. Besides, this paper also compares the inductance of the dog-bone structure with and without ground planes, which illustrates the effectiveness of the plane size to the total inductance of the dog-bone structure