1Cademy @ Causal News Corpus 2022: Leveraging Self-Training in Causality Classification of Socio-Political Event Data

This paper details our participation in the Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) workshop @ EMNLP 2022, where we take part in Subtask 1 of Shared Task 3. We approach the given task of event causality detection by proposing a self-training pipeline that follows a teacher-student classifier method. More specifically, we initially train a teacher model on the true, original task data, and use that teacher model to self-label data to be used in the training of a separate student model for the final task prediction. We test how restricting the number of positive or negative self-labeled examples in the self-training process affects classification performance. Our final results show that using self-training produces a comprehensive performance improvement across all models and self-labeled training sets tested within the task of event causality sequence classification. On top of that, we find that self-training performance did not diminish even when restricting either positive/negative examples used in training. Our code is be publicly available at https://github.com/Gzhang-umich/1CademyTeamOfCASE.Comment: Paper from CASE workshop at EMNLP 202

On the Effectiveness of Speech Self-supervised Learning for Music

Self-supervised learning (SSL) has shown promising results in various speech and natural language processing applications. However, its efficacy in music information retrieval (MIR) still remains largely unexplored. While previous SSL models pre-trained on music recordings may have been mostly closed-sourced, recent speech models such as wav2vec2.0 have shown promise in music modelling. Nevertheless, research exploring the effectiveness of applying speech SSL models to music recordings has been limited. We explore the music adaption of SSL with two distinctive speech-related models, data2vec1.0 and Hubert, and refer to them as music2vec and musicHuBERT, respectively. We train $12$ SSL models with 95M parameters under various pre-training configurations and systematically evaluate the MIR task performances with 13 different MIR tasks. Our findings suggest that training with music data can generally improve performance on MIR tasks, even when models are trained using paradigms designed for speech. However, we identify the limitations of such existing speech-oriented designs, especially in modelling polyphonic information. Based on the experimental results, empirical suggestions are also given for designing future musical SSL strategies and paradigms

MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is primarily due to the distinctive challenges associated with modelling musical knowledge, particularly its tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified a superior combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). These teachers effectively guide our student model, a BERT-style transformer encoder, to better model music audio. In addition, we introduce an in-batch noise mixture augmentation to enhance the representation robustness. Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attains state-of-the-art (SOTA) overall scores. The code and models are online: https://github.com/yizhilll/MERT

Membrane Biofouling Control by Surface Modification of Quaternary Ammonium Compound Using Atom-Transfer Radical-Polymerization Method with Silica Nanoparticle as Interlayer

A facile approach to fabricate antibiofouling membrane was developed by grafting quaternary ammonium compounds (QACs) onto polyvinylidene fluoride (PVDF) membrane via surface-initiated activators regenerated by electron transfer atom-transfer radical-polymerization (ARGET ATRP) method. During the modification process, a hydrophilic silica nanoparticle layer was also immobilized onto the membrane surface as an interlayer through silicification reaction for QAC grafting, which imparted the membrane with favorable surface properties (e.g., hydrophilic and negatively charged surface). The QAC-modified membrane (MQ) showed significantly improved hydrophilicity and permeability mainly due to the introduction of silica nanoparticles and exposure of hydrophilic quaternary ammonium groups instead of long alkyl chains. Furthermore, the coverage of QAC onto membrane surface enabled MQ membrane to have clear antibacterial effect, with an inhibition rate ~99.9% of Escherichia coli (Gram-negative) and Staphylococcus aureus (Gram-positive), respectively. According to the batch filtration test, MQ had better antibiofouling performance compared to the control membrane, which was ascribed to enhanced hydrophilicity and antibacterial activity. Furthermore, the MQ membrane also exhibited impressive stability of QAC upon suffering repeated fouling–cleaning tests. The modification protocols provide a new robust way to fabricate high-performance antibiofouling QAC-based membranes for wastewater treatment