Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.Comment: Accepted to AAAI 201

Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons

An activation boundary for a neuron refers to a separating hyperplane that determines whether the neuron is activated or deactivated. It has been long considered in neural networks that the activations of neurons, rather than their exact output values, play the most important role in forming classification friendly partitions of the hidden feature space. However, as far as we know, this aspect of neural networks has not been considered in the literature of knowledge transfer. In this paper, we propose a knowledge transfer method via distillation of activation boundaries formed by hidden neurons. For the distillation, we propose an activation transfer loss that has the minimum value when the boundaries generated by the student coincide with those by the teacher. Since the activation transfer loss is not differentiable, we design a piecewise differentiable loss approximating the activation transfer loss. By the proposed method, the student learns a separating boundary between activation region and deactivation region formed by each neuron in the teacher. Through the experiments in various aspects of knowledge transfer, it is verified that the proposed method outperforms the current state-of-the-art.Comment: Accepted to AAAI 201

AAD-2004, a potent spin trapping molecule and microsomal prostaglandin E synthase-1 inhibitor, shows safety and efficacy in a mouse model of ALS

While free radicals and inflammation constitute major routes of neuronal injury occurring in neurodegenerative diseases, neither antioxidants nor nonsteroidal anti-inflammatory drugs (NSAIDs) have shown significant efficacy in human clinical trials. To explore the possibility that concurrent blockade of free radicals and PGE2-mediated inflammation might constitute a safe and effective therapeutic approach to certain neurodegenerative diseases, we have developed 2-hydroxy-5-[2-(4-trifluoromethylphenyl)-ethylaminobezoic acid (AAD-2004) as a derivative of aspirin. AAD-2004 completely removed free radicals at 50 nM as a potent spin trapping molecule and inhibited microsomal prostaglandin E synthase-1 (mPGES-1) with an IC50 of 230 nM. Oral administration of AAD-2004 blocked free radical formation, PGE2 formation, and microglial activation in the spinal motor neurons of SOD1G93A mice. As a consequence, AAD-2004 reduced autophagosome formation, axonopathy, and motor neuron degeneration, improving motor function and increasing life span. In these assays, AAD-2004 was superior to ibuprofen or riluzole. Gastric bleeding was not induced by AAD-2004 even at a dose 400-fold higher than that required to obtain maximal therapeutic efficacy in SOD1G93A mice. Targeting both mPGES-1 and free radicals may be a promising approach to reduce neurodegeneration in ALS and possibly other neurodegenerative diseases

Microstructure reconstruction using diffusion-based generative models

Microstructure reconstruction has been an essential part of computational material engineering to reveal the relationship between the microstructures and the material properties. However, it is still challenging to find a general solution for microstructure characterization and reconstruction (MCR) tasks although there have been many attempts such as the descriptor-based reconstruction methods. To address this generality problem, the denoising diffusion probabilistic models are first employed for the microstructure reconstruction task which can be applied to various types of material systems. Several microstructures (e.g., carbonate, ceramics, copolymer, etc.) are considered to be reproduced for validating the proposed models while addressing the quality of the generated images with the quantitative evaluation metrics (FID score, precision and recall). The results show that the proposed diffusion model based approach is applicable for reproducing various types of microstructures with different spatial distributions of morphological features. The present approach also provides a stable training procedure with simple implementation for generating visually similar microstructures (and also statistically equivalent) without requiring expert knowledge and some time-consuming parametric studies. The proposed approach has the potential of being a universal microstructure reconstruction method for handling complex microstructures for materials science

Multi-plane denoising diffusion-based dimensionality expansion for 2D-to-3D reconstruction of microstructures with harmonized sampling

Acquiring reliable microstructure datasets is a pivotal step toward the systematic design of materials with the aid of integrated computational materials engineering (ICME) approaches. However, obtaining three-dimensional (3D) microstructure datasets is often challenging due to high experimental costs or technical limitations, while acquiring two-dimensional (2D) micrographs is comparatively easier. To deal with this issue, this study proposes a novel framework for 2D-to-3D reconstruction of microstructures called Micro3Diff using diffusion-based generative models (DGMs). Specifically, this approach solely requires pre-trained DGMs for the generation of 2D samples, and dimensionality expansion (2D-to-3D) takes place only during the generation process (i.e., reverse diffusion process). The proposed framework incorporates a new concept referred to as multi-plane denoising diffusion, which transforms noisy samples (i.e., latent variables) from different planes into the data structure while maintaining spatial connectivity in 3D space. Furthermore, a harmonized sampling process is developed to address possible deviations from the reverse Markov chain of DGMs during the dimensionality expansion. Combined, we demonstrate the feasibility of Micro3Diff in reconstructing 3D samples with connected slices that maintain morphologically equivalence to the original 2D images. To validate the performance of Micro3Diff, various types of microstructures (synthetic and experimentally observed) are reconstructed, and the quality of the generated samples is assessed both qualitatively and quantitatively. The successful reconstruction outcomes inspire the potential utilization of Micro3Diff in upcoming ICME applications while achieving a breakthrough in comprehending and manipulating the latent space of DGMs

Ten years of MIREX: reflections, challenges and opportunities

The Music Information Retrieval Evaluation eXchange (MIREX) has been run annually since 2005, with the October 2014 plenary marking its tenth iteration. By 2013, MIREX has evaluated approximately 2000 individual music information retrieval (MIR) algorithms for a wide range of tasks over 37 different test collections. MIREX has involved researchers from over 29 different contrives with a median of 109 individual participants per year. This pater summarizes the history of MIREX form its earliest planning meeting in 2001 to the present. It reflects upon the administrative, financial, and technological challenges MIREX has faced and describes how those challenges have been surmounted. We propose new funding models, a distributed evaluation framework, and more holistic user experience evaluation tasks-some evolutionary, some revolutionary-for the continued success of MIREX. We hope that this paper will inspire MIR community members to contribute their ideas so MIREX can have many more successful years to come