Reinforced Multi-Teacher Selection for Knowledge Distillation

In natural language processing (NLP) tasks, slow inference speed and huge footprints in GPU usage remain the bottleneck of applying pre-trained deep models in production. As a popular method for model compression, knowledge distillation transfers knowledge from one or multiple large (teacher) models to a small (student) model. When multiple teacher models are available in distillation, the state-of-the-art methods assign a fixed weight to a teacher model in the whole distillation. Furthermore, most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled. We systematically develop a reinforced method to dynamically assign weights to teacher models for different training instances and optimize the performance of student model. Our extensive experimental results on several NLP tasks clearly verify the feasibility and effectiveness of our approach.Comment: AAAI 202

Disorder recognition in clinical texts using multi-label structured SVM

Background: Information extraction in clinical texts enables medical workers to find out problems of patients faster as well as makes intelligent diagnosis possible in the future. There has been a lot of work about disorder mention recognition in clinical narratives. But recognition of some more complicated disorder mentions like overlapping ones is still an open issue. This paper proposes a multi-label structured Support Vector Machine (SVM) based method for disorder mention recognition. We present a multi-label scheme which could be used in complicated entity recognition tasks. Results: We performed three sets of experiments to evaluate our model. Our best F-1-Score on the 2013 Conference and Labs of the Evaluation Forum data set is 0.7343. There are six types of labels in our multi-label scheme, all of which are represented by 24-bit binary numbers. The binary digits of each label contain information about different disorder mentions. Our multi-label method can recognize not only disorder mentions in the form of contiguous or discontiguous words but also mentions whose spans overlap with each other. The experiments indicate that our multi-label structured SVM model outperforms the condition random field (CRF) model for this disorder mention recognition task. The experiments show that our multi-label scheme surpasses the baseline. Especially for overlapping disorder mentions, the F-1-Score of our multi-label scheme is 0.1428 higher than the baseline BIOHD1234 scheme. Conclusions: This multi-label structured SVM based approach is demonstrated to work well with this disorder recognition task. The novel multi-label scheme we presented is superior to the baseline and it can be used in other models to solve various types of complicated entity recognition tasks as well.NSF projects of China [61373108, 61133012]; National Social Science Main Project of China [11ZD189]; Hubei NSF project of China [2012FFA088]SCI(E)ARTICLE1

The Spatial Associations of Cerebral Blood Flow and Spontaneous Brain Activities with White Matter Hyperintensities—An Exploratory Study Using Multimodal Magnetic Resonance Imaging

White matter hyperintensities (WMHs) have been reported to be correlated with functional brain changes, but the association of the specific WMHs distribution pattern with regional functional changes remains uncertain. The aim of this study is to explore the possible spatial correlation of WMH with changes in cerebral blood flow (CBF) and spontaneous brain activities in elderly using a novel approach. The WMHs, CBF, and spontaneous brain activities measured by intrinsic connectivity contrast (ICC), were quantified using multimodal magnetic resonance imaging for 69 elderly subjects. Such approach enables us to expand our search for newly identified correlated areas by drawing strengths of different modes and provides a means for triangulation as well as complementary insights. The results showed significant positive correlations between WMH volumes in the right superior corona radiata and CBF in the left supplementary motor area, as well as between WMH volumes in left anterior limb internal capsule and CBF in the right putamen. Significant correlations of regional WMH volumes and ICC were also detected between the right anterior corona radiata and the left cuneus, and the right superior occipital cortex, as well as between the right superior corona radiata and the left superior occipital cortex. These findings may suggest a regional compensatory functional enhancement accounting for the maintenance of cognitively normal status, which can be supported by the widely observed phenomenon that mild to moderate WMH load could have little effect on global cognitive performance

Neural evidence for long-term marriage shaping the functional brain network organization between couples

Long-term married couples have been reported to share personality and behavioural similarities, but whether long-term marriage would shape the brain is hitherto unknown. In this study, 35 pairs of long-term married couples, who have married and living together at least 30 years, were recruited, and resting state functional magnetic resonance imaging was used to examine the neural correlates of long-term marriage between couples. Seven intrinsic connectivity networks were extracted using spatially constrained group independent component analysis, and the spatial similarity of each network as well as functional connectome similarity between couples were investigated respectively. The significant spatial similarities in the salience and frontoparietal networks as well as marginally significant connectome similarity were observed in long-term married couples. In addition, the marital duration showed a significantly positive correlation with the spatial similarity in the frontoparietal network and connectome similarity. The results provide objective evidence that long-term marriage would shape brain network organization, and the combination of initial personality traits and long-term common experience of the couples may be potential factors that account for similar brain network organizations between couples

Assessing Long-Term Trend of Particulate Matter Pollution in the Pearl River Delta Region Using Satellite Remote Sensing

Serious particulate matter (PM) pollution problems in many polluted regions of China have been frequently reported in recent years. Long-term exposure to ambient PM pollution is significantly associated with adverse health effects. Characterizing the long-term trends and variation in PM pollution is a basic requirement for evaluating long-term exposure and for guiding future policies to reduce the effects of air pollution on health. However, long-term, ground-based PM measurements are only available at a few fixed stations. In this study, an algorithm is developed and validated to estimate PM concentrations based on the satellite atmospheric optical depth with 1 km spatial resolution. The long-term trends of PM₁₀ concentrations in the entire Pearl River Delta (PRD) region and different cities are quantified and discussed. From 2001 to 2013, the PM₁₀ pollution of the entire PRD region was dominated by a decreasing trend of −0.15 ± 0.23 μg/m³·yr. This decreasing PM₁₀ trend was apparent over 75% of the PRD area, with the most significant decreases observed in the center of the region. However, the remaining 25%, mostly located in the outskirts of the region, showed an increasing PM₁₀ trend. This overall decreasing trend indicates the effectiveness of the control measures applied in the past decade for the primary pollutants