882 research outputs found
A Comparative Study on Regularization Strategies for Embedding-based Neural Networks
This paper aims to compare different regularization strategies to address a
common phenomenon, severe overfitting, in embedding-based neural networks for
NLP. We chose two widely studied neural models and tasks as our testbed. We
tried several frequently applied or newly proposed regularization strategies,
including penalizing weights (embeddings excluded), penalizing embeddings,
re-embedding words, and dropout. We also emphasized on incremental
hyperparameter tuning, and combining different regularizations. The results
provide a picture on tuning hyperparameters for neural NLP models.Comment: EMNLP '1
Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path
Relation classification is an important research arena in the field of
natural language processing (NLP). In this paper, we present SDP-LSTM, a novel
neural network to classify the relation of two entities in a sentence. Our
neural architecture leverages the shortest dependency path (SDP) between two
entities; multichannel recurrent neural networks, with long short term memory
(LSTM) units, pick up heterogeneous information along the SDP. Our proposed
model has several distinct features: (1) The shortest dependency paths retain
most relevant information (to relation classification), while eliminating
irrelevant words in the sentence. (2) The multichannel LSTM networks allow
effective information integration from heterogeneous sources over the
dependency paths. (3) A customized dropout strategy regularizes the neural
network to alleviate overfitting. We test our model on the SemEval 2010
relation classification task, and achieve an -score of 83.7\%, higher than
competing methods in the literature.Comment: EMNLP '1
Building Program Vector Representations for Deep Learning
Deep learning has made significant breakthroughs in various fields of
artificial intelligence. Advantages of deep learning include the ability to
capture highly complicated features, weak involvement of human engineering,
etc. However, it is still virtually impossible to use deep learning to analyze
programs since deep architectures cannot be trained effectively with pure back
propagation. In this pioneering paper, we propose the "coding criterion" to
build program vector representations, which are the premise of deep learning
for program analysis. Our representation learning approach directly makes deep
learning a reality in this new field. We evaluate the learned vector
representations both qualitatively and quantitatively. We conclude, based on
the experiments, the coding criterion is successful in building program
representations. To evaluate whether deep learning is beneficial for program
analysis, we feed the representations to deep neural networks, and achieve
higher accuracy in the program classification task than "shallow" methods, such
as logistic regression and the support vector machine. This result confirms the
feasibility of deep learning to analyze programs. It also gives primary
evidence of its success in this new field. We believe deep learning will become
an outstanding technique for program analysis in the near future.Comment: This paper was submitted to ICSE'1
An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation
Open-domain dialogue systems aim to interact with humans through natural
language texts in an open-ended fashion. Despite the recent success of super
large dialogue systems such as ChatGPT, using medium-to-small-sized dialogue
systems remains the common practice as they are more lightweight and
accessible; however, generating diverse dialogue responses is challenging,
especially with smaller models. In this work, we propose an Equal-size Hard
Expectation--Maximization (EqHard-EM) algorithm to train a multi-decoder model
for diverse dialogue generation. Our algorithm assigns a sample to a decoder in
a hard manner and additionally imposes an equal-assignment constraint to ensure
that all decoders are well-trained. We provide detailed theoretical analysis to
justify our approach. Further, experiments on two large-scale open-domain
dialogue datasets verify that our EqHard-EM algorithm generates high-quality
diverse responses.Comment: Accepted by ICLR 202
Impaired Functional Criticality of Human Brain during Alzheimer's Disease Progression
The progression of Alzheimer's Disease (AD) has been proposed to comprise three stages, subjective cognitive decline (SCD), mild cognitive impairment (MCI), and AD. Was brain dynamics across the three stages smooth? Was there a critical transition? How could we characterize and study functional criticality of human brain? Based on dynamical characteristics of critical transition from nonlinear dynamics, we proposed a vertex-wise Index of Functional Criticality (vIFC) of fMRI time series in this study. Using 42 SCD, 67 amnestic MCI (aMCI), 34 AD patients as well as their age-, sex-, years of education-matched 54 NC, our new method vIFC successfully detected significant patient-normal differences for SCD and aMCI, as well as significant negative correlates of vIFC in the right middle temporal gyrus with total scores of Montreal Cognitive Assessment (MoCA) in SCD. In comparison, standard deviation of fMRI time series only detected significant differences between AD patients and normal controls. As an index of functional criticality of human brain derived from nonlinear dynamics, vIFC could serve as a sensitive neuroimaging marker for future studies; considering much more vIFC impairments in aMCI compared to SCD and AD, our study indicated aMCI as a critical stage across AD progression
Feature Extraction and Classification of EHG between Pregnancy and Labour Group Using Hilbert-Huang Transform and Extreme Learning Machine
Preterm birth (PTB) is the leading cause of perinatal mortality and long-term morbidity, which results in significant health and economic problems. The early detection of PTB has great significance for its prevention. The electrohysterogram (EHG) related to uterine contraction is a noninvasive, real-time, and automatic novel technology which can be used to detect, diagnose, or predict PTB. This paper presents a method for feature extraction and classification of EHG between pregnancy and labour group, based on Hilbert-Huang transform (HHT) and extreme learning machine (ELM). For each sample, each channel was decomposed into a set of intrinsic mode functions (IMFs) using empirical mode decomposition (EMD). Then, the Hilbert transform was applied to IMF to obtain analytic function. The maximum amplitude of analytic function was extracted as feature. The identification model was constructed based on ELM. Experimental results reveal that the best classification performance of the proposed method can reach an accuracy of 88.00%, a sensitivity of 91.30%, and a specificity of 85.19%. The area under receiver operating characteristic (ROC) curve is 0.88. Finally, experimental results indicate that the method developed in this work could be effective in the classification of EHG between pregnancy and labour group
- …