A Comparative Study on Regularization Strategies for Embedding-based Neural Networks

This paper aims to compare different regularization strategies to address a common phenomenon, severe overfitting, in embedding-based neural networks for NLP. We chose two widely studied neural models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout. We also emphasized on incremental hyperparameter tuning, and combining different regularizations. The results provide a picture on tuning hyperparameters for neural NLP models.Comment: EMNLP '1

OPTIMIZATION UNDER STOCHASTIC ENVIRONMENT

Stochastic optimization (SO) is extensively studied in various fields, such as control engineering, operations research, and computer science. It has found wide applications ranging from path planning (civil engineering) and tool-life testing (industrial engineering) to Go-playing artificial intelligence (computer science). However, SO is usually a hard problem primarily because of the added complexity from random variables. The objective of this research is to investigate three types of SO problems: single-stage SO, multi-stage SO and fast real-time parameter estimation under stochastic environment.\par We first study the single-stage optimization problem. We propose Direct Gradient Augmented Response Surface Methodology (DiGARSM), a new sequential first-order method for optimizing a stochastic function. In this approach, gradients of the objective function with respect to the desired parameters are utilized in addition to response measurements. We intend to establish convergence of the proposed method, as well as traditional approaches which do not use gradients. We expect an improvement in convergence speed with the added derivative information. \par Second, we analyze a tree search problem with an underlying Markov decision process. Unlike traditional tree search algorithms where the goal is to maximize the cumulative reward in the learning process, the proposed method aims at identifying the best action at the root that achieves the highest reward. A new tree algorithm based on ranking and selection is proposed. The selection policy at each node aims at maximizing the probability of correctly selecting the best action. \par The third topic is motivated by problems arising in neuroscience, specifically, a Maximum Likelihood (ML) parameter estimation of linear models with noise-corrupted observations. We developed an optimization algorithm designed for non-convex, linear state-space model parameter estimation. The ML estimation is carried out by the Expectation-Maximization algorithm, which iteratively updates parameter estimates based on the previous estimates. Since the likelihood surface is in general non-convex, a model-based global optimization method called Model Reference Adaptive Search (MRAS) is applied

Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path

Relation classification is an important research arena in the field of natural language processing (NLP). In this paper, we present SDP-LSTM, a novel neural network to classify the relation of two entities in a sentence. Our neural architecture leverages the shortest dependency path (SDP) between two entities; multichannel recurrent neural networks, with long short term memory (LSTM) units, pick up heterogeneous information along the SDP. Our proposed model has several distinct features: (1) The shortest dependency paths retain most relevant information (to relation classification), while eliminating irrelevant words in the sentence. (2) The multichannel LSTM networks allow effective information integration from heterogeneous sources over the dependency paths. (3) A customized dropout strategy regularizes the neural network to alleviate overfitting. We test our model on the SemEval 2010 relation classification task, and achieve an $F_1$-score of 83.7\%, higher than competing methods in the literature.Comment: EMNLP '1

Determinants of price fluctuations in the electricity market: a study with PCA and NARDL models

In the modern electricity markets, negative prices and spike prices coexist as a pair of opposite economic phenomena. This study investigates how these extreme prices play as the determinants to drive price fluctuations in the electricity market. We construct a two-stage analysis including a principal component analysis (PCA) and a nonlinear autoregressive distributed lags model (NARDL). We apply this analytical method to the wholesale Pennsylvania, New Jersey and Maryland (PJM) electricity market. We find that according to PCA, in the individual transmission lines, spike prices are determinants with largest explanatory power to the variation of prices, while according to NARDL, from the standpoint of the overall market, negative prices have a larger potential effect on both the real-time market and the forward market. These results are valuable and contributive to managers and operators in the electricity markets for policy decision making

Combination model of heterogeneous data for security measurement

Measuring security is a core step for guaranteeing security of network and information systems. Due to massiveness and heterogeneity of measurement data, it is difficult toclassify and combinethem on demand. In thispaper, consideringimplication relationship of metrics, we propose a combination model and combination policy for security measurement. Several examples demonstrate the effectiveness of our model

Combination Model of Heterogeneous Data for Security Measurement

Measuring security is a core step for guaranteeing security of network and information systems. Due to massiveness and heterogeneity of measurement data, it is difficult to classify and combine them on demand. In this paper, considering implication relationship of metrics, we propose a combination model and combination policy for security measurement. Several examples demonstrate the effectiveness of our model

Comparison of Diagnostic Performance of Three-Dimensional Positron Emission Mammography versus Whole Body Positron Emission Tomography in Breast Cancer

Objective. To compare the diagnostic performance of three-dimensional (3D) positron emission mammography (PEM) versus whole body positron emission tomography (WBPET) for breast cancer. Methods. A total of 410 women with normal breast or benign or highly suspicious malignant tumors were randomized at 1 : 1 ratio to undergo 3D-PEM followed by WBPET or WBPET followed by 3D-PEM. Lumpectomy or mastectomy was performed on eligible participants after the scanning. Results. The sensitivity and specificity of 3D-PEM were 92.8% and 54.5%, respectively. WBPET showed a sensitivity of 95.7% and specificity of 56.8%. After exclusion of the patients with lesions beyond the detecting range of the 3D-PEM instrument, 3D-PEM showed higher sensitivity than WBPET (97.0% versus 95.5%, P = 0.913), particularly for small lesions (<1 cm) (72.0% versus 60.0%, P = 0.685). Conclusions. The 3D-PEM appears more sensitive to small lesions than WBPET but may fail to detect lesions that are beyond the detecting range. This study was approved by the Ethics Committee (E2012052) at the Tianjin Medical University Cancer Institute and Hospital (Tianjin, China). The instrument positron emission mammography (PEMi) was approved by China State Food and Drug Administration under the registration number 20153331166

The efficacy of upfront craniocerebral radiotherapy and epidermal growth factor receptor-tyrosine kinase inhibitors in patients with epidermal growth factor receptor-positive non-small cell lung cancer with brain metastases

The present study aims to investigate the therapeutic value of third-generation epidermal growth factor receptor (EGFR) tyrosine kinase inhibitors (TKIs) combined with cranial radiotherapy (RT) in patients with EGFR-positive non-small cell lung cancer (NSCLC) and brain metastases (BMs).MethodologyThis is a retrospective study that involved 213 patients with EGFR-NSCLC and BMs, with the patients divided into two groups: the upfront cranial RT (ucRT) group (n = 96) and the non-ucRT group (n = 117). All patients were administered with osimertinib, and those in the ucRT group also underwent RT. The overall survival (OS), progression-free survival (PFS) and intracranial PFS (IPFS) of the two groups were compared.ResultsThe ucRT group manifested a markedly higher IPFS than the non-ucRT group (29.65 months vs 21.8 months; P &lt; 0.0001). The subgroup analysis revealed that patients with oligometastases (OLOGO-BMs; 1–3 BMs) demonstrated a notably longer OS (44.5 months vs 37.3 months; P &lt; 0.0001), PFS (32.3 months vs 20.8 months; P = 0.6884) and IPFS (37.8 months vs 22.1 months; P &lt; 0.0001) in the ucRT group than in the non-ucRT group. However, for patients with multiple BMs, there was no significant difference in OS (27.3 months vs 34.4 months; P = 0.0710) and PFS (13.7 months vs 13.2 months; P = 0.0516) between the ucRT group and the non-ucRT group; the ucRT group exhibited a higher IPFS (26.4 months vs 21.35 months; P = 0.0028). Cox’s multivariate analysis of patients with OLOGO-BM indicated that the use of ucRT was linked to a better OS (heart rate [HR] = 0.392; 95% confidence interval [CI]: 0.178–0.863; P = 0.020) and PFS (HR = 0.558; 95% CI: 0.316–0.986; P = 0.044).ConclusionUpfront cerebral cranial stereotactic radiosurgery can improve outcomes in EGFR-positive patients with NSCLC and OLOGO-BM. However, for patients with multiple BMs, the preferable strategy may be pre-treatment with EGFR-TKIs