SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Transfer learning has fundamentally changed the landscape of natural language processing (NLP) research. Many existing state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model. To address the above issue in a more principled manner, we propose a new computational framework for robust and efficient fine-tuning for pre-trained language models. Specifically, our proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the capacity of the model; 2. Bregman proximal point optimization, which is a class of trust-region methods and can prevent knowledge forgetting. Our experiments demonstrate that our proposed method achieves the state-of-the-art performance on multiple NLP benchmarks.Comment: The 58th annual meeting of the Association for Computational Linguistics (ACL 2020

SDSS J163459.82+204936.0: A Ringed Infrared-Luminous Quasar with Outflows in both Absorption and Emission Lines

SDSS J1634+2049 is a local (z = 0.1293) infrared-luminous quasar with LIR= 10^11.91 Lsun. We present a detailed multiwavelength study of both the host galaxy and the nucleus. The host galaxy demonstrates violent, obscured star formation activities with SFR ~ 140 Msun yr^-1, estimated from either the PAH emission or IR luminosity. The optical to NIR spectra exhibit a blueshifted narrow cuspy component in Hb, HeI5876,10830 and other emission lines consistently with an offset velocity of ~900 km/s, as well as additional blueshifting phenomena in high-ionization lines , while there exist blueshifted broad absorption lines (BALs) in NaID and HeI*3889,10830, indicative of the AGN outflows producing BALs and emission lines. Constrained mutually by the several BALs with CLOUDY, the physical properties of the absorption-line outflow are derived as follows: 10^4 < n_H <= 10^5 cm^-3, 10^-1.3 <= U <= 10^-0.7 and 10^22.5<= N_H <= 10^22.9 cm^-2 , similar to those derived for the emission-line outflows. The similarity suggests a common origin. Taking advantages of both the absorption lines and outflowing emission lines, we find that the outflow gas is located at a distance of 48 - 65 pc from the nucleus, and that the kinetic luminosity of the outflow is 10^44-10^46 erg s^-1. J1634+2049 has a off-centered galactic ring on the scale of ~ 30 kpc that is proved to be formed by a recent head-on collision by a nearby galaxy. Thus this quasar is a valuable object in the transitional phase emerging out of dust enshrouding as depicted by the co-evolution scenario.Comment: 13 figures, 6 tables; accepted for publication in Ap

Web Mining For Financial Market Prediction Based On Online Sentiments

Financial market prediction is a critically important research topic in financial data mining because of its potential commerce application and attractive profits. Previous studies in financial market prediction mainly focus on financial and economic indicators. Web information, as an information repository, has been used in customer relationship management and recommendation, but it is rarely considered to be useful in financial market prediction. In this paper, a combined web mining and sentiment analysis method is proposed to forecast financial markets using web information. In the proposed method, a spider is firstly employed to crawl tweets from Twitter. Secondly, Opinion Finder is offered to mining the online sentiments hidden in tweets. Thirdly, some new sentiment indicators are suggested and a stochastic time effective function (STEF) is introduced to integrate everyday sentiments. Fourthly, support vector regressions (SVRs) are used to model the relationship between online sentiments and financial market prices. Finally, the selective model can be serviced for financial market prediction. To validate the proposed method, Standard and Poor’s 500 Index (S&P 500) is used for evaluation. The empirical results show that our proposed forecasting method outperforms the traditional forecasting methods, and meanwhile, the proposed method can also capture individual behavior in financial market quickly and easily. These findings imply that the proposed method is a promising approach for financial market prediction