Combining textual features with sentence embeddings

학위논문(박사) -- 서울대학교대학원 : 인문대학 언어학과, 2021.8. 박수지.이 논문의 목표는 한국어 기사 품질을 예측하기 위한 언어 모형을 개발하는 것이다. 기사 품질 예측 과제는 최근 가짜뉴스 등의 범람으로 그 필요성이 대두되면서도 자연언어처리의 최신 기법이 아직 적용되지 못하는 실정에 있다. 이 논문에서는 이러한 한계를 극복하기 위해 문장의 의미를 표상하는 SBERT 모형을 개발하고, 기사의 언어학적 자질을 활용하여 품질 분류의 성능을 높일 수 있는지를 검토하고자 한다. 그 결과 기사의 가독성, 응집성 등의 텍스트 자질을 사용한 기계학습 모형과 SBERT에서 자동으로 추출된 문맥 자질을 사용한 전이학습 모형이 모두 선행연구의 심층학습 결과보다 높은 성능을 보였고, 구체적으로는 SBERT 학습시 훈련 데이터를 확장하고 정제할 때, 그리고 텍스트 자질과 문맥 자질을 함께 사용할 때 성능이 더욱 향상되는 것을 관측하였다. 이를 통해 기사의 품질에서 언어학적 자질이 중요한 역할을 하며 자연언어처리의 최신 기법인 SBERT가 언어학적 자질을 추출하고 활용하는 데 실질적으로 기여할 수 있다는 결론을 내릴 수 있다.1 Introduction 1 2 Literature Review 5 2.1 Background 5 2.1.1 Text Classification 5 2.1.1.1 Initial Studies 5 2.1.1.2 News Classification 6 2.1.2 Text Quality Assessment 8 2.2 News Quality Prediction Task 9 2.2.1 News Data 9 2.2.1.1 Online vs. Offline 9 2.2.1.2 Expert-rated vs. User-rated 9 2.2.2 Prediction Methods 11 2.2.2.1 Manually Engineered Features v. Automatically Extracted Features 11 2.2.2.2 Machine Learning vs. Deep Learning 12 2.3 Instruments and Techniques 14 2.3.1 Sentence and Document Embeddings 14 2.3.1.1 Static Embeddings 14 2.3.1.2 Contextual Embeddings 16 2.3.2 Fusion Models 18 2.4 Summary 27 3 Methods 29 3.1 Data from Choi, Shin, and Kang (2021) 29 3.1.1 News Corpus 29 3.1.2 Quality Levels 29 3.1.3 Journalism Values 30 3.2 Linguistic Features 31 3.2.1 Justification of Using Linguistic Features Only 31 3.2.2 Two Types of Linguistic Features 32 3.2.2.1 Textual Features 32 3.2.2.2 Contextual Features 33 3.3 Summary 33 4 Ordinal Logistic Regression Models with Textual Features 35 4.1 Textual Features 35 4.1.1 Coh-Metrix 35 4.1.2 KOSAC Lexicon 36 4.1.3 K-LIWC 38 4.1.4 Others 38 4.2 Ordinal Logistic Regression 38 4.3 Results 39 4.3.1 Feature Selection 39 4.3.2 Impacts on Quality Evaluation 40 4.4 Discussion 40 4.4.1 Effect of Cosine Similarity by Issue 41 4.4.2 Effect of Quantitative Evidence 47 4.4.3 Effect of Sentiment 48 4.5 Summary 51 5 Deep Transfer Learning Models with Contextual Features 53 5.1 Contextual Features from SentenceBERT 53 5.1.1 Necessity of Sentence Embeddings 54 5.1.2 KR-SBERT 55 5.2 Deep Transfer Learning 56 5.3 Results 59 5.3.1 Measures of Multiclass Classification 59 5.3.2 Performances of news quality prediction models 60 5.4 Discussion 62 5.4.1 Effect of Data Size 62 5.4.2 Effect of Data Augmentation 62 5.4.3 Effect of Data Refinement 635.5 Summary 63 6 Fusion Models Combining Textual Features with Contextual Sentence Embeddings 65 6.1 Model Fusion 65 6.1.1 Feature-level Fusion: Concatenation 65 6.1.2 Logit-level Fusion: Interpolation 65 6.2 Results 68 6.2.1 Optimization of the Presentational Attribute Model 68 6.2.2 Performances of News Quality Prediction Models 68 6.3 Discussion 68 6.3.1 Effects of Fusion 70 6.3.2 Comparison with Choi et al. (2021) 71 6.4 Summary 71 7 Conclusion 73 References 75 A List of Words Used for Textual Feature Extraction 93 A.1 Coh-Metrix Features 93 A.2 Predicate Type Features 94 B Codes Used in Chapter 4 97 B.1 Python Code for Textual Feature Extraction 97 C Results of VIF test and Brant test 101 C.1 VIF Test in R 101 C.2 Brant Test in R 103 D Codes Used in Chapter 6 107 D.1 Python Code for Feature-Level Fusion 107 D.2 Python Code for Logit-Level Fusion 108박

트위터 게시물간 발화 연쇄·담화 분절 탐지 및 질의어 비포함 트윗 검색에의 활용

학위논문 (석사)-- 서울대학교 대학원 : 언어학과, 2014. 8. 신효필.This thesis describes a phenomenon where multiple tweets constitute a single discourse segment, and builds two rule-based models to detect whether two consecutive tweets under the same authorship convey a single message. Given the length limit of 140 characters, a tweet should be interpreted as an element of a larger unit rather than an individual document. Considering such a larger unit as a discourse segment and a tweet as an utterance, this study makes the following assumptions based on Centering Theory: (a) A tweet has at most one topic. (b) In non-initial tweets of a discourse segment, a topic word is realized as an anaphora, in particular a zero form in Korean. (c) Coherence between two tweets written by the same author is considered only if there is no tweet between them. (d) In two consecutive tweets, a topic is preferred to be continued. To predict tweet serialization and discourse segmentation, two criteria were used: temporal proximity and discourse markers. Temporal proximity shows whether the time interval between two tweets is less than a threshold level, which can be a constant or user-specific value. Discourse markers are classified into continuation markers and shift markers. Continuation markers include web-specific ones such as `>>', `(continued)', and numbers, and linguistic ones such as conjunctions and referring expressions. Shift markers include web-specific ones such as `RT' and URLS, and linguistic ones such as interjections and temporal adverbs. These factors are treated differently in two different models. The Strict Serialization (SS) model regards two tweets as serialized only if their interval is extremely short or they have a continuation marker. On the contrary, the Serialization Plus Discourse Segmentation (SPDS) model, following the assumption (d) that continuation is preferred to shifting, considers two tweets as serialized if their interval is not too long, and terminates a discourse segment only if the current tweet has a shift marker. To verify whether the proposed models are useful, an information retrieval task is implemented. It is predicted by the assumption (b) and observed in the data that topic words were implicit in some tweets in discourse segments consisting of multiple tweets. The current search system cannot retrieve such tweets and thus fails to satisfy users' information need to find diverse opinions in Twitter. When finding discourse segments compiled by the proposed models, the system can retrieve tweets that belong to the same discourse segment as some explicitly relevant one, without retrieving too many irrelevant tweets. Consequently, the proposed models achieve higher means of precision rates than those of the Query Matching model and TF-IDF Weighting model. Furthermore, since the SPDS model outperforms the SS model, the principle of unmarkedness of topic continuation seems to be also valid for social media. Lastly, this thesis also discovers that linguistic markers such as interjections, which have been typically treated as stopwords in information retrieval, are useful for discourse segment detection.List of Tables vii List of Figures viii 1 Introduction 1 1.1 Subject 1 1.2 Purposes 2 1.2.1 Detection of discourse segments in Twitter data 2 1.2.2 Retrieval of tweets without an explicit query term 3 1.3 Structure 4 2 Previous Work 5 2.1 General NLP studies 5 2.1.1 On social media data 5 2.1.2 Using discourse knowledge 6 2.2 Task-specific studies 6 2.2.1 Finding a proper unit for unstructured short texts 6 2.2.2 Discourse markers in Twitter data 7 2.2.3 Classification of tweets without an overt topic word 8 2.2.4 Summary 9 3 Centering Theory 11 3.1 Overview 11 3.2 Major concepts used in this thesis 13 3.2.1 Uniqueness of the backward-looking center 13 3.2.2 Highest rank of zero pronouns as centers 14 3.2.3 Locality of coherence 15 3.2.4 Preference of center continuation 15 3.3 Summary 16 4 Tweet Serialization and Discourse Segmentation 17 4.1 Tweet Serialization 17 4.1.1 Phenomenon 17 4.1.2 Constraints 21 4.2 Discourse segmentation of serialized tweets 24 4.2.1 Two strategies for discourse segments detection 24 4.2.2 Discourse markers in tweets 28 4.2.3 Algorithm of the SDPS model 36 5 Retrieval of Implicitly Relevant Tweets 39 5.1 Overview: Current search system in Twitter 39 5.2 Data 41 5.2.1 Description 41 5.3 Baselines 43 5.3.1 Query matching model 43 5.3.2 Tf-idf Weighting model 44 5.4 Proposed models 46 5.4.1 Strict serialization Model 46 5.4.2 Serialization Plus Discourse Segmentation Model 47 5.5 Evaluation 49 5.5.1 Measures 49 5.5.2 Results 51 5.6 Discussion 57 5.6.1 Proper formalization of temporal proximity 57 5.6.2 Effects of discourse markers 59 6 Conclusion 61 Bibliography 63 초록 73Maste

Lactic acid bacteria(LAB)세포에 의한 진세노사이드의 전환 연구

학위논문 (석사)-- 서울대학교 대학원 : 식품영양학과, 2012. 2. 지근억.인삼의 성분 중 의학적 및 생물학적 활성을 나타내는 것은 ginsenoside 라고 불리는 인삼 특유의 사포닌이다. 이 연구의 목적은 ginsenoside 를 식품미생물의 효소와 반응시켜 특이적인 deglycosylated ginsenoside 를 생산하고 그 특징을 규명하는 것이었다. 대표적인 식품미생물인 Leuconostoc 과 Lactobacilli 세포의 효소와 반응시켜 major ginsenoside 를 minor ginsenoside 로의 전환을 시도하였으며, 전환 과정 중의 최적의 전환조건을 확인하고자 하였다. 이 중, Leuconostoc mesenteroides KFRI 690, Leuconostoc paramesenteroides KFRI 159, 그리고 Lactobacillus delbruckii KCCM 35486 균주가 생물전환을 통해 compound K를 생산하는 활성을 가지고 있음을 확인하였다. 특히, Leu. mesenteroides KFRI 690 균주의 경우 비배당체 형태의 compound K로의 전환활성이 가장 높음을 확인하였으며, 전환조건의 최적화 실험을 통해 2% sucrose를 첨가하여 배양시킨 후의 효소를 사용하여 반응과정동안 pH 7.0, 37℃의 최적조건에서 전환시켰을 경우 compound K의 생산효율이 가장 높음을 확인하였다. 이 연구는 처음으로 유산균의 파쇄하지 않은 효소액을 이용하여 생물전환 시킨 것에 의미가 있으며, 식품산업에서 잠재적 경제적 가치를 지니고 있다고 보여진다.Various strains of Lactobacillus and Leuconostoc species were evaluated to select the most promising strain to carry out transforming major ginsenosides into minor ginsenosides. Among the experimental lactic acid bacteria(LAB), Leuconostoc mesenteroides KFRI 690, Leuconostoc paramesenteroides KFRI 159, and Lactobacillus delbrueckii KCCM 35486 produced compound K from major ginsenoside precursors (Rb1, Rc, Rd and F2). KFRI 690 showed the best transforming activity among them. Furthermore, these LABs could biotransform ginsenosides without disrupting the cell to release enzyme activity. The production yield of compound K using KFRI 690 has been enhanced by adding 2% sucrose into the culture medium and incubating at pH 7.0 and 37℃ for 96 h during the transformation reaction. This is the first report on the production of compound K using whole cells of Leuconostoc mesenteroides, Leuconostoc paramesenteroides, and Lactobacillus delbrueckii, which are food grade lactic acid bacteria.Maste

X-ray Imaging of wetting ridge on a soft solid

1

Interface Dynamics of Soft Systems Studied by X-ray Imaging

DoctorA “soft system” is a system that consists of “soft materials” which are intermediate between crystalline solids and pure liquids. The soft materials are ubiquitous in our daily life as well as industries, in which a number of biological materials like cells, blood and soft tissues and a wide range of industrial products such as polymer gels, foam, and granular materials are involved. Because the “soft” materials are “mechanically weak” or “easily deformed”, their mechanical responses against various external and internal stimuli have been focused on by many scientists and engineers. In particular, at an “interface” in the soft system, distinctive static and dynamic behaviors, which cannot be understood by classical physics for general solids and liquids, have been observed or expected. However, interface phenomena, especially, interface dynamics, are not fully understood in many soft systems, mostly due to experimental restrictions in observation with conventional optical imaging techniques. In this thesis, the interface dynamics in two soft systems is studied: a droplet on a soft solid and cells in a multicellular organ. X-ray imaging is adapted to directly visualize the interfaces among soft solids, liquid, and vapor and plant cells in static and dynamic situation. In Chapter I, motivation of this thesis and current issues in interface dynamics research, particularly, for soft material systems. To comprehend various interface dynamics, direct visualization of the very interface region is necessary. Besides, the limitation of conventional techniques and the advantage of x-ray imaging techniques are suggested. In Chapter II, general contrast mechanisms in x-ray imaging, i.e. absorption- and phase contrast-based imaging, are explained and then, 3D tomography technique and reconstruction principle is presented. Chapter III and IV contain main results of this thesis about wetting on soft viscoelastic solids are presented. In this research, the heart of issues has been an accurate geometry of a microscopic protrusion formed by vertical component of liquid surface tension at the three-phase contact line, i.e., a “wetting ridge”, over a half of a century. Herein, for the first time, direct visualization of the wetting ridge is successfully achieved using transmission x-ray microscope. In Chapter III, based on the accurate measurement of the geometry of a ridge-tip, universal wetting principle is revealed, which can be applicable to describe general static wetting phenomena on various materials in a wide range of elasticity. Furthermore, in Chapter IV, with high temporal resolution of transmission x-ray microscopy, wetting ridge dynamics in three spreading behaviors is investigated as a pioneering work. In this study, the existence of two ridge growth stages is revealed and then, from real time movies recorded during spreading, pinning enhancement mechanism depending on the ridge geometry and the growth stages is investigated. In Chapter V, as a promising application of x-ray imaging technique to biological fields, a possible quantitative analysis of plant cell growth in multicellular organ is suggested using a 3D fast x-ray tomography technique. In particular, the expansion of individual cells in different cell layers of the root is clearly demonstrated. Interestingly, the individual cells show different anisotropic growth patterns that may generate internal tissue tensions or control the growth pattern of the organ. Finally, in Chapter VI, the total contents of this thesis is summarized.고분자, 유체, 생체 연조직 등을 포함하는 연체 시스템은 그 기계적 강도가 고체와 액체의 중간 정도로 무르기 때문에 다양한 형태로 변형이 가능하다는 특성을 지닌다. 특히 그 계면에서 나타나는 다양한 역동적인 현상들에 대한 이해는 각종 물리적 해석과 그 응용에 있어 중요한 비중을 차지하고 있으나, 기존의 가시광선을 이용한 실험적 접근은 계면의 특성상 나타나는 굴절과 반사 등의 각종 광학 현상으로 인해 직접적인 관찰이 매우 어려웠기 때문에 계면에서 나타나는 현상들에 대한 정확한 이해가 부족하였다. 반면, 굴절률이 매우 작은 엑스선은, 계면에서 나타나는 특유의 작은 굴절을 이용하여 위상 대비를 얻어 계면 영상화에 매우 유리하다. 본 학위 논문에서는 가시광선의 한계를 극복할 수 있는 엑스선 영상 기법을 연체 시스템에 적용함으로써 역동적인 계면 현상에 대한 이해를 높이고자 하였다. 먼저, 연체 상에서 일어나는 젖음 현상에 대한 연구를 수행하였다. 연체의 젖음 현상은 단단한 고체나 액체에 적용 가능한 영의 법칙이나 뉴만의 법칙을 그대로 적용하여 이해할 수 없다. 무른 표면이 액체의 계면 장력에 의해 접촉선이 융기하면서 젖음 주름 (wetting ridge)이라는 독특한 미세 구조물을 형성하기 때문이다. 본 학위 논문에서는 엑스선 영상을 이용하여 최초로 직접 영상화된 젖음 주름을 통해 연체의 정적 젖음 현상의 기본 원리를 밝히고, 나아가 연체 상에서의 독특한 퍼짐 현상에 대한 실시간 관찰을 기반으로 세 가지 퍼짐 현상의 근본 메커니즘을 설명하였다. 두 번째로, 다세포 식물 기관 내에서의 식물 세포의 3차원 성장을 영상화를 목표로 진행하였다. 식물 기관은 여러 세포의 집합으로 각 세포의 성장은 기관의 성장 및 움직임을 조절한다. 특히, 3차원적으로 발생하는 식물의 거동을 올바로 이해하기 위해서는 세포 단위의 3차원 성장 패턴 이해가 필요하다. 하지만 가시광선은 전체 기관을 영상화 하기에 투과성이 낮고 성장하는 기관을 실시간 관찰하기에 시간 해상도가 떨어지기 때문에 식물 기관의 성장을 세포 성장 단위에서 이해하는데 한계가 있었다. 본 학위 논문에서는 애기 장대 뿌리의 성장을 실시간 3차원 영상화 하였으며, 뿌리 내부 각 세포층에 존재하는 개별 세포의 성장을 정량 분석함으로써 세포의 부피팽창 패턴 연구 가능성과 그 중요성을 확인하였다

Modality-based Sentiment Analysis through the Utilization of the Korean Sentiment Analysis Corpus

This study develops a practical application of language resources from the Korean Sentiment Analysis Corpus (KOSAC) for sentiment analysis research. With this in mind, based on their sentiment properties and the probabilistic factors of annotated expressions from KOSAC, we extracted annotated expressions and refined them to be a sentiment analysis research resource. This study attempted to break away from simple calculation methods dependant on the distribution of lexical polarity items seen in previous research. Additionally, in order to perform more sophisticated sentiment analysis, we attempted to introduce pragmatic information which includes modality. In order to achieve this, we cataloged expressions that include pragmatic information related to the speaker's attitude, based on their relative probability in KOSAC. After doing so, this study shows a practical application of this new language resource to subjectivity analysis research. When using this new resource, this research demonstrates an accuracy improvement of around 6%. This demonstrates very clearly that, in addition to polarity items, there exists a need to include a variety of aspects and lexical information when doing this type of research. Moreover, this extraction of sentiment expressions, depending on their semantic and pragmatic properties, not only shows an additional use of KOSAC, but also establishes a new resource in the field of sentiment analysis.N

A Small-Scale Korean-Specific BERT Language Model

최근 자연어처리에서 문장 단위의 임베딩을 위한 모델들은 거대한 말뭉치와 파라미터를 이용하기 때문에 큰 하드웨어와 데이터를 요구하고 학습하는 데 시간이 오래 걸린다는 단점을 갖는다. 따라서 규모가 크지 않더라도 학습 데이터를 경제적으로 활용하면서 필적할만한 성능을 가지는 모델의 필요성이 제기된다. 본 연구는 음절 단위의 한국어 사전, 자소 단위의 한국어 사전을 구축하고 자소 단위의 학습과 양방향 WordPiece 토크나이저를 새롭게 소개하였다. 그 결과 기존 모델의 1/10 사이즈의 학습 데이터를 이용하고 적절한 크기의 사전을 사용해 더 적은 파라미터로 계산량은 줄고 성능은 비슷한 KR-BERT 모델을 구현할 수 있었다. 이로써 한국어와 같이 고유의 문자 체계를 가지고 형태론적으로 복잡하며 자원이 적은 언어에 대해 모델을 구축할 때는 해당 언어에 특화된 언어학적 현상을 반영해야 한다는 것을 확인하였다. Recent models for the sentence embedding use huge corpus and parameters. They have massive data and large hardware and it incurs extensive time to pre-train. This tendency raises the need for a model with comparable performance while economically using training data. In this study, we proposed a Korean-specific model KR-BERT, using sub-character level to character-level Korean dictionaries and BidirectionalWordPiece Tokenizer. As a result, our KR-BERT model performs comparably and even better than other existing pre-trained models using one-tenth the size of training data from the existing models. It demonstrates that in a morphologically complex and resourceless language, using sub-character level and BidirectionalWordPiece Tokenizer captures language-specific linguistic phenomena that the Multilingual BERT model missed.N