Tagging Prosody and Discourse Structure in Elicited Spontaneous Speech

This paper motivates and describes the annotation and analysis of prosody and discourse structure for several large spoken language corpora. The annotation schema are of two types: tags for prosody and intonation, and tags for several aspects of discourse structure. The choice of the particular tagging schema in each domain is based in large part on the insights they provide in corpus-based studies of the relationship between discourse structure and the accenting of referring expressions in American English. We first describe these results and show that the same models account for the accenting of pronouns in an extended passage from one of the Speech Warehouse hotel-booking dialogues. We then turn to corpora described in Venditti [Ven00], which adapts the same models to Tokyo Japanese. Japanese is interesting to compare to English, because accent is lexically specified and so cannot mark discourse focus in the same way. Analyses of these corpora show that local pitch range expansion serves the analogous focusing function in Japanese. The paper concludes with a section describing several outstanding questions in the annotation of Japanese intonation which corpus studies can help to resolve.Work reported in this paper was supported in part by a grant from the Ohio State University Office of Research, to Mary E. Beckman and co-principal investigators on the OSU Speech Warehouse project, and by an Ohio State University Presidential Fellowship to Jennifer J. Venditti

On the Multiple Clause Linkage Structure of Japanese: A Corpus-based Study

In this paper, we will describe the distribution of the multiple clause linkage structure within actual spoken and written Japanese. We will examine three Japanese corpora: BCCWJ, CSJ and OCOJ. By identifying distributions of multiple clause linkage structures in corpora of contemporary Japanese (BCCWJ and CSJ), we shed light on what kinds of settings give rise to what type of clause linkage structures through what processes. The dynamic rewriting rule proposed by Kondo (2005) is introduced as a model for the incremental production of multiple clause linkage structures. Some common patterns of such structures occurring in Old Japanese are identified by OCOJ and compared to patterns in BCCWJ and CSJ

A Corpus-Based Comparison of Syntactic Complexity in Spoken and Written Learner Language

Despite writing and speaking being related activities, their end-products are entirely different. However, previous studies have not shown consistency in terms of grammar use in these two modes. Accordingly, in the present study, I aim to define the syntactic characteristics in these two modes with large-scale data and organized research designs. This study examined 14 indices of syntactic complexity and specific grammar factors in 224 monologues and 139 writings of Korean EFL undergraduates. The results revealed that learners tended to use more finite complement clauses and relative clauses while writing but used because- fragments independently and ‘and’ sentence-initially more frequently while speaking. When compared with previous studies, the characteristics of syntactic complexity of Korean EFL learners, regardless of age, are defined by the use of coordination in speaking and the use of subordination in writing.  L’écrit et l’oral sont des activités clairement liées, mais le résultat final est tout à fait différent.Toutefois, des études antérieures n'ont pas montrées de cohérence dans l'utilisation de la grammaire dans les deux modes. Par conséquent, dans la présente étude, le but est de définir les caractéristiques syntaxiques des deux modes avec des données à grande échelle et des plans de recherche organisés. Cette étude a examiné 14 indices de complexité syntaxique et des facteurs grammaticaux spécifiques dans 224 monologues et 139 écrits d'étudiants coréens de premier cycle EFL. Les résultats ont révélé que les apprenants ont tendance à utiliser des clauses complémentaires limitées et des clauses relatives lorsqu'ils écrivent, mais qu'ils utilisent les fragments ‘parce que’ de manière indépendante et les fragments ‘et’ en début de phrase plus fréquemment à l’oral. En comparaison des études précédentes, les caractéristiques de la complexité syntaxique des apprenants coréens de l'EFL, quel que soit leur âge, sont définies par l'utilisation de la conjonction de coordination dans la parole à l’oral et de la conjonction de subordination par à l’écrit

The COPLE2 Corpus: a Learner Corpus for Portuguese

We present the COPLE2 corpus, a learner corpus of Portuguese that includes written and spoken texts produced by learners of Portuguese as a second or foreign language. The corpus includes at the moment a total of 182,474 tokens and 978 texts, classified according to the CEFR scales. The original handwritten productions are transcribed in TEI compliant XML format and keep record of all the original information, such as reformulations, insertions and corrections made by the teacher, while the recordings are transcribed and aligned with EXMARaLDA. The TEITOK environment enables different views of the same document (XML, student version, corrected version), a CQP-based search interface, the POS, lemmatization and normalization of the tokens, and will soon be used for error annotation in stand-off format. The corpus has already been a source of data for phonological, lexical and syntactic interlanguage studies and will be used for a data-informed selection of language features for each proficiency level.info:eu-repo/semantics/publishedVersio

Towards error annotation in a learner corpus of Portuguese

In this article, we present COPLE2, a new corpus of Portuguese that encompasses written and spoken data produced by foreign learners of Portuguese as a foreign or second language (FL/L2). Following the trend towards learner corpus research applied to less commonly taught languages, it is our aim to enhance the learning data of Portuguese L2. These data may be useful not only for educational purposes (design of learning materials, curricula, etc.) but also for the development of NLP tools to support students in their learning process. The corpus is available online using TEITOK environment, a web-based framework for corpus treatment that provides several built-in NLP tools and a rich set of functionalities (multiple orthographic transcription layers, lemmatization and POS, normalization of the tokens, error annotation) to automatically process and annotate texts in xml format. A CQP-based search interface allows searching the corpus for different fields, such as words, lemmas, POS tags or error tags. We will describe the work in progress regarding the constitution and linguistic annotation of this corpus, particularly focusing on error annotation.info:eu-repo/semantics/publishedVersio

Coordinating in dialogue: Using compound contributions to join a party

PhDCompound contributions (CCs) – dialogue contributions that continue or complete an earlier contribution – are an important and common device conversational participants use to extend their own and each other’s turns. The organisation of these cross-turn structures is one of the defining characteristics of natural dialogue, and cross-person CCs provide the paradigm case of coordination in dialogue. This thesis combines corpus analysis, experiments and theoretical modelling to explore how CCs are used, their effects on coordination and implications for dialogue models. The syntactic and pragmatic distribution of CCs is mapped using corpora of ordinary and task-oriented dialogues. This indicates that the principal factors conditioning the distribution of CCs are pragmatic and that same- and cross-person CCs tend to occur in different contexts. In order to test the impact of CCs on other conversational participants, two experiments are presented. These systematically manipulate, for the first time, the occurrence of CCs in live dialogue using text-based communication. The results suggest that syntax does not directly constrain the interpretation of CCs, and the primary effect of a cross-person CC on third parties is to suggest to them a strong form of coordination or coalition has formed between the people producing the two parts of the CC. A third experiment explores the conditions under which people will produce a completion for a truncated turn. Manipulations of the structural and contextual predictability of the truncated turn show that while syntax provides a resource for the construction of a CC it does not place significant constraints on where the split point may occur. It also shows that people are more likely to produce continuations when they share common ground. An analysis using the Dynamic Syntax framework is proposed, which extends previous work to account for these findings, and limitations and further research possibilities are outlined

음성언어 이해에서의 중의성 해소

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2022. 8. 김남수.언어의 중의성은 필연적이다. 그것은 언어가 의사 소통의 수단이지만, 모든 사람이 생각하는 어떤 개념이 완벽히 동일하게 전달될 수 없는 것에 기인한다. 이는 필연적인 요소이기도 하지만, 언어 이해에서 중의성은 종종 의사 소통의 단절이나 실패를 가져오기도 한다. 언어의 중의성에는 다양한 층위가 존재한다. 하지만, 모든 상황에서 중의성이 해소될 필요는 없다. 태스크마다, 도메인마다 다른 양상의 중의성이 존재하며, 이를 잘 정의하고 해소될 수 있는 중의성임을 파악한 후 중의적인 부분 간의 경계를 잘 정하는 것이 중요하다. 본고에서는 음성 언어 처리, 특히 의도 이해에 있어 어떤 양상의 중의성이 발생할 수 있는지 알아보고, 이를 해소하기 위한 연구를 진행한다. 이러한 현상은 다양한 언어에서 발생하지만, 그 정도 및 양상은 언어에 따라서 다르게 나타나는 경우가 많다. 우리의 연구에서 주목하는 부분은, 음성 언어에 담긴 정보량과 문자 언어의 정보량 차이로 인해 중의성이 발생하는 경우들이다. 본 연구는 운율(prosody)에 따라 문장 형식 및 의도가 다르게 표현되는 경우가 많은 한국어를 대상으로 진행된다. 한국어에서는 다양한 기능이 있는(multi-functional한) 종결어미(sentence ender), 빈번한 탈락 현상(pro-drop), 의문사 간섭(wh-intervention) 등으로 인해, 같은 텍스트가 여러 의도로 읽히는 현상이 발생하곤 한다. 이것이 의도 이해에 혼선을 가져올 수 있다는 데에 착안하여, 본 연구에서는 이러한 중의성을 먼저 정의하고, 중의적인 문장들을 감지할 수 있도록 말뭉치를 구축한다. 의도 이해를 위한 말뭉치를 구축하는 과정에서 문장의 지향성(directivity)과 수사성(rhetoricalness)이 고려된다. 이것은 음성 언어의 의도를 서술, 질문, 명령, 수사의문문, 그리고 수사명령문으로 구분하게 하는 기준이 된다. 본 연구에서는 기록된 음성 언어(spoken language)를 충분히 높은 일치도(kappa = 0.85)로 주석한 말뭉치를 이용해, 음성이 주어지지 않은 상황에서 중의적인 텍스트를 감지하는 데에 어떤 전략 혹은 언어 모델이 효과적인가를 보이고, 해당 태스크의 특징을 정성적으로 분석한다. 또한, 우리는 텍스트 층위에서만 중의성에 접근하지 않고, 실제로 음성이 주어진 상황에서 중의성 해소(disambiguation)가 가능한지를 알아보기 위해, 텍스트가 중의적인 발화들만으로 구성된 인공적인 음성 말뭉치를 설계하고 다양한 집중(attention) 기반 신경망(neural network) 모델들을 이용해 중의성을 해소한다. 이 과정에서 모델 기반 통사적/의미적 중의성 해소가 어떠한 경우에 가장 효과적인지 관찰하고, 인간의 언어 처리와 어떤 연관이 있는지에 대한 관점을 제시한다. 본 연구에서는 마지막으로, 위와 같은 절차로 의도 이해 과정에서의 중의성이 해소되었을 경우, 이를 어떻게 산업계 혹은 연구 단에서 활용할 수 있는가에 대한 간략한 로드맵을 제시한다. 텍스트에 기반한 중의성 파악과 음성 기반의 의도 이해 모듈을 통합한다면, 오류의 전파를 줄이면서도 효율적으로 중의성을 다룰 수 있는 시스템을 만들 수 있을 것이다. 이러한 시스템은 대화 매니저(dialogue manager)와 통합되어 간단한 대화(chit-chat)가 가능한 목적 지향 대화 시스템(task-oriented dialogue system)을 구축할 수도 있고, 단일 언어 조건(monolingual condition)을 넘어 음성 번역에서의 에러를 줄이는 데에 활용될 수도 있다. 우리는 본고를 통해, 운율에 민감한(prosody-sensitive) 언어에서 의도 이해를 위한 중의성 해소가 가능하며, 이를 산업 및 연구 단에서 활용할 수 있음을 보이고자 한다. 본 연구가 다른 언어 및 도메인에서도 고질적인 중의성 문제를 해소하는 데에 도움이 되길 바라며, 이를 위해 연구를 진행하는 데에 활용된 리소스, 결과물 및 코드들을 공유함으로써 학계의 발전에 이바지하고자 한다.Ambiguity in the language is inevitable. It is because, albeit language is a means of communication, a particular concept that everyone thinks of cannot be conveyed in a perfectly identical manner. As this is an inevitable factor, ambiguity in language understanding often leads to breakdown or failure of communication. There are various hierarchies of language ambiguity. However, not all ambiguity needs to be resolved. Different aspects of ambiguity exist for each domain and task, and it is crucial to define the boundary after recognizing the ambiguity that can be well-defined and resolved. In this dissertation, we investigate the types of ambiguity that appear in spoken language processing, especially in intention understanding, and conduct research to define and resolve it. Although this phenomenon occurs in various languages, its degree and aspect depend on the language investigated. The factor we focus on is cases where the ambiguity comes from the gap between the amount of information in the spoken language and the text. Here, we study the Korean language, which often shows different sentence structures and intentions depending on the prosody. In the Korean language, a text is often read with multiple intentions due to multi-functional sentence enders, frequent pro-drop, wh-intervention, etc. We first define this type of ambiguity and construct a corpus that helps detect ambiguous sentences, given that such utterances can be problematic for intention understanding. In constructing a corpus for intention understanding, we consider the directivity and rhetoricalness of a sentence. They make up a criterion for classifying the intention of spoken language into a statement, question, command, rhetorical question, and rhetorical command. Using the corpus annotated with sufficiently high agreement on a spoken language corpus, we show that colloquial corpus-based language models are effective in classifying ambiguous text given only textual data, and qualitatively analyze the characteristics of the task. We do not handle ambiguity only at the text level. To find out whether actual disambiguation is possible given a speech input, we design an artificial spoken language corpus composed only of ambiguous sentences, and resolve ambiguity with various attention-based neural network architectures. In this process, we observe that the ambiguity resolution is most effective when both textual and acoustic input co-attends each feature, especially when the audio processing module conveys attention information to the text module in a multi-hop manner. Finally, assuming the case that the ambiguity of intention understanding is resolved by proposed strategies, we present a brief roadmap of how the results can be utilized at the industry or research level. By integrating text-based ambiguity detection and speech-based intention understanding module, we can build a system that handles ambiguity efficiently while reducing error propagation. Such a system can be integrated with dialogue managers to make up a task-oriented dialogue system capable of chit-chat, or it can be used for error reduction in multilingual circumstances such as speech translation, beyond merely monolingual conditions. Throughout the dissertation, we want to show that ambiguity resolution for intention understanding in prosody-sensitive language can be achieved and can be utilized at the industry or research level. We hope that this study helps tackle chronic ambiguity issues in other languages ​​or other domains, linking linguistic science and engineering approaches.1 Introduction 1 1.1 Motivation 2 1.2 Research Goal 4 1.3 Outline of the Dissertation 5 2 Related Work 6 2.1 Spoken Language Understanding 6 2.2 Speech Act and Intention 8 2.2.1 Performatives and statements 8 2.2.2 Illocutionary act and speech act 9 2.2.3 Formal semantic approaches 11 2.3 Ambiguity of Intention Understanding in Korean 14 2.3.1 Ambiguities in language 14 2.3.2 Speech act and intention understanding in Korean 16 3 Ambiguity in Intention Understanding of Spoken Language 20 3.1 Intention Understanding and Ambiguity 20 3.2 Annotation Protocol 23 3.2.1 Fragments 24 3.2.2 Clear-cut cases 26 3.2.3 Intonation-dependent utterances 28 3.3 Data Construction . 32 3.3.1 Source scripts 32 3.3.2 Agreement 32 3.3.3 Augmentation 33 3.3.4 Train split 33 3.4 Experiments and Results 34 3.4.1 Models 34 3.4.2 Implementation 36 3.4.3 Results 37 3.5 Findings and Summary 44 3.5.1 Findings 44 3.5.2 Summary 45 4 Disambiguation of Speech Intention 47 4.1 Ambiguity Resolution 47 4.1.1 Prosody and syntax 48 4.1.2 Disambiguation with prosody 50 4.1.3 Approaches in SLU 50 4.2 Dataset Construction 51 4.2.1 Script generation 52 4.2.2 Label tagging 54 4.2.3 Recording 56 4.3 Experiments and Results 57 4.3.1 Models 57 4.3.2 Results 60 4.4 Summary 63 5 System Integration and Application 65 5.1 System Integration for Intention Identification 65 5.1.1 Proof of concept 65 5.1.2 Preliminary study 69 5.2 Application to Spoken Dialogue System 75 5.2.1 What is 'Free-running' 76 5.2.2 Omakase chatbot 76 5.3 Beyond Monolingual Approaches 84 5.3.1 Spoken language translation 85 5.3.2 Dataset 87 5.3.3 Analysis 94 5.3.4 Discussion 95 5.4 Summary 100 6 Conclusion and Future Work 103 Bibliography 105 Abstract (In Korean) 124 Acknowledgment 126박

Defective connective constructions: Some cases in Catalan and Spanish

Connectives typically relate two content units. However, corpus analysis shows several variants of the general connective construction (i.e., 'S1 Cn S2'), in which one of either segment 1 (S1) or segment 2 (S2) is optional or missing. The aim of this paper is to shed some light on the description of some variants of the connective construction where the connective is not followed by any explicit S2 or S2 is optional. These constructions are complete utterances but they can be considered defective constructions, since one of the slots of the prototypical construction does not include any linguistic material. The analysis focuses on corpus examples including a refutation marker where S2 is implicit, a case that is especially productive and varied in Catalan and in Spanish. Three defective constructions are identified, namely, (i) truncated constructions, (ii) embedded uses of a connective and (ii) reactive constructions. The data show that these defective connective constructions differ as for syntax, prosody, semantics and pragmatics. In monologic contexts, when the second segment is missing in the syntactic and prosodic unit considered, the connective is syntactically and prosodically related to S1. The connective can be located at the right-periphery of S1 (truncated construction) or at S1 middle field (embedded use of a connective). In dialogic contexts, the connective can act as a response to a previous turn and S2 can be either present or absent (reactive constructions). The different configurations match different intonation contours and pause patterns. In all cases, the connective weakens its connective function and adds a modal load, related to (inter)subjectification and intensification. This can be represented as a cline from discourse marking to modal marking