The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG : Overview and Results

Publisher PD

Non-Repeatable Experiments and Non-Reproducible Results : The Reproducibility Crisis in Human Evaluation in NLP

Publisher PD

Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models

While large language models (LLMs) have demonstrated remarkable capabilities across a range of downstream tasks, a significant concern revolves around their propensity to exhibit hallucinations: LLMs occasionally generate content that diverges from the user input, contradicts previously generated context, or misaligns with established world knowledge. This phenomenon poses a substantial challenge to the reliability of LLMs in real-world scenarios. In this paper, we survey recent efforts on the detection, explanation, and mitigation of hallucination, with an emphasis on the unique challenges posed by LLMs. We present taxonomies of the LLM hallucination phenomena and evaluation benchmarks, analyze existing approaches aiming at mitigating LLM hallucination, and discuss potential directions for future research.Comment: work in progress; 32 page

Scenario pedagogy as a negotiated, multimodal approach to developing professional communication practices in higher education

Includes bibliographical references.The focus of this study is pedagogy - the 'how' of teaching. In particular, a negotiated and multimodal pedagogical approach which I have coined scenario pedagogy is of interest. Scenario pedagogy involves embedding an entire curriculum into a topical and authentic scenario, relevant to a particular group of students in higher education. The course in question is professional communication and the target group comprises senior and post-graduate accounting and other finance and information systems students registered in the commerce faculty. They are not communication students per se but register for a one-semester professional communication course towards their respective commerce degrees. In this study I examine how these students develop their professional communication practices using a wide variety of verbal and visual semiotic resources. Their selection of hybrid discursive, generic and modal resources are foregrounded at both draft and final product stage and include their communicative processes as well as the material artefacts they deliver in class. How students instantiate their meaning making and emerging identity as professionals-to-be is highlighted against a pedagogical framework of negotiated design. This framework combines a multiliteracies cum multimodal perspective which is underpinned by the notion of transformed practice. As pivotal elements of transformation - personally, collectively and societally - education and communication play significant roles, particularly in post-Apartheid South Africa still characterised by enormous socio-economic disparities and disadvantage

Systemic-functional linguistics and computation : New directions, new challenges

Peer reviewe

Evaluative meaning in scientific writing : macro- and micro-analytic perspectives using data mining

In this thesis, we elaborate characteristics of evaluative meaning of different scientific disciplines and trace their diachronic linguistic evolution. A main focus lies on newly emerged disciplines, such as computational linguistics, which emerged through contact between two other disciplines, such as computer science and linguistics. Here, we consider (1) whether these newly emerged disciplines have created characteristics of their own over time, showing a process of diversification, and (2) whether they have also adopted characteristics from their disciplines of origin, reflected in a linguistic imprint, and if this might have changed over time. The newly emerged disciplines considered are computational linguistics, bioinformatics, digital construction and microelectronics, which have emerged through contact between computer science and a further discipline (linguistics, biology, mechanical engineering, and electrical engineering, respectively). In terms of theory, this work is grounded in a linguistic theory rooted in sociolinguistics, Systemic Functional Linguistics (SFL; Halliday, 2004), which with its functional perspective on language allowed us to position evaluative meaning within a linguistic theory and to create a model of analysis to trace choices made in the semantic system on the level of lexico-grammar. Moreover, its notion of register, concerned with functional variation, i.e. variation according to language use, combined with the sociolinguistic perspective made it possible to compare the linguistic choices made according to different social contexts, to which the disciplines belong. This allowed us to trace register diversification processes and registerial imprint of evaluative meaning across disciplines. In terms of methods, we apply classification as a data mining technique, taking a macro- and micro-analytic perspective (cf. Jockers, 2013) on the results. Doing so we gain insights on the degree of diversification and imprint (macro-analysis) and the kind of diversification and imprint (micro-analysis). Studies so far have considered either the macro- or the micro-analytic perspective. By considering both, we are able to investigate generalizable trends as well as detailed linguistic characteristics of evaluative meaning across disciplines and time. The approach presented in this thesis draws its strength from being grounded in a linguistic theory, which proved to be extremely useful in defining and testing hypotheses and interpreting results. Moreover, an empirical analysis of evaluative meaning across disciplines and time was possible by combining corpus-based methods with data mining techniques.In der vorliegenden Dissertation werden Bewertungscharakteristiken verschiedener Wissenschaftsdisziplinen erarbeitet und ihre diachrone linguistische Entwicklung untersucht. Ein Hauptfokus liegt auf in neuerer Zeit entstandenen Disziplinen (z. B. Computerlinguistik), die sich durch Kontakt zwischen zwei anderen Disziplinen gebildet haben (z. B. Informatik und Linguistik). In diesem Zusammenhang wird erforscht, (1) ob diese neu entstandenen Disziplinen diachron ihre eigenen Charakteristiken entwickeln und somit einen Diversifikationsprozess aufzeigen und (2) ob sie auch Charakteristiken der Ursprungsdisziplinen übernehmen und somit eine linguistische Prägung aus der Ursprungsdisziplin vorweisen und ob sich diese möglicherweise diachron verändert hat. Die untersuchten relativ neu entstandenen Disziplinen sind die Computerlinguistik, Bioinformatik, Bauinformatik und Mikroelektronik, die durch Kontakt zwischen der Informatik und einer anderen Disziplin entstanden sind, in unserem Fall entsprechend aus der Linguistik, Biologie, dem Maschinenbau und der Elektrotechnik. Die Arbeit basiert auf der soziolinguistischen Theorie der Systemisch Funktionalen Linguistik (SFL; Halliday (2004)). Aufgrund ihrer funktionalen Perspektive auf die Sprache war es uns möglich, das semantische Konzept der Bewertung in eine linguistische Theorie zu positionieren und ein Analysemodel zu entwickeln, um die Auswahl aus dem semantischen System auf der lexicogrammatischen Ebene nachzuverfolgen. Besonders wichtig ist hierbei auch das Registerkonzept aus der SFL, das sich mit funktionaler Variation befasst, d.h. Variation in Bezug auf den Sprachgebrauch. Die Kombination aus funktionaler Variation und soziolinguistischer Perspektive hat es erlaubt, die linguistischen Entscheidungen in Bezug auf Bewertungen, die in unterschiedlichen sozialen Kontexten (d.h. den verschiedenen Disziplinen) gefällt wurden, zu untersuchen und diese zu vergleichen. Dadurch konnten für die untersuchten Disziplinen registerspezifische Diversifikationsprozesse und Prägungen bezüglich Bewertungen ausgemacht werden. Methodisch wurde aus dem Bereich des Data Mining die Klassifikation angewandt, die es erlaubt hat, die Ergebnisse aus einer makro- und mikro-analytischen Perspektive (vgl. Jockers (2013)) zu erforschen. Dadurch konnten Erkenntnisse erlangt werden in Bezug auf den Diversifikations- und Prägungsgrad (Makro-Analyse) sowie der Art der Diversifikation und Prägung (Mikro-Analyse). Studien haben bislang entweder die makro- oder die mikro-analytische Perspektive angewandt. Durch den Einbezug beider Ebenen ist es uns gelungen, sowohl generalisierbare Tendenzen festzustellen als auch detaillierte linguistische Charakteristiken und diachrone Veränderungen von Bewertungsausdrücken in verschiedenen Disziplinen zu untersuchen. Die Stärken des in der vorliegenden Dissertation präsentierten Ansatzes liegen darin, dass er in einer linguistischen Theorie fundiert ist, die sich sehr hilfreich erwiesen hat bei der Hypothesenaufstellung und beim Testen der Hypothesen sowie auch bei der Interpretation der Ergebnisse. Darüber hinaus hat der Ansatz eine empirische Analyse von Bewertungen in wissenschaftlichen Disziplinen durch das Zusammenspiel von korpus-basierten Methoden und Techniken aus dem Data Mining ermöglicht

Evaluative meaning in scientific writing : macro- and micro-analytic perspectives using data mining

In this thesis, we elaborate characteristics of evaluative meaning of different scientific disciplines and trace their diachronic linguistic evolution. A main focus lies on newly emerged disciplines, such as computational linguistics, which emerged through contact between two other disciplines, such as computer science and linguistics. Here, we consider (1) whether these newly emerged disciplines have created characteristics of their own over time, showing a process of diversification, and (2) whether they have also adopted characteristics from their disciplines of origin, reflected in a linguistic imprint, and if this might have changed over time. The newly emerged disciplines considered are computational linguistics, bioinformatics, digital construction and microelectronics, which have emerged through contact between computer science and a further discipline (linguistics, biology, mechanical engineering, and electrical engineering, respectively). In terms of theory, this work is grounded in a linguistic theory rooted in sociolinguistics, Systemic Functional Linguistics (SFL; Halliday, 2004), which with its functional perspective on language allowed us to position evaluative meaning within a linguistic theory and to create a model of analysis to trace choices made in the semantic system on the level of lexico-grammar. Moreover, its notion of register, concerned with functional variation, i.e. variation according to language use, combined with the sociolinguistic perspective made it possible to compare the linguistic choices made according to different social contexts, to which the disciplines belong. This allowed us to trace register diversification processes and registerial imprint of evaluative meaning across disciplines. In terms of methods, we apply classification as a data mining technique, taking a macro- and micro-analytic perspective (cf. Jockers, 2013) on the results. Doing so we gain insights on the degree of diversification and imprint (macro-analysis) and the kind of diversification and imprint (micro-analysis). Studies so far have considered either the macro- or the micro-analytic perspective. By considering both, we are able to investigate generalizable trends as well as detailed linguistic characteristics of evaluative meaning across disciplines and time. The approach presented in this thesis draws its strength from being grounded in a linguistic theory, which proved to be extremely useful in defining and testing hypotheses and interpreting results. Moreover, an empirical analysis of evaluative meaning across disciplines and time was possible by combining corpus-based methods with data mining techniques.In der vorliegenden Dissertation werden Bewertungscharakteristiken verschiedener Wissenschaftsdisziplinen erarbeitet und ihre diachrone linguistische Entwicklung untersucht. Ein Hauptfokus liegt auf in neuerer Zeit entstandenen Disziplinen (z. B. Computerlinguistik), die sich durch Kontakt zwischen zwei anderen Disziplinen gebildet haben (z. B. Informatik und Linguistik). In diesem Zusammenhang wird erforscht, (1) ob diese neu entstandenen Disziplinen diachron ihre eigenen Charakteristiken entwickeln und somit einen Diversifikationsprozess aufzeigen und (2) ob sie auch Charakteristiken der Ursprungsdisziplinen übernehmen und somit eine linguistische Prägung aus der Ursprungsdisziplin vorweisen und ob sich diese möglicherweise diachron verändert hat. Die untersuchten relativ neu entstandenen Disziplinen sind die Computerlinguistik, Bioinformatik, Bauinformatik und Mikroelektronik, die durch Kontakt zwischen der Informatik und einer anderen Disziplin entstanden sind, in unserem Fall entsprechend aus der Linguistik, Biologie, dem Maschinenbau und der Elektrotechnik. Die Arbeit basiert auf der soziolinguistischen Theorie der Systemisch Funktionalen Linguistik (SFL; Halliday (2004)). Aufgrund ihrer funktionalen Perspektive auf die Sprache war es uns möglich, das semantische Konzept der Bewertung in eine linguistische Theorie zu positionieren und ein Analysemodel zu entwickeln, um die Auswahl aus dem semantischen System auf der lexicogrammatischen Ebene nachzuverfolgen. Besonders wichtig ist hierbei auch das Registerkonzept aus der SFL, das sich mit funktionaler Variation befasst, d.h. Variation in Bezug auf den Sprachgebrauch. Die Kombination aus funktionaler Variation und soziolinguistischer Perspektive hat es erlaubt, die linguistischen Entscheidungen in Bezug auf Bewertungen, die in unterschiedlichen sozialen Kontexten (d.h. den verschiedenen Disziplinen) gefällt wurden, zu untersuchen und diese zu vergleichen. Dadurch konnten für die untersuchten Disziplinen registerspezifische Diversifikationsprozesse und Prägungen bezüglich Bewertungen ausgemacht werden. Methodisch wurde aus dem Bereich des Data Mining die Klassifikation angewandt, die es erlaubt hat, die Ergebnisse aus einer makro- und mikro-analytischen Perspektive (vgl. Jockers (2013)) zu erforschen. Dadurch konnten Erkenntnisse erlangt werden in Bezug auf den Diversifikations- und Prägungsgrad (Makro-Analyse) sowie der Art der Diversifikation und Prägung (Mikro-Analyse). Studien haben bislang entweder die makro- oder die mikro-analytische Perspektive angewandt. Durch den Einbezug beider Ebenen ist es uns gelungen, sowohl generalisierbare Tendenzen festzustellen als auch detaillierte linguistische Charakteristiken und diachrone Veränderungen von Bewertungsausdrücken in verschiedenen Disziplinen zu untersuchen. Die Stärken des in der vorliegenden Dissertation präsentierten Ansatzes liegen darin, dass er in einer linguistischen Theorie fundiert ist, die sich sehr hilfreich erwiesen hat bei der Hypothesenaufstellung und beim Testen der Hypothesen sowie auch bei der Interpretation der Ergebnisse. Darüber hinaus hat der Ansatz eine empirische Analyse von Bewertungen in wissenschaftlichen Disziplinen durch das Zusammenspiel von korpus-basierten Methoden und Techniken aus dem Data Mining ermöglicht

Upward Bound is College Bound : Pre-College Outreach Programs\u27 Sponsorship of Academic Writing

This study examines how Upward Bound (UB), a federally-funded pre-college outreach program for underrepresented students, impacted participants\u27 access to academic writing and higher education. Based on the perspective that writing practices both reflect and shape identity, I constructed a series of case studies that followed five linguistically and culturally diverse students from the Upward Bound summer program to determine how this intervention impacted their identifications with academic writing during their senior year of high school and the college admissions process. This year-long qualitative study used transcribed student interviews and focus groups along with writing samples as primary data sources. These sources were then triangulated with visual artifacts, institutional documents, fieldnotes from observations in the high schools and UB, and transcribed interviews with key informants, such as teachers and advisors. Based on themes emerging within and across the case studies, I turned to the following theoretical perspectives to analyze the sociocultural influences on participants\u27 writing practices and identities: difference as resource, Communities of Practice, performance, and literacy sponsorship. As analytical frameworks, these theories allowed me to analyze how the unique and often conflicting writing practices participants experienced in their homes, high schools, and UB shaped their writerly identities and educational trajectories. Findings revealed that UB provided a sort of liminal time and space, betwixt and between high school and college, home and school. Due to this unique context, the practices and identities participants developed within the program did not always transfer to the writing they experienced during high school and the admissions process. For instance, the participants found that the authentic writing and social supports they experienced within the program helped them to gain confidence, motivation, and rhetorical awareness as writers. However, limited resources and tracking within their high schools disrupted these trajectories, resulting in a loss of confidence and motivation. Similarly, participants were able to draw on linguistic and cultural resources to write their college admissions essay. However, over the course of the admissions process, the gatekeeping function of college admissions and scholarship essays turned these cultural connections to conflicts. This study bears significant implications for several groups of educators. It suggests that traditional educational institutions and pre-college outreach programs could benefit from working together in order to promote transfer of learning and to better support the many students not served by these programs. It also suggests that literacy reforms need to establish opportunity-to-learn standards in order to give districts, schools, and teachers the material resources they need to help all students achieve the same outcomes. English educators and high school writing teachers can use insights from this study to create safe communities of writers and to use information about students\u27 needs, strengths, and interests to develop authentic and culturally-relevant writing experiences. Finally, writing program administrators and first-year writing instructors should familiarize themselves with their students\u27 high school writing experiences and avoid perpetuating the stigmatizing institutional policies and identities they may have experienced there

Applications of Discourse Structure for Spoken Dialogue Systems

Language exhibits structure beyond the sentence level (e.g. the syntactic structure of a sentence). In particular, dialogues, either human-human or human-computer, have an inherent structure called the discourse structure. Models of discourse structure attempt to explain why a sequence of random utterances combines to form a dialogue or no dialogue at all. Due to the relatively simple structure of the dialogues that occur in the information-access domains of typical spoken dialogue systems (e.g. travel planning), discourse structure has often seen limited application in such systems. In this research, we investigate the utility of discourse structure for spoken dialogue systems in more complex domains, e.g. tutoring. This work was driven by two intuitions.First, we believed that the "position in the dialogue" is a critical information source for two tasks: performance analysis and characterization of dialogue phenomena. We define this concept using transitions in the discourse structure. For performance analysis, these transitions are used to create a number of novel factors which we show to be predictive of system performance. One of these factors informs a promising modification of our system which is implemented and compared with the original version of the system through a user study. Results show that the modification leads to objective improvements. For characterization of dialogue phenomena, we find statistical dependencies between discourse structure transitions and two dialogue phenomena which allow us to speculate where and why these dialogue phenomena occur and to better understand system behavior.Second, we believed that users will benefit from direct access to discourse structure information. We enable this through a graphical representation of discourse structure called the Navigation Map. We demonstrate the subjective and objective utility of the Navigation Map through two user studies.Overall, our work demonstrates that discourse structure is an important information source for designers of spoken dialogue systems