The economic trade-offs of large language models: A case study

Contacting customer service via chat is a common practice. Because employing customer service agents is expensive, many companies are turning to NLP that assists human agents by auto-generating responses that can be used directly or with modifications. Large Language Models (LLMs) are a natural fit for this use case; however, their efficacy must be balanced with the cost of training and serving them. This paper assesses the practical cost and impact of LLMs for the enterprise as a function of the usefulness of the responses that they generate. We present a cost framework for evaluating an NLP model's utility for this use case and apply it to a single brand as a case study in the context of an existing agent assistance product. We compare three strategies for specializing an LLM - prompt engineering, fine-tuning, and knowledge distillation - using feedback from the brand's customer service agents. We find that the usability of a model's responses can make up for a large difference in inference cost for our case study brand, and we extrapolate our findings to the broader enterprise space.Comment: Paper to be published at the Association for Computational Linguistics in the Industry Track 202

Dialogical Signals of Stance Taking in Spontaneous Conversation

Thesis (Ph.D.)--University of Washington, 2020This is one of the first computational studies to investigate dialogical aspects of stance taking in spontaneous, spoken dialogue with a focus on lexical similarities. In any dialogic inter- action, each speaker influences the others’ lexical choices and aspects of their grammatical style (Brennan, 1996; Niederhoffer and Pennebaker, 2002). For this study, I leverage two distinct corpora. The ATAROS corpus (Freeman et al., 2014; Freeman, 2015) contains a series of task-oriented, collaborative tasks recorded in a controlled laboratory environment. The other corpus is drawn from a United States Homeland Security Subcommittee hearing regarding the 2007 - 2008 financial crisis. As such, it represents stance taking in an inherently adversarial environment, where high-stakes, real-world issues are being discussed. Both are annotated at the spurt-level with a 3-way stance strength annotation. I will show, through various experimental studies, that speakers show different patterns of dialogical behaviour when expressing stance versus when they are not; they show a higher level of engagement, as demonstrated by the use of similar or related terminology, and the rate of convergence in linguistic style, as measured by function word use, is also higher when expressing stance

An Independent Assessment of Phonetic Distinctive Feature Sets used to Model Pronunciation Variation

Thesis (Master's)--University of Washington, 2014It has been consistently shown that Automatic Speech Recognition (ASR) performance on casual, spontaneous speech is much worse than on carefully planned or read speech by as much as double the word error rate, and that variation in pronunciation is the main reason for this degradation of performance. Thus far, any attempts to mitigate this have fallen well below expectations. Phonetic Distinctive Features show promise from a theoretical standpoint, but have thus far not been fully incorporated into an end-to-end ASR system. Work incorporating distinctive features into ASR is widespread and varied, and each project uses a unique set of features based on the authors' linguistic intuitions, so the results of these experiments cannot be fully and fairly compared. In this work, I attempt to determine which style of distinctive feature set is best suited to model pronunciation variation in ASR based on measures of surface phone prediction accuracy and efficiency of the decision tree model. Using a non-exhaustive, representative set of phonetic distinctive feature sets, decision trees were trained, one per canonical base form phone, under two experimental conditions: words in isolation, and words in sequence. These models were tested against a comparable held-out test set, and an additional data set of canonical pronunciations used to simulate formal speech. It was found that a multi-valued articulatory-based feature set provided a far more compact model that yielded comparable accuracy results, while in a comparison of binary feature sets, the model with feature redundancy provided a far more robust model, with slightly higher accuracy and, where it predicted an incorrect phone, it was closer to the actual gold standard phone than the other feature sets' predictions

UBC Food Systems Project : scenario 1

The UBC Farm is a practical example of agriculture within an urban centre that embraces and promotes sustainable agricultural production, food security and safety, and the health of human communities. Unfortunately, the Farm is running at an annual deficit. In response to recommendations from past UBC Food System Project (UBCFSP) groups, our group examined the possibility of expanding production on the Farm as a way to increase revenue. A recurring recommendation in past UBCFSPs was to increase the Farm’s production area in order to increase revenues. However, as expansion has not yet occurred, our project goal was to determine the limiting factors and what modifications were needed in order to facilitate such growth. Through interviews and research into past groups’ UBCFSP projects we identified that the number of Farm volunteers has declined over the past year and that this declining group of volunteers is suffering from burnout due to the tediously repetitive and laborious tasks (such as weeding) that they must perform. Without an adequate labour force the Farm cannot expand. In order to address this limitation to expansion, our group explored both options to strengthen the current volunteer program at the Farm, and to explore ecologically sound, labour-saving farming techniques to alleviate volunteer burnout. We developed these findings into recommendations, which, if implemented, we believe will allow the Farm to expand production. Disclaimer: “UBC SEEDS provides students with the opportunity to share the findings of their studies, as well as their opinions, conclusions and recommendations with the UBC community. The reader should bear in mind that this is a student project/report and is not an official document of UBC. Furthermore readers should bear in mind that these reports may not reflect the current status of activities at UBC. We urge you to contact the research persons mentioned in a report or the SEEDS Coordinator about the current status of the subject matter of a project/report.”Land and Food Systems, Faculty ofUnreviewedUndergraduat