Expanding the Set of Pragmatic Considerations in Conversational AI

Despite considerable performance improvements, current conversational AI systems often fail to meet user expectations. We discuss several pragmatic limitations of current conversational AI systems. We illustrate pragmatic limitations with examples that are syntactically appropriate, but have clear pragmatic deficiencies. We label our complaints as "Turing Test Triggers" (TTTs) as they indicate where current conversational AI systems fall short compared to human behavior. We develop a taxonomy of pragmatic considerations intended to identify what pragmatic competencies a conversational AI system requires and discuss implications for the design and evaluation of conversational AI systems.Comment: Pre-print version of paper that appeared at Multidisciplinary Perspectives on COntext-aware embodied Spoken Interactions (MP-COSIN) workshop at IEEE RO-MAN 202

Evaluating the Deductive Competence of Large Language Models

The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance.Comment: 7 pages, 7 figures, under revie

Creating a common trajectory: Shared decision making and distributed cognition in medical consultations

The growing literature on shared decision making and patient centered care emphasizes the patient’s role in clinical care, but research on clinical reasoning almost exclusively addresses physician cognition. In this article, we suggest clinical cognition is distributed between physicians and patients and assess how distributed clinical cognition functions during interactions between medical professionals and patients with Multiple Sclerosis (MS). A combination of cognitive task analysis and discourse analysis reveals the distribution of clinical reasoning between 24 patients and 3 medical professionals engaged in MS management. Findings suggest that cognition was distributed between patients and physicians in all major tasks except for the interpretation of MRI results. Otherwise, patients and physicians collaborated through discourse to develop a common trajectory to guide clinical reasoning. The patients’ role in clinical cognition expands the concept of patient-centered care and suggests the need to optimize physician-patient distributed cognition rather than physician cognition in isolation

Requesting Distant Robotic Action: An Ontology for Naming and Action Identification for Planning on the Mars Exploration Rover Mission.

This paper focuses on the development of a naming convention and the use of abbreviated names and a related ontology for science work and distant robotic action that comprise requests for a robotic rover during the NASA Mars Exploration Rover (MER) mission, run by the Jet Propulsion Laboratory (JPL). We demonstrate how abbreviated names and an associated ontology support sharing and identifying information among teams and software tools. An ontology of distant action must take into account a dynamic environment, changing in response to physical events and intentional actions, and reflect the influence of context on the meaning of action. The nascent domain of Martian tele-robotic science, in which specialists request work from a rover moving through a distant landscape, as well as the need to consider the interdisciplinary teams involved in completing that work, required an empirical approach. The formulation of this ontology used ethnographic methods and grounded theory to study human behavior and work practice with software tools

Towards Efficient Scoring of Student-generated Long-form Analogies in STEM

Switching from an analogy pedagogy based on comprehension to analogy pedagogy based on production raises an impractical manual analogy scoring problem. Conventional symbol-matching approaches to computational analogy evaluation focus on positive cases, and challenge computational feasibility. This work presents the Discriminative Analogy Features (DAF) pipeline to identify the discriminative features of strong and weak long-form text analogies. We introduce four feature categories (semantic, syntactic, sentiment, and statistical) used with supervised vector-based learning methods to discriminate between strong and weak analogies. Using a modestly sized vector of engineered features with SVM attains a 0.67 macro F1 score. While a semantic feature is the most discriminative, out of the top 15 discriminative features, most are syntactic. Combining these engineered features with an ELMo-generated embedding still improves classification relative to an embedding alone. While an unsupervised K-Means clustering-based approach falls short, similar hints of improvement appear when inputs include the engineered features used in supervised learning

A Cognitively-Oriented Approach to Task Analysis and Test Development

Clear descriptions of job expertise are required to support applications and improvements in personnel training and job performance. This report describes a practical approach to task analysis that integrates the issues, content, and methods of cognitive science and personnel psychology. Cognitively oriented task analysis employs a breadth, then depth, strategy for identifying job expertise. Starting with a task-by-knowledge framework, job expertise is successively elaborated using interviews, expert ratings, and protocol analyses. The application of task analysis results to the development of written performance measures is described to illustrate the contributions of this approach to measurement validity. Task analysis results show that much of what has been missing in using existing task analysis methods is the mental aspects of performance related to interactions among task dimensions, task characteristics, and contexts. Two appendixes provide an example of knowledge elicitation and representation and item writing guidelines for performance measures

Navigating the Decision Space: Shared Medical Decision Making as Distributed Cognition

Despite increasing prominence, little is known about the cognitive processes underlying shared decision making. To investigate these processes, we conceptualize shared decision making as a form of distributed cognition. We introduce a Decision Space Model to identify physical and social influences on decision making. Using field observations and interviews, we demonstrate that patients and physicians in both acute and chronic care consider these influences when identifying the need for a decision, searching for decision parameters, making actionable decisions Based on the distribution of access to information and actions, we then identify four related patterns: physician dominated; physician-defined, patient-made; patient-defined, physician-made; and patient-dominated decisions. Results suggests that (a) decision making is necessarily distributed between physicians and patients, (b) differential access to information and action over time requires participants to transform a distributed task into a shared decision, and (c) adverse outcomes may result from failures to integrate physician and patient reasoning. Our analysis unifies disparate findings in the medical decision-making literature and has implications for improving care and medical training

Analyzing and Learning the Language for Different Types of Harassment

THIS ARTICLE USES WORDS OR LANGUAGE THAT IS CONSIDERED PROFANE, VULGAR, OR OFFENSIVE BY SOME READERS. The presence of a significant amount of harassment in user-generated content and its negative impact calls for robust automatic detection approaches. This requires the identification of different types of harassment. Earlier work has classified harassing language in terms of hurtfulness, abusiveness, sentiment, and profanity. However, to identify and understand harassment more accurately, it is essential to determine the contextual type that captures the interrelated conditions in which harassing language occurs. In this paper we introduce the notion of contextual type in harassment by distinguishing between five contextual types: (i) sexual, (ii) racial, (iii) appearance-related, (iv) intellectual and (v) political. We utilize an annotated corpus from Twitter distinguishing these types of harassment. We study the context of each kind to shed light on the linguistic meaning, interpretation, and distribution, with results from two lines of investigation: an extensive linguistic analysis, and the statistical distribution of uni-grams. We then build type- aware classifiers to automate the identification of type-specific harassment. Our experiments demonstrate that these classifiers provide competitive accuracy for identifying and analyzing harassment on social media. We present extensive discussion and significant observations about the effectiveness of type-aware classifiers using a detailed comparison setup, providing insight into the role of type-dependent features