Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis

Large language models (LLMs) offer significant promise as a knowledge source for task learning. Prompt engineering has been shown to be effective for eliciting knowledge from an LLM, but alone it is insufficient for acquiring relevant, situationally grounded knowledge for an embodied agent learning novel tasks. We describe a cognitive-agent approach, STARS, that extends and complements prompt engineering, mitigating its limitations and thus enabling an agent to acquire new task knowledge matched to its native language capabilities, embodiment, environment, and user preferences. The STARS approach is to increase the response space of LLMs and deploy general strategies, embedded within the autonomous agent, to evaluate, repair, and select among candidate responses produced by the LLM. We describe the approach and experiments that show how an agent, by retrieving and evaluating a breadth of responses from the LLM, can achieve 77-94% task completion in one-shot learning without user oversight. The approach achieves 100% task completion when human oversight (such as an indication of preference) is provided. Further, the type of oversight largely shifts from explicit, natural language instruction to simple confirmation/discomfirmation of high-quality responses that have been vetted by the agent before presentation to a user.Comment: 7 pages, 8 figures, 3 tables, bibliography, appendix (34 pages total). Accepted to AAAI 202

Improving Language Model Prompting in Support of Semi-autonomous Task Learning

Language models (LLMs) offer potential as a source of knowledge for agents that need to acquire new task competencies within a performance environment. We describe efforts toward a novel agent capability that can construct cues (or "prompts") that result in useful LLM responses for an agent learning a new task. Importantly, responses must not only be "reasonable" (a measure used commonly in research on knowledge extraction from LLMs) but also specific to the agent's task context and in a form that the agent can interpret given its native language capacities. We summarize a series of empirical investigations of prompting strategies and evaluate responses against the goals of targeted and actionable responses for task learning. Our results demonstrate that actionable task knowledge can be obtained from LLMs in support of online agent task learning.Comment: Submitted to ACS 202

Exploring the patient-reported impact of the pharmacist on pre-bariatric surgical assessment

Background The effects of surgical procedures and the need for life-long nutrient supplementation may impact on medication regimes, requiring changes to dosage and formulation of medicines, which can be difficult for patients following surgery. Our pre-surgical assessment pathway involves a pharmacist with specialist knowledge of bariatric surgery, to help prepare patients for these changes. Objective To explore the patient-reported impact of the specialist bariatric pharmacist in pre-surgical assessment. Setting National Health Service Hospital, United Kingdom Methods A two phased, retrospective study design using participants recruited from pre-surgical clinic lists. The first phase consisted of confidential, face to face semi-structured interviews. A constant comparative analytic framework informed the construction of the second phase, which consisted of a confidential survey to test the generalizability of the findings with a larger cohort of patients. Results A total of 40 participants (12 interviews, 28 surveys) were recruited to the study. The majority of participants were female (n=33), mean age 50 years, mean pre-surgical weight 124kg (n=38). The most common comorbidity was Type 2 diabetes. Participants on medication had at least one comorbidity, with the majority of conditions improved or eliminated after surgery. Conclusions The pre-surgical consultation with the pharmacist was highly valued by the participants, providing information and support which helped prepare for medication changes after bariatric surgery. Many felt a post-surgical appointment with the pharmacist would provide support and improve compliance with vitamins and medications. Future research into the role of pharmacists in the bariatric multi-disciplinary team and patient support are recommende

Special Issue "On Defining Artificial Intelligence"—Commentaries and Author's Response

n/

Constructing Meaning, Piece by Piece: A Computational Cognitive Model of Human Sentence Comprehension

AI systems with language for robots don’t try to model human processing. Psycholinguistic models of human language processing don’t have operational computational models. To solve these problems, this thesis contributes to progress in answering two interlocking scientific questions: how does the human mind do sentence comprehension, and how can we enable artificial agents to use natural language to collaborate with humans. We do this with a system called Lucia, which is a computational cognitive model of human sentence comprehension that works by constructing the meaning of a sentence piece by piece. The Lucia model is designed according to five overriding qualitative principles of human language comprehension. To show that its results are useful, it does embodied, end-to-end comprehension (E3C) within an artificial agent called Rosie. To model key characteristics of human comprehension, it draws on research in cognitive linguistics, psycholinguistics, artificial intelligence, and robotics to: represent composable knowledge of the meaning of linguistic forms (CKM), do incremental, immediate interpretation processing (I3P), and do it using general cognitive mechanisms (GCM). The model leads to a theory of language acquisition from experience (LAE), some parts of which have been implemented experimentally. To conform to these principles, the Lucia model is implemented in a robotic agent called Rosie to do E3C. It uses Embodied Construction Grammar (ECG) as its method of representing composable knowledge of meaning (CKM), and demonstrates that this knowledge can be processed incrementally (I3P) using a novel comprehension algorithm that relies on the general cognitive mechanisms (GCM) of the Soar cognitive architecture to produce embodied, end-to-end comprehension (E3C). Lucia makes several contributions to answering the broader scientific questions. It provides a novel theory for incremental processing (I3P) based on a three-phase construction cycle. It provides a theory of how memories interact during comprehension. It demonstrates grounded comprehension in an embodied robotic agent. Finally, it provides a detailed, functional model of cognitive E3C processing that can serve as a basis for further research in modeling human language processing in the brain and in designing larger-scale language models for artificial agents.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/172668/1/plindes_1.pd

Constructing Meaning in Small Increments

Humans comprehend natural language sentences in real time, processing the elements of each sentence incrementally withimmediate interpretation, while working within the limitations of general cognitive abilities. While much research hasbeen devoted to human sentence comprehension, a detailed computational theory of how this is done has been lacking.In this work we explore some fundamental principles of human sentence comprehension, propose a novel computationaltheory of knowledge representation and incremental processing to comprehend sentences using general cognitive abilities,and discuss results of an implementation of this theory in a robotic agent. We then explore the theorys implications forfuture work in various areas of cognitive science

Modeling Human-Like Acquisition of Language and Concepts

Humans acquire language and related concepts in a trajectory over a lifetime. Concepts for simple interaction with the world are learned before language. Later, words are learned to name these concepts along with structures needed to represent larger meanings. Eventually, language advances to where it can drive the learning of new concepts. Throughout this trajectory a language processing capability uses architectural mechanisms to process language using the knowledge already acquired. We assume that this growing body of knowledge is made up of small units of form-meaning mapping that can be composed in many ways, suggesting that these units are learned incrementally from experience. In prior work we have built a system to comprehend human language within an autonomous robot using knowledge in such units developed by hand. Here we propose a research program to develop the ability of an artificial agent to acquire this knowledge incrementally and autonomously from its experience in a similar trajectory. We then propose a strategy for evaluating this human-like learning system using a large benchmark created as a tool for training deep learning systems. We expect that our human-like learning system will produce better task performance from training on only a small subset of this benchmark

Toward Human-Like Representation Learning for Cognitive Architectures

Human-like learning includes an ability to learn concepts from a stream of embodiment sensor data. Echoing previous thoughts such as those from Barsalou that cognition and perception share a common representation system, we suggest an addendum to the common model of cognition. This addendum poses a simultaneous semantic memory and perception learning that bypasses working memory, and that uses parallel processing to learn concepts apart from deliberate reasoning. The goal is to provide a general outline for how to extend a class of cognitive architectures to implement a more human-like interface between cognition and embodiment of an agent, where a critical aspect of that interface is that it is dynamic because of learning

Ontology-Based Information Extraction with a Cognitive Agent

Machine reading is a relatively new field that features computer programs designed to read flowing text and extract fact assertions expressed by the narrative content. This task involves two core technologies: natural language processing (NLP) and information extraction (IE). In this paper we describe a machine reading system that we have developed within a cognitive architecture. We show how we have integrated into the framework several levels of knowledge for a particular domain, ideas from cognitive semantics and construction grammar, plus tools from prior NLP and IE research. The result is a system that is capable of reading and interpreting complex and fairly idiosyncratic texts in the family history domain. We describe the architecture and performance of the system. After presenting the results from several evaluations that we have carried out, we summarize possible future directions