274 research outputs found
On General Language Understanding
Natural Language Processing prides itself to be an empirically-minded, if not
outright empiricist field, and yet lately it seems to get itself into
essentialist debates on issues of meaning and measurement ("Do Large Language
Models Understand Language, And If So, How Much?"). This is not by accident:
Here, as everywhere, the evidence underspecifies the understanding. As a
remedy, this paper sketches the outlines of a model of understanding, which can
ground questions of the adequacy of current methods of measurement of model
quality. The paper makes three claims: A) That different language use situation
types have different characteristics, B) That language understanding is a
multifaceted phenomenon, bringing together individualistic and social
processes, and C) That the choice of Understanding Indicator marks the limits
of benchmarking, and the beginnings of considerations of the ethics of NLP use.Comment: Findings of EMNLP 202
Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
In this work, we propose a method that combines two popular research areas by
injecting linguistic structures into pre-trained language models in the
parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel
adapter modules encoding different linguistic structures are combined using a
novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates
are used to determine the importance of these modules at each layer of the
model. To reduce the number of parameters, we first train the model for a fixed
small number of steps before pruning the experts based on their importance
scores. Our experiment results with three different pre-trained models show
that our approach can outperform state-of-the-art PEFT methods with a
comparable number of parameters. In addition, we provide additional analysis to
examine the experts selected by each model at each layer to provide insights
for future studies.Comment: 14 pages, 3 figures, Camera-Ready for EMNLP 2023 Findings (Long
Paper
Case-based medical informatics
BACKGROUND: The "applied" nature distinguishes applied sciences from theoretical sciences. To emphasize this distinction, we begin with a general, meta-level overview of the scientific endeavor. We introduce the knowledge spectrum and four interconnected modalities of knowledge. In addition to the traditional differentiation between implicit and explicit knowledge we outline the concepts of general and individual knowledge. We connect general knowledge with the "frame problem," a fundamental issue of artificial intelligence, and individual knowledge with another important paradigm of artificial intelligence, case-based reasoning, a method of individual knowledge processing that aims at solving new problems based on the solutions to similar past problems. We outline the fundamental differences between Medical Informatics and theoretical sciences and propose that Medical Informatics research should advance individual knowledge processing (case-based reasoning) and that natural language processing research is an important step towards this goal that may have ethical implications for patient-centered health medicine. DISCUSSION: We focus on fundamental aspects of decision-making, which connect human expertise with individual knowledge processing. We continue with a knowledge spectrum perspective on biomedical knowledge and conclude that case-based reasoning is the paradigm that can advance towards personalized healthcare and that can enable the education of patients and providers. We center the discussion on formal methods of knowledge representation around the frame problem. We propose a context-dependent view on the notion of "meaning" and advocate the need for case-based reasoning research and natural language processing. In the context of memory based knowledge processing, pattern recognition, comparison and analogy-making, we conclude that while humans seem to naturally support the case-based reasoning paradigm (memory of past experiences of problem-solving and powerful case matching mechanisms), technical solutions are challenging. Finally, we discuss the major challenges for a technical solution: case record comprehensiveness, organization of information on similarity principles, development of pattern recognition and solving ethical issues. SUMMARY: Medical Informatics is an applied science that should be committed to advancing patient-centered medicine through individual knowledge processing. Case-based reasoning is the technical solution that enables a continuous individual knowledge processing and could be applied providing that challenges and ethical issues arising are addressed appropriately
A Kind Introduction to Lexical and Grammatical Aspect, with a Survey of Computational Approaches
Aspectual meaning refers to how the internal temporal structure of situations
is presented. This includes whether a situation is described as a state or as
an event, whether the situation is finished or ongoing, and whether it is
viewed as a whole or with a focus on a particular phase. This survey gives an
overview of computational approaches to modeling lexical and grammatical aspect
along with intuitive explanations of the necessary linguistic concepts and
terminology. In particular, we describe the concepts of stativity, telicity,
habituality, perfective and imperfective, as well as influential inventories of
eventuality and situation types. We argue that because aspect is a crucial
component of semantics, especially when it comes to reporting the temporal
structure of situations in a precise way, future NLP approaches need to be able
to handle and evaluate it systematically in order to achieve human-level
language understanding.Comment: Accepted at EACL 2023, camera ready versio
Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures
In this work we build upon negative results from an attempt at language
modeling with predicted semantic structure, in order to establish empirical
lower bounds on what could have made the attempt successful. More specifically,
we design a concise binary vector representation of semantic structure at the
lexical level and evaluate in-depth how good an incremental tagger needs to be
in order to achieve better-than-baseline performance with an end-to-end
semantic-bootstrapping language model. We envision such a system as consisting
of a (pretrained) sequential-neural component and a hierarchical-symbolic
component working together to generate text with low surprisal and high
linguistic interpretability. We find that (a) dimensionality of the semantic
vector representation can be dramatically reduced without losing its main
advantages and (b) lower bounds on prediction quality cannot be established via
a single score alone, but need to take the distributions of signal and noise
into account.Comment: To appear at *SEM 2023, Toront
- …