17,274 research outputs found
On Quantifying Qualitative Geospatial Data: A Probabilistic Approach
Living in the era of data deluge, we have witnessed a web content explosion,
largely due to the massive availability of User-Generated Content (UGC). In
this work, we specifically consider the problem of geospatial information
extraction and representation, where one can exploit diverse sources of
information (such as image and audio data, text data, etc), going beyond
traditional volunteered geographic information. Our ambition is to include
available narrative information in an effort to better explain geospatial
relationships: with spatial reasoning being a basic form of human cognition,
narratives expressing such experiences typically contain qualitative spatial
data, i.e., spatial objects and spatial relationships.
To this end, we formulate a quantitative approach for the representation of
qualitative spatial relations extracted from UGC in the form of texts. The
proposed method quantifies such relations based on multiple text observations.
Such observations provide distance and orientation features which are utilized
by a greedy Expectation Maximization-based (EM) algorithm to infer a
probability distribution over predefined spatial relationships; the latter
represent the quantified relationships under user-defined probabilistic
assumptions. We evaluate the applicability and quality of the proposed approach
using real UGC data originating from an actual travel blog text corpus. To
verify the quality of the result, we generate grid-based maps visualizing the
spatial extent of the various relations
Empowering Active Learning to Jointly Optimize System and User Demands
Existing approaches to active learning maximize the system performance by
sampling unlabeled instances for annotation that yield the most efficient
training. However, when active learning is integrated with an end-user
application, this can lead to frustration for participating users, as they
spend time labeling instances that they would not otherwise be interested in
reading. In this paper, we propose a new active learning approach that jointly
optimizes the seemingly counteracting objectives of the active learning system
(training efficiently) and the user (receiving useful instances). We study our
approach in an educational application, which particularly benefits from this
technique as the system needs to rapidly learn to predict the appropriateness
of an exercise to a particular user, while the users should receive only
exercises that match their skills. We evaluate multiple learning strategies and
user types with data from real users and find that our joint approach better
satisfies both objectives when alternative methods lead to many unsuitable
exercises for end users.Comment: To appear as a long paper in Proceedings of the 58th Annual Meeting
of the Association for Computational Linguistics (ACL 2020). Download our
code and simulated user models at github:
https://github.com/UKPLab/acl2020-empowering-active-learnin
Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling
Uncertainty decomposition refers to the task of decomposing the total
uncertainty of a model into data (aleatoric) uncertainty, resulting from the
inherent complexity or ambiguity of the data, and model (epistemic)
uncertainty, resulting from the lack of knowledge in the model. Performing
uncertainty decomposition for large language models (LLMs) is an important step
toward improving the reliability, trustworthiness, and interpretability of
LLMs, but this research task is very challenging and remains unresolved. The
existing canonical method, Bayesian Neural Network (BNN), cannot be applied to
LLMs, because BNN requires training and ensembling multiple variants of models,
which is infeasible or prohibitively expensive for LLMs. In this paper, we
introduce an uncertainty decomposition framework for LLMs, called input
clarifications ensemble, which bypasses the need to train new models. Rather
than ensembling models with different parameters, our approach generates a set
of clarifications for the input, feeds them into the fixed LLMs, and ensembles
the corresponding predictions. We show that our framework shares a symmetric
decomposition structure with BNN. Empirical evaluations demonstrate that the
proposed framework provides accurate and reliable uncertainty quantification on
various tasks. Code will be made publicly available at
https://github.com/UCSB-NLP-Chang/llm_uncertainty .Comment: 15 pages, 3 figure
Calibrated Prediction Intervals for Neural Network Regressors
Ongoing developments in neural network models are continually advancing the
state of the art in terms of system accuracy. However, the predicted labels
should not be regarded as the only core output; also important is a
well-calibrated estimate of the prediction uncertainty. Such estimates and
their calibration are critical in many practical applications. Despite their
obvious aforementioned advantage in relation to accuracy, contemporary neural
networks can, generally, be regarded as poorly calibrated and as such do not
produce reliable output probability estimates. Further, while post-processing
calibration solutions can be found in the relevant literature, these tend to be
for systems performing classification. In this regard, we herein present two
novel methods for acquiring calibrated predictions intervals for neural network
regressors: empirical calibration and temperature scaling. In experiments using
different regression tasks from the audio and computer vision domains, we find
that both our proposed methods are indeed capable of producing calibrated
prediction intervals for neural network regressors with any desired confidence
level, a finding that is consistent across all datasets and neural network
architectures we experimented with. In addition, we derive an additional
practical recommendation for producing more accurate calibrated prediction
intervals. We release the source code implementing our proposed methods for
computing calibrated predicted intervals. The code for computing calibrated
predicted intervals is publicly available
- …