1,789 research outputs found
Entropy and Graph Based Modelling of Document Coherence using Discourse Entities: An Application
We present two novel models of document coherence and their application to
information retrieval (IR). Both models approximate document coherence using
discourse entities, e.g. the subject or object of a sentence. Our first model
views text as a Markov process generating sequences of discourse entities
(entity n-grams); we use the entropy of these entity n-grams to approximate the
rate at which new information appears in text, reasoning that as more new words
appear, the topic increasingly drifts and text coherence decreases. Our second
model extends the work of Guinaudeau & Strube [28] that represents text as a
graph of discourse entities, linked by different relations, such as their
distance or adjacency in text. We use several graph topology metrics to
approximate different aspects of the discourse flow that can indicate
coherence, such as the average clustering or betweenness of discourse entities
in text. Experiments with several instantiations of these models show that: (i)
our models perform on a par with two other well-known models of text coherence
even without any parameter tuning, and (ii) reranking retrieval results
according to their coherence scores gives notable performance gains, confirming
a relation between document coherence and relevance. This work contributes two
novel models of document coherence, the application of which to IR complements
recent work in the integration of document cohesiveness or comprehensibility to
ranking [5, 56]
Collaborative Evaluation: Exploring the Synergy of Large Language Models and Humans for Open-ended Generation Evaluation
Humans are widely involved in the evaluation of open-ended natural language
generation tasks (NLG) that demand creativity, as automatic metrics often
exhibit weak correlations with human judgments. Large language models (LLMs)
recently have emerged as a scalable and cost-effective alternative to human
evaluations. However, both humans and LLMs have limitations, i.e., inherent
subjectivity and unreliable judgments, particularly for open-ended tasks that
require adaptable metrics tailored to diverse task requirements. To explore the
synergy between humans and LLM-based evaluators and address the challenges of
existing inconsistent evaluation criteria in open-ended NLG tasks, we propose a
Collaborative Evaluation pipeline CoEval, involving the design of a checklist
of task-specific criteria and the detailed evaluation of texts, in which LLM
generates initial ideation, and then humans engage in scrutiny. We conducted a
series of experiments to investigate the mutual effects between LLMs and humans
in CoEval. Results show that, by utilizing LLMs, CoEval effectively evaluates
lengthy texts, saving significant time and reducing human evaluation outliers.
Human scrutiny still plays a role, revising around 20% of LLM evaluation scores
for ultimate reliability.Comment: We release our resources at \url{https://github.com/qtli/CoEval
MIRACLE: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control
Personalized dialogue systems aim to endow the chatbot agent with more
anthropomorphic traits for human-like interactions. Previous approaches have
explored explicitly user profile modeling using text descriptions, implicit
derivation of user embeddings, or utilizing handicraft prompts for ChatGPT-like
models. However, textual personas are limited in describing multi-faceted
attributes (\emph{e.g.}, \emph{language style, inner character nuances}),
implicit embedding suffers from personality sparsity, and handicraft prompts
lack fine-grained and stable controllability. Hence, these approaches may
struggle with complex personalized dialogue generation tasks that require
generating controllable responses with multiple personal attributes. To this
end, we propose \textbf{\textsc{Miracle}}, a novel personalized dialogue
generation method through \textbf{M}ult\textbf{I}ple Pe\textbf{R}sonal
\textbf{A}ttributes \textbf{C}ontrol within \textbf{L}atent-Space
\textbf{E}nergy-based Models. ttributes \textbf{C}ontrol within
\textbf{L}atent-Space \textbf{E}nergy-based Models. Specifically, our approach
first disentangles complex personality into multi-faceted attributes.
Subsequently, we employ a conditional variational auto-encoder to align with
the dense personalized responses within a latent joint attribute space. We have
also tailored a dedicated energy function and customized the ordinary
differential equations sampling method to offer flexible attribute composition
and precise attribute control. Extensive experiments demonstrate that
\textsc{Miracle} outperforms several strong baselines in terms of personality
controllability and response generation quality. Our dataset and code are
available at \url{https://github.com/LZY-the-boys/MIRACLE}Comment: Accepted by EMNLP2023 finding
Onsager Loop-Transition and First Order Flux-Line Lattice Melting in High- Superconductors
Monte-Carlo simulations in conjunction with finite-size scaling analysis are
used to investigate the -phase diagram in uniaxial anisotropic high-
superconductors, both in zero magnetic field and in intermediate magnetic
fields for various mass-anisotropies. The model we consider is the uniformly
frustrated anisotropic Villain Model. In zero magnetic field, and for all
anisotropies considered, we find one single second order phase transition,
mediated by an Onsager vortex-loop blowout. This is the superconductor-normal
metal transition.A comparison with numerical simulations and a critical scaling
analysis of the zero-field loop-transition yields the same exponent of the loop
distribution function at the critical point. In the intermediate magnetic field
regime, we find two anomalies in the specific heat. The first anomaly at a
temperature is associated with the melting transition of the flux-line
lattice. The second anomaly at a temperature is one where phase coherence
along the field direction is destroyed. We argue that in the
thermodynamic and continuum limit. Hence, there is no regime where the flux
line lattice melts into a disentangled flux-line liquid. The loss of phase
coherence parallel to the magnetic field in the sample is argued to be due to
the proliferation of closed non-field induced vortex loops on the scale of the
magnetic length in the problem, resulting in flux-line cutting and
recombination. In the flux-line liquid phase, therefore, flux-lines appear no
longer to be well defined entities. A finite-size scaling analysis of the delta
function peak specific heat anomaly at the melting transition is used to
extract the discontinuity of the entropy at the melting transition.This entropy
discontinuity is found to increase rapidly with mass-anisotropy.Comment: 22 pages, 11 figures included, to be published in Phys. Rev. B, 57
xxx (1998
Interactional Slingshots: Providing Support Structure to User Interactions in Hybrid Intelligence Systems
The proliferation of artificial intelligence (AI) systems has enabled us to engage more deeply and powerfully with our digital and physical environments, from chatbots to autonomous vehicles to robotic assistive technology. Unfortunately, these state-of-the-art systems often fail in contexts that require human understanding, are never-before-seen, or complex. In such cases, though the AI-only approaches cannot solve the full task, their ability to solve a piece of the task can be combined with human effort to become more robust to handling complexity and uncertainty. A hybrid intelligence system—one that combines human and machine skill sets—can make intelligent systems more operable in real-world settings.
In this dissertation, we propose the idea of using interactional slingshots as a means of providing support structure to user interactions in hybrid intelligence systems. Much like how gravitational slingshots provide boosts to spacecraft en route to their final destinations, so do interactional slingshots provide boosts to user interactions en route to solving tasks. Several challenges arise: What does this support structure look like? How much freedom does the user have in their interactions? How is user expertise paired with that of the machine’s?
To do this as a tractable socio-technical problem, we explore this idea in the context of data annotation problems, especially in those domains where AI methods fail to solve the overall task. Getting annotated (labeled) data is crucial for successful AI methods, and becomes especially more difficult in domains where AI fails, since problems in such domains require human understanding to fully solve, but also present challenges related to annotator expertise, annotation freedom, and context curation from the data. To explore data annotation problems in this space, we develop techniques and workflows whose interactional slingshot support structure harnesses the user’s interaction with data.
First, we explore providing support in the form of nudging non-expert users’ interactions as they annotate text data for the task of creating conversational memory. Second, we add support structure in the form of assisting non-expert users during the annotation process itself for the task of grounding natural language references to objects in 3D point clouds. Finally, we supply support in the form of guiding expert and non-expert users both before and during their annotations for the task of conversational disentanglement across multiple domains.
We demonstrate that building hybrid intelligence systems with each of these interactional slingshot support mechanisms—nudging, assisting, and guiding a user’s interaction with data—improves annotation outcomes, such as annotation speed, accuracy, effort level, even when annotators’ expertise and skill levels vary.
Thesis Statement: By providing support structure that nudges, assists, and guides user interactions, it is possible to create hybrid intelligence systems that enable more efficient (faster and/or more accurate) data annotation.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/163138/1/sairohit_1.pd
- …