14 research outputs found
CoAIcoder: Examining the Effectiveness of AI-assisted Collaborative Qualitative Analysis
While the domain of individual-level AI-assisted analysis has been
extensively explored in previous studies, the field of AI-assisted
collaborative qualitative analysis remains relatively unexplored. After
identifying CQA practices and design opportunities through formative
interviews, we introduce our collaborative qualitative coding tool, CoAIcoder,
and designed the four different collaboration methods. We subsequently
implemented a between-subject design involving 32 pairs of users who have
undergone training in CQA across three commonly utilized phases under four
methods. Our results suggest that CoAIcoder, which employs AI and a Shared
Model, could potentially improve the efficiency of the coding process in CQA by
fostering a quicker shared understanding and promoting early-stage discussions.
However, this may come with the potential downside of reduced code diversity.
We also underscored the existence of a trade-off between the level of
independence and the coding outcome when humans collaborate during the early
coding stages. Lastly, we identify design implications that could inspire and
inform the future design of CQA systems
Evaluating GPT-3 Generated Explanations for Hateful Content Moderation
Recent research has focused on using large language models (LLMs) to generate
explanations for hate speech through fine-tuning or prompting. Despite the
growing interest in this area, these generated explanations' effectiveness and
potential limitations remain poorly understood. A key concern is that these
explanations, generated by LLMs, may lead to erroneous judgments about the
nature of flagged content by both users and content moderators. For instance,
an LLM-generated explanation might inaccurately convince a content moderator
that a benign piece of content is hateful. In light of this, we propose an
analytical framework for examining hate speech explanations and conducted an
extensive survey on evaluating such explanations. Specifically, we prompted
GPT-3 to generate explanations for both hateful and non-hateful content, and a
survey was conducted with 2,400 unique respondents to evaluate the generated
explanations. Our findings reveal that (1) human evaluators rated the
GPT-generated explanations as high quality in terms of linguistic fluency,
informativeness, persuasiveness, and logical soundness, (2) the persuasive
nature of these explanations, however, varied depending on the prompting
strategy employed, and (3) this persuasiveness may result in incorrect
judgments about the hatefulness of the content. Our study underscores the need
for caution in applying LLM-generated explanations for content moderation. Code
and results are available at https://github.com/Social-AI-Studio/GPT3-HateEval.Comment: 9 pages, 2 figures, Accepted by International Joint Conference on
Artificial Intelligence(IJCAI
Smartwatch-based early gesture detection & trajectory tracking for interactive gesture-driven applications
Ministry of Education, Singapore under its Academic Research Funding Tier
Empath-D: VR-based empathetic app design for accessibility
Singapore National Research Foundation under IDM Futures Funding Initiative; Ministry of Education, Singapore under its Academic Research Funding Tier
Impact of Human-AI Interaction on User Trust and Reliance in AI-Assisted Qualitative Coding
While AI shows promise for enhancing the efficiency of qualitative analysis,
the unique human-AI interaction resulting from varied coding strategies makes
it challenging to develop a trustworthy AI-assisted qualitative coding system
(AIQCs) that supports coding tasks effectively. We bridge this gap by exploring
the impact of varying coding strategies on user trust and reliance on AI. We
conducted a mixed-methods split-plot 3x3 study, involving 30 participants, and
a follow-up study with 6 participants, exploring varying text selection and
code length in the use of our AIQCs system for qualitative analysis. Our
results indicate that qualitative open coding should be conceptualized as a
series of distinct subtasks, each with differing levels of complexity, and
therefore, should be given tailored design considerations. We further observed
a discrepancy between perceived and behavioral measures, and emphasized the
potential challenges of under- and over-reliance on AIQCs systems. Additional
design implications were also proposed for consideration.Comment: 27 pages with references, 9 figures, 5 table
Empath-D: Empathetic design for accessibility
Singapore National Research Foundation under IDM Futures Funding Initiative; Ministry of Education, Singapore under its Academic Research Funding Tier