CoAIcoder: Examining the Effectiveness of AI-assisted Collaborative Qualitative Analysis

While the domain of individual-level AI-assisted analysis has been extensively explored in previous studies, the field of AI-assisted collaborative qualitative analysis remains relatively unexplored. After identifying CQA practices and design opportunities through formative interviews, we introduce our collaborative qualitative coding tool, CoAIcoder, and designed the four different collaboration methods. We subsequently implemented a between-subject design involving 32 pairs of users who have undergone training in CQA across three commonly utilized phases under four methods. Our results suggest that CoAIcoder, which employs AI and a Shared Model, could potentially improve the efficiency of the coding process in CQA by fostering a quicker shared understanding and promoting early-stage discussions. However, this may come with the potential downside of reduced code diversity. We also underscored the existence of a trade-off between the level of independence and the coding outcome when humans collaborate during the early coding stages. Lastly, we identify design implications that could inspire and inform the future design of CQA systems

Evaluating GPT-3 Generated Explanations for Hateful Content Moderation

Recent research has focused on using large language models (LLMs) to generate explanations for hate speech through fine-tuning or prompting. Despite the growing interest in this area, these generated explanations' effectiveness and potential limitations remain poorly understood. A key concern is that these explanations, generated by LLMs, may lead to erroneous judgments about the nature of flagged content by both users and content moderators. For instance, an LLM-generated explanation might inaccurately convince a content moderator that a benign piece of content is hateful. In light of this, we propose an analytical framework for examining hate speech explanations and conducted an extensive survey on evaluating such explanations. Specifically, we prompted GPT-3 to generate explanations for both hateful and non-hateful content, and a survey was conducted with 2,400 unique respondents to evaluate the generated explanations. Our findings reveal that (1) human evaluators rated the GPT-generated explanations as high quality in terms of linguistic fluency, informativeness, persuasiveness, and logical soundness, (2) the persuasive nature of these explanations, however, varied depending on the prompting strategy employed, and (3) this persuasiveness may result in incorrect judgments about the hatefulness of the content. Our study underscores the need for caution in applying LLM-generated explanations for content moderation. Code and results are available at https://github.com/Social-AI-Studio/GPT3-HateEval.Comment: 9 pages, 2 figures, Accepted by International Joint Conference on Artificial Intelligence(IJCAI

Smartwatch-based early gesture detection & trajectory tracking for interactive gesture-driven applications

Ministry of Education, Singapore under its Academic Research Funding Tier

Empath-D: VR-based empathetic app design for accessibility

Singapore National Research Foundation under IDM Futures Funding Initiative; Ministry of Education, Singapore under its Academic Research Funding Tier

Impact of Human-AI Interaction on User Trust and Reliance in AI-Assisted Qualitative Coding

While AI shows promise for enhancing the efficiency of qualitative analysis, the unique human-AI interaction resulting from varied coding strategies makes it challenging to develop a trustworthy AI-assisted qualitative coding system (AIQCs) that supports coding tasks effectively. We bridge this gap by exploring the impact of varying coding strategies on user trust and reliance on AI. We conducted a mixed-methods split-plot 3x3 study, involving 30 participants, and a follow-up study with 6 participants, exploring varying text selection and code length in the use of our AIQCs system for qualitative analysis. Our results indicate that qualitative open coding should be conceptualized as a series of distinct subtasks, each with differing levels of complexity, and therefore, should be given tailored design considerations. We further observed a discrepancy between perceived and behavioral measures, and emphasized the potential challenges of under- and over-reliance on AIQCs systems. Additional design implications were also proposed for consideration.Comment: 27 pages with references, 9 figures, 5 table

Empath-D: Empathetic design for accessibility

Singapore National Research Foundation under IDM Futures Funding Initiative; Ministry of Education, Singapore under its Academic Research Funding Tier