CoAIcoder: Examining the Effectiveness of AI-assisted Collaborative Qualitative Analysis

Abstract

While the domain of individual-level AI-assisted analysis has been extensively explored in previous studies, the field of AI-assisted collaborative qualitative analysis remains relatively unexplored. After identifying CQA practices and design opportunities through formative interviews, we introduce our collaborative qualitative coding tool, CoAIcoder, and designed the four different collaboration methods. We subsequently implemented a between-subject design involving 32 pairs of users who have undergone training in CQA across three commonly utilized phases under four methods. Our results suggest that CoAIcoder, which employs AI and a Shared Model, could potentially improve the efficiency of the coding process in CQA by fostering a quicker shared understanding and promoting early-stage discussions. However, this may come with the potential downside of reduced code diversity. We also underscored the existence of a trade-off between the level of independence and the coding outcome when humans collaborate during the early coding stages. Lastly, we identify design implications that could inspire and inform the future design of CQA systems