Does Artificial Intelligence Really Benefit Reviewers with Reduced Workload? A Mixed-Methods Usability Study on Systematic Review Software

Systematic review (SR) is widely used in evidence-based healthcare. Conducting a systematic review requires great mental efforts, especially during the article screening process. This challenge has motivated researchers to develop intelligent software to streamline the process. This study used both quantitative and qualitative research methods to investigate whether Colandr, an SR application which is equipped with artificial intelligence (AI) and claims to facilitate decision-making by relevance prediction, can reduce reviewers’ mental workload compared to Covidence, a popular SR software without any AI feature. Both perceived and objective cognitive workload were measured by NASA-TLX and pupil size change. The results indicate that Colandr neither helped reviewers achieve higher screening accuracy nor reduce their mental workload compared with Covidence. The qualitative interview results also provide valuable suggestions to the design of SR software. This study is the first usability investigation on the mental workload demand associated with the SR screening process.Master of Science in Information Scienc