Artificial intelligence-assisted decision-making in long-term care: a qualitative study on opportunities and prerequisites for responsible innovation (Preprint)

Background: While use of artificial intelligence (AI)-based technologies such as decision-support systems (AI-DSSs) could help sustaining and improving the quality and efficiency of care, their deployment also creates ethical and social challenges. In recent years, there has been a growing prevalence of high-level guidelines and frameworks to provide guidance on responsible AI innovation. However, few studies specify how AI-based technologies such as AI-DSSs can be responsibly embedded in specific contexts such as the nursing process in the long-term care (LTC) for older adults. Objective: Opportunities and prerequisites for responsible AI-assisted decision-making in the nursing process were explored from the perspectives of nurses and other professional stakeholders in LTC. Methods: Semi-structured interviews were conducted with 24 care professionals in Dutch LTC, including nurses, care coordinators, data specialists and care centralists. Two imaginary scenarios about the future use of AI-DSSs were developed beforehand and used to enable participants to articulate their expectations regarding the opportunities and risks of AI-assisted decision-making. After first openly discussing opportunities and possible risks associated with both scenarios, six high-level principles for responsible AI were used as probing themes to evoke further consideration on risks of using AI-DSSs in LTC. Further, participants were asked to brainstorm about possible strategies and actions in the design, implementation and use of AI-DSSs to address or mitigate the mentioned risks. A thematic analysis was carried out to identify opportunities and prerequisites for responsible innovation in this area. Results: Professionals’ stance towards the use of AI-DSSs is not a matter of purely positive or negative expectations, but rather a nuanced interplay of positive and negative elements that lead to a weighed perception of opportunities and prerequisites for responsible AI-assisted decision-making. Both opportunities and risks were identified in relation to early identification of care needs, guidance in devising care strategies, shared decision-making, and caregivers’ workload and work experience. To optimally balance opportunities and risks of AI-assisted decision-making, seven categories of prerequisites for responsible AI-assisted decision-making in the nursing process were identified: (1) regular deliberation on data collection, (2) a balanced proactive nature of AI-DSSs, (3) incremental advancements aligned with trust and experience, (4) customization for all user groups including clients and caregivers, (5) measures to counteract bias and narrow perspectives, (6) human-centric learning loops, and (7) routinization of using AI-DSSs. Conclusions: Opportunities of AI-assisted decision-making in the nursing process could turn into drawbacks, depending on the specific shaping of the design and the deployment of AI-DSSs. Therefore, we recommend viewing the responsible use of AI-DSSs as a balancing act. Moreover, given the interrelatedness of the identified prerequisites, we call for various actors, including developers and users of AI-DSSs, to cohesively address different factors important to the responsible embedding of AI-DSSs in practice