Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

Evaluating conversational agents (CAs) that are supposed to be applied in healthcare settings and ensuring their quality is essential to avoid patient harm and ensure efficacy of the CA-delivered intervention. However, a guideline for a standardized quality assessment of health CAs is still missing. The objective of this work is to describe a framework that provides guidance for development and evaluation of health CAs. In previous work, consensus on categories for evaluating health CAs has been found. In this work, we identify concrete metrics, heuristics, and checklists for these evaluation categories to form a framework. We focus on a specific type of health CA, namely rule-based systems that are based on written input and output, have a simple personality without any kind of embodiment. First, we identified relevant metrics, heuristics, and checklists to be linked to the evaluation categories through a literature search. Second, five experts judged the metrics regarding their relevance to be considered within evaluation and development of health CAs. The final framework considers nine aspects from a general perspective, five aspects from a response understanding perspective, one aspect from a response generation perspective, and three aspects from an aesthetics perspective. Existing tools and heuristics specifically designed for evaluating CAs were linked to these evaluation aspects (e.g., Bot usability scale, design heuristics for CAs); tools related to mHealth evaluation were adapted when necessary (e.g., aspects from the ISO technical specification for mHealth Apps). The resulting framework comprises aspects to be considered not only as part of a system evaluation, but already during the development. In particular, aspects related to accessibility or security have to be addressed in the design phase (e.g., which input and output options are provided to ensure accessibility?) and have to be verified after the implementation phase. As a next step, transfer of the framework to other types of health CAs has to be studied. The framework has to be validated by applying it during health CA design and development

Concept-Based Retrieval from Critical Incident Reports

Background: Critical incident reporting systems (CIRS) are used as a means to collect anonymously entered information of incidents that occurred for example in a hospital. Analyzing this information helps to identify among others problems in the workflow, in the infrastructure or in processes. Objectives: The entire potential of these sources of experiential knowledge remains often unconsidered since retrieval of relevant reports and their analysis is difficult and time-consuming, and the reporting systems often do not provide support for these tasks. The objective of this work is to develop a method for retrieving reports from the CIRS related to a specific user query. Methods: atural language processing (NLP) and information retrieval (IR) methods are exploited for realizing the retrieval. We compare standard retrieval methods that rely upon frequency of words with an approach that includes a semantic mapping of natural language to concepts of a medical ontology. Results: By an evaluation, we demonstrate the feasibility of semantic document enrichment to improve recall in incident reporting retrieval. It is shown that a combination of standard keyword-based retrieval with semantic search results in highly satisfactory recall values. Conclusion: In future work, the evaluation should be repeated on a larger data set and real-time user evaluation need to be performed to assess user satisfactory with the system and results. Keywords. Information Retrieval, Data Mining, Natural Language Processing, Critical Incidents Reporting

Does Enrichment of Clinical Texts by Ontology Concepts Increases Classification Accuracy?

In the medical domain, multiple ontologies and terminology systems are available. However, existing classification and prediction algorithms in the clinical domain often ignore or insufficiently utilize semantic information as it is provided in those ontologies. To address this issue, we introduce a concept for augmenting embeddings, the input to deep neural networks, with semantic information retrieved from ontologies. To do this, words and phrases of sentences are mapped to concepts of a medical ontology aggregating synonyms in the same concept. A semantically enriched vector is generated and used for sentence classification. We study our approach on a sentence classification task using a real world dataset which comprises 640 sentences belonging to 22 categories. A deep neural network model is defined with an embedding layer followed by two LSTM layers and two dense layers. Our experiments show, classification accuracy without content enriched embeddings is for some categories higher than without enrichment. We conclude that semantic information from ontologies has potential to provide a useful enrichment of text. Future research will assess to what extent semantic relationships from the ontology can be used for enrichment

A Mobile System for Music Anamnesis and Receptive Music Therapy in the Personal Home.

Receptive music therapy is active hearing of music that is specifically selected to cause a certain effect on a person, such as pain reduction, mental opening, confrontation etc. This active, guided hearing could be helpful as a supporting ritual for patients at home and could extend traditional therapy. However, patients are often unable to select the music pieces that might be helpful for them in a current situation. We are suggesting a self-learning decision support system that allows a patient to answer questions on music anamnesis, is ready for inclusion into an electronic health record, and which enables a therapist to compile a therapeutic music program for the patient at home. Beyond this, the system also suggests appropriate music and duration of listening based on the patient’s reported current mental state. In this paper, a concept for such a mobile system for receptive music therapy will be proposed. Keywords Music Therapy; Decision Support Techniques; Mobile Application

What Characterizes Safety of Ambient Assisted Living Technologies?

Ambient assisted living (AAL) technologies aim at increasing an individual's safety at home by early recognizing risks or events that might otherwise harm the individual. A clear definition of safety in the context of AAL is still missing and facets of safety still have to be shaped. The objective of this paper is to characterize the facets of AAL-related safety, to identify opportunities and challenges of AAL regarding safety and to identify open research issues in this context. Papers reporting aspects of AAL-related safety were selected in a literature search. Out of 395 citations retrieved, 28 studies were included in the current review. Two main facets of safety were identified: user safety and system safety. System safety concerns an AAL system's reliability, correctness and data quality. User safety reflects impact on physical and mental health of an individual. Privacy, data safety and security issues, sensor quality and integration of sensor data, as well as technical failures of sensors and systems are reported challenges. To conclude, there is a research gap regarding methods and metrics for measuring user and system safety in the context of AAL technologies

Towards Safe Conversational Agents in Healthcare

Conversational agents (CA) are becoming very popular to deliver digital health interventions. These dialog-based systems are interacting with patients using natural language which might lead to misunderstandings and misinterpretations. To avoid patient harm, safety of health CA has to be ensured. This paper raises awareness on safety when developing and distributing health CA. For this purpose, we identify and describe facets of safety and make recommendations for ensuring safety in health CA. We distinguish three facets of safety: 1) system safety, 2) patient safety, and 3) perceived safety. System safety comprises data security and privacy which has to be considered when selecting technologies and developing the health CA. Patient safety is related to risk monitoring and risk management, to adverse events and content accuracy. Perceived safety concerns a user's perception of the level of danger and user's level of comfort during the use. The latter can be supported when data security is guaranteed and relevant information on the system and its capabilities are provided

Classification of user queries according to a hierarchical medical procedure encoding system using an ensemble classifier

The Swiss classification of surgical interventions (CHOP) has to be used in daily practice by physicians to classify clinical procedures. Its purpose is to encode the delivered healthcare services for the sake of quality assurance and billing. For encoding a procedure, a code of a maximal of 6-digits has to be selected from the classification system, which is currently realized by a rule-based system composed of encoding experts and a manual search in the CHOP catalog. In this paper, we will investigate the possibility of automatic CHOP code generation based on a short query to enable automatic support of manual classification. The wide and deep hierarchy of CHOP and the differences between text used in queries and catalog descriptions are two apparent obstacles for training and deploying a learning-based algorithm. Because of these challenges, there is a need for an appropriate classification approach. We evaluate different strategies (multi-class non-terminal and per-node classifications) with different configurations so that a flexible modular solution with high accuracy and efficiency can be provided. The results clearly show that the per-node binary classification outperforms the non-terminal multi-class classification with an F1-micro measure between 92.6 and 94%. The hierarchical prediction based on per-node binary classifiers achieved a high exact match by the single code assignment on the 5-fold cross-validation. In conclusion, the hierarchical context from the CHOP encoding can be employed by both classifier training and representation learning. The hierarchical features have all shown improvement in the classification performances under different configurations, respectively: the stacked autoencoder and training examples aggregation using true path rules as well as the unified vocabulary space have largely increased the utility of hierarchical features. Additionally, the threshold adaption through Bayesian aggregation has largely increased the vertical reachability of the per node classification. All the trainable nodes can be triggered after the threshold adaption, while the F1 measures at code levels 3–6 have been increased from 6 to 89% after the threshold adaption

How to Evaluate Health Applications with Conversational User Interface?

Application of conversational user interfaces (CUI) or chatbots to healthcare is gaining interest fueled by the rising power of artificial intelligence, increasing popularity of mobile health applications and the desire for engagement and usability. While their use is mainly justified by increasing adherence to mobile health applications and facilitating interactions with the system, the question arises: How can such systems be evaluated in a reliable manner? This paper introduces an evaluation framework for health systems whose core interaction principle is a CUI. We derive quality dimensions and attributes by collecting relevant evaluation aspects from applications that have been developed in previous work and from literature on health chatbots. The collected aspects are aggregated into six thematic categories for chatbot quality, including user experience, linguistic, task-oriented and artificial intelligence perspectives, but also healthcare quality and system quality perspectives. The framework is intended to support developers and researchers in the domain of chatbots in healthcare in selecting relevant quality attributes to be assessed before their systems are distributed to patients