How to Evaluate Health Applications with Conversational User Interface?

Application of conversational user interfaces (CUI) or chatbots to healthcare is gaining interest fueled by the rising power of artificial intelligence, increasing popularity of mobile health applications and the desire for engagement and usability. While their use is mainly justified by increasing adherence to mobile health applications and facilitating interactions with the system, the question arises: How can such systems be evaluated in a reliable manner? This paper introduces an evaluation framework for health systems whose core interaction principle is a CUI. We derive quality dimensions and attributes by collecting relevant evaluation aspects from applications that have been developed in previous work and from literature on health chatbots. The collected aspects are aggregated into six thematic categories for chatbot quality, including user experience, linguistic, task-oriented and artificial intelligence perspectives, but also healthcare quality and system quality perspectives. The framework is intended to support developers and researchers in the domain of chatbots in healthcare in selecting relevant quality attributes to be assessed before their systems are distributed to patients

Spatial database implementation of fuzzy region connection calculus for analysing the relationship of diseases

Analyzing huge amounts of spatial data plays an important role in many emerging analysis and decision-making domains such as healthcare, urban planning, agriculture and so on. For extracting meaningful knowledge from geographical data, the relationships between spatial data objects need to be analyzed. An important class of such relationships are topological relations like the connectedness or overlap between regions. While real-world geographical regions such as lakes or forests do not have exact boundaries and are fuzzy, most of the existing analysis methods neglect this inherent feature of topological relations. In this paper, we propose a method for handling the topological relations in spatial databases based on fuzzy region connection calculus (RCC). The proposed method is implemented in PostGIS spatial database and evaluated in analyzing the relationship of diseases as an important application domain. We also used our fuzzy RCC implementation for fuzzification of the skyline operator in spatial databases. The results of the evaluation show that our method provides a more realistic view of spatial relationships and gives more flexibility to the data analyst to extract meaningful and accurate results in comparison with the existing methods.Comment: ICEE201

Designing Gamification Concepts for Expert Explainable Artificial Intelligence Evaluation Tasks: A Problem Space Exploration

Artificial intelligence (AI) models are often complex and require additional explanations for use in high-stakes decision-making contexts like healthcare. To this end, explainable AI (XAI) developers must evaluate their explanations with domain experts to ensure understandability. As these evaluations are tedious and repetitive, we look at gamification as a means to motivate and engage experts in XAI evaluation tasks. We explore the problem space associated with gamified expert XAI evaluation. Based on a literature review of 22 relevant studies and seven interviews with experts in XAI evaluation, we elicit knowledge about affected stakeholders, eight needs, eight goals, and seven requirements. Our results help us understand better the problems associated with expert XAI evaluation and paint a broad application potential for gamification to improve XAI expert evaluations. In doing so, we lay the foundation for the design of successful gamification concepts for expert XAI evaluation

Neural-signature methods for structured EHR prediction

Models that can effectively represent structured Electronic Healthcare Records (EHR) are central to an increasing range of applications in healthcare. Due to the sequential nature of health data, Recurrent Neural Networks have emerged as the dominant component within state-of-the-art architectures. The signature transform represents an alternative modelling paradigm for sequential data. This transform provides a non-learnt approach to creating a fixed vector representation of temporal features and has shown strong performances across an increasing number of domains, including medical data. However, the signature method has not yet been applied to structured EHR data. To this end, we follow recent work that enables the signature to be used as a differentiable layer within a neural architecture enabling application in high dimensional domains where calculation would have previously been intractable. Using a heart failure prediction task as an exemplar, we provide an empirical evaluation of different variations of the signature method and compare against state-of-the-art baselines. This first application of neural-signature methods in real-world healthcare data shows a competitive performance when compared to strong baselines and thus warrants further investigation within the health domain

A Process Modelling Framework Based on Point Interval Temporal Logic with an Application to Modelling Patient Flows

This thesis considers an application of a temporal theory to describe and model the patient journey in the hospital accident and emergency (A&E) department. The aim is to introduce a generic but dynamic method applied to any setting, including healthcare. Constructing a consistent process model can be instrumental in streamlining healthcare issues. Current process modelling techniques used in healthcare such as flowcharts, unified modelling language activity diagram (UML AD), and business process modelling notation (BPMN) are intuitive and imprecise. They cannot fully capture the complexities of the types of activities and the full extent of temporal constraints to an extent where one could reason about the flows. Formal approaches such as Petri have also been reviewed to investigate their applicability to the healthcare domain to model processes. Additionally, to schedule patient flows, current modelling standards do not offer any formal mechanism, so healthcare relies on critical path method (CPM) and program evaluation review technique (PERT), that also have limitations, i.e. finish-start barrier. It is imperative to specify the temporal constraints between the start and/or end of a process, e.g., the beginning of a process A precedes the start (or end) of a process B. However, these approaches failed to provide us with a mechanism for handling these temporal situations. If provided, a formal representation can assist in effective knowledge representation and quality enhancement concerning a process. Also, it would help in uncovering complexities of a system and assist in modelling it in a consistent way which is not possible with the existing modelling techniques. The above issues are addressed in this thesis by proposing a framework that would provide a knowledge base to model patient flows for accurate representation based on point interval temporal logic (PITL) that treats point and interval as primitives. These objects would constitute the knowledge base for the formal description of a system. With the aid of the inference mechanism of the temporal theory presented here, exhaustive temporal constraints derived from the proposed axiomatic system’ components serves as a knowledge base. The proposed methodological framework would adopt a model-theoretic approach in which a theory is developed and considered as a model while the corresponding instance is considered as its application. Using this approach would assist in identifying core components of the system and their precise operation representing a real-life domain deemed suitable to the process modelling issues specified in this thesis. Thus, I have evaluated the modelling standards for their most-used terminologies and constructs to identify their key components. It will also assist in the generalisation of the critical terms (of process modelling standards) based on their ontology. A set of generalised terms proposed would serve as an enumeration of the theory and subsume the core modelling elements of the process modelling standards. The catalogue presents a knowledge base for the business and healthcare domains, and its components are formally defined (semantics). Furthermore, a resolution theorem-proof is used to show the structural features of the theory (model) to establish it is sound and complete. After establishing that the theory is sound and complete, the next step is to provide the instantiation of the theory. This is achieved by mapping the core components of the theory to their corresponding instances. Additionally, a formal graphical tool termed as point graph (PG) is used to visualise the cases of the proposed axiomatic system. PG facilitates in modelling, and scheduling patient flows and enables analysing existing models for possible inaccuracies and inconsistencies supported by a reasoning mechanism based on PITL. Following that, a transformation is developed to map the core modelling components of the standards into the extended PG (PG*) based on the semantics presented by the axiomatic system. A real-life case (from the King’s College hospital accident and emergency (A&E) department’s trauma patient pathway) is considered to validate the framework. It is divided into three patient flows to depict the journey of a patient with significant trauma, arriving at A&E, undergoing a procedure and subsequently discharged. Their staff relied upon the UML-AD and BPMN to model the patient flows. An evaluation of their representation is presented to show the shortfalls of the modelling standards to model patient flows. The last step is to model these patient flows using the developed approach, which is supported by enhanced reasoning and scheduling

Development of User Warrant Ontology for Improving Online Health Information Provision

Health information portals (HIP) are gateways to reliable and personalised online health information. In practice, however, searching for information in HIP is still far from being effective due to the intricate nature of health information provision. Previous studies have shown the emerging trend of using domain ontology to address the retrieval issue in online healthcare information. Yet, the suitability of domain ontology alone for HIPs is still questionable due to the varied levels of user behaviour and preferences in information search. Inspired by this problem, we propose an ontology development method grounded on the collaboration between user warrant principles, knowledge engineering, and design science framework. The paper reports the development method and the implementation of such an user-warrant ontology that accommodates user-sensitivity into HIP. The evaluation process is conducted by domain experts responsible for portal management and validates the external semantic of the ontology according to a set of pre-defined evaluation criteria. Results from the application of this methodology to an actual HIP are also reported as this research demonstrates the potential of user warrant ontology to resolve information retrieval problem in HIP

GeoHealth:a location-based service for home healthcare workers

We describe a map-based location-based service ‘GeoHealth ’ for home healthcare workers who attend patients at home within a large geographical area. Informed by field studies of work activities and interviews with care providers, we have designed a mobile location-based service prototype supporting collaboration through information sharing and distributed electronic patient records. The GeoHealth prototype gives the users live contextual information about patients, coworkers, current and scheduled work activities and alarms adapted to their geographical location. The application is web-based and uses Google Maps, Global Positioning System (GPS) and Web 2.0 technology to provide a lightweight, dynamic and interactive representation of the work domain supporting distributed collaboration, communication and peripheral awareness among nomadic workers. Through a user-based evaluation, we found that the healthcare workers were positive towards the use of location-based services in their work, and that the dynamic and interactive geospatial representation of the work domain provided by GeoHealth supported distributed collaboration, communication and peripheral awareness. We also identified areas for improvements

Human factors in mental healthcare : A work system analysis of a community-based program for older adults with depression and dementia

Mental healthcare is a critical but largely unexplored application domain for human factors/ergonomics. This paper reports on a work system evaluation of a home-based dementia and depression care program for older adults, the Aging Brain Care program. The Workflow Elements Model was used to guide data collection and analysis of 59 h of observation, supplemented by key informant input. We identified four actors, 37 artifacts across seven types, ten action categories, and ten outcomes including improved health and safety. Five themes emerged regarding barriers and facilitators to care delivery in the program: the centrality of relationship building; the use of adaptive workarounds; performance of duplicate work; travel and scheduling challenges; and communication-related factors. Findings offer new insight into how mental healthcare services are delivered in a community-based program and key work-related factors shaping program outcomes