Expert-augmented machine learning.

Machine learning is proving invaluable across disciplines. However, its success is often limited by the quality and quantity of available data, while its adoption is limited by the level of trust afforded by given models. Human vs. machine performance is commonly compared empirically to decide whether a certain task should be performed by a computer or an expert. In reality, the optimal learning strategy may involve combining the complementary strengths of humans and machines. Here, we present expert-augmented machine learning (EAML), an automated method that guides the extraction of expert knowledge and its integration into machine-learned models. We used a large dataset of intensive-care patient data to derive 126 decision rules that predict hospital mortality. Using an online platform, we asked 15 clinicians to assess the relative risk of the subpopulation defined by each rule compared to the total sample. We compared the clinician-assessed risk to the empirical risk and found that, while clinicians agreed with the data in most cases, there were notable exceptions where they overestimated or underestimated the true risk. Studying the rules with greatest disagreement, we identified problems with the training data, including one miscoded variable and one hidden confounder. Filtering the rules based on the extent of disagreement between clinician-assessed risk and empirical risk, we improved performance on out-of-sample data and were able to train with less data. EAML provides a platform for automated creation of problem-specific priors, which help build robust and dependable machine-learning models in critical applications

Expert-Augmented Machine Learning

Machine Learning is proving invaluable across disciplines. However, its success is often limited by the quality and quantity of available data, while its adoption by the level of trust that models afford users. Human vs. machine performance is commonly compared empirically to decide whether a certain task should be performed by a computer or an expert. In reality, the optimal learning strategy may involve combining the complementary strengths of man and machine. Here we present Expert-Augmented Machine Learning (EAML), an automated method that guides the extraction of expert knowledge and its integration into machine-learned models. We use a large dataset of intensive care patient data to predict mortality and show that we can extract expert knowledge using an online platform, help reveal hidden confounders, improve generalizability on a different population and learn using less data. EAML presents a novel framework for high performance and dependable machine learning in critical applications

Division of Labor between Humans and Algorithms in Healthcare: The Case of Surgery Duration Predictions

For many healthcare applications a collaboration of humans and algorithms has been shown to be superior to pure automation in terms of performance. However, the healthcare sector is characterized by shortages in personnel, which can lead to an excessive workload for the employees and thus makes automation highly beneficial to reduce human workload. In our paper, we consider a combination of different work modes and evaluate whether humans have to be involved in every instance of a task or whether they can be replaced by an AI for some instances. We analyze the potential of segmenting tasks based on who is involved in their completion: Either an AI or a human complete the task individually, or they complete the task together. Considering the case of surgery duration predictions and using a dataset from a university hospital, we observe that human effort could be decreased while maintaining a high prediction performance

Augmenting public sector data-driven decision support systems with expert knowledge: case of OTT

Public sector data-driven decision support systems are uniquely challenging to design due to the ramifications they have on the societal level. Accountability and ethical considerations require these systems to arrive at an equilibirium between accuracy and interpretability amid various implementation and data constraints. While these systems need to contribute to legitimate governance through reasoned and explainable decision-making, they also need to accurately model the policy outcomes they were designed to support. Inopportunely, inductive data-driven systems struggle to solve problems that rely on heuristic input. In this thesis, a particular knowledge engineering technique was adopted to augment a public sector Machine Learning decision support tool with domain expert knowledge. The case in question is OTT – a job-seeker profiling tool used by the Estonian Unemployment Insurance Fund to predict the long-term unemployment risks of their clients. Upon augmenting it with knowledge from caseworkers and data scientists associated with the project, some evidence was found that accounting for expert knowledge in probabilistic data-driven models can lead to a model that performs better on new out-of-sample data and is more in line with underlying domain rules. This yields important implications on the future of Machine Learning in the public sector as it opens up new potential use cases in avenues where 1) labelled training data is hard to come by, 2) a more generalizable model is preferred due to frequent changes in the surrounding context, 3) a model has to perfectly mimic domain logic for interpretability and explainability reasons.https://www.ester.ee/record=b5508371*es

From Artificial Intelligence (AI) to Intelligence Augmentation (IA): Design Principles, Potential Risks, and Emerging Issues

We typically think of artificial intelligence (AI) as focusing on empowering machines with human capabilities so that they can function on their own, but, in truth, much of AI focuses on intelligence augmentation (IA), which is to augment human capabilities. We propose a framework for designing intelligent augmentation (IA) systems and it addresses six central questions about IA: why, what, who/whom, how, when, and where. To address the how aspect, we introduce four guiding principles: simplification, interpretability, human-centeredness, and ethics. The what aspect includes an IA architecture that goes beyond the direct interactions between humans and machines by introducing their indirect relationships through data and domain. The architecture also points to the directions for operationalizing the IA design simplification principle. We further identify some potential risks and emerging issues in IA design and development to suggest new questions for future IA research and to foster its positive impact on humanity

Requirements towards optimizing analytics in industrial processes

Algorithms and the Foundations of Software technolog

Improving ECG Classification Interpretability Using Saliency Maps

Cardiovascular disease is a large worldwide healthcare issue; symptoms often present suddenly with minimal warning. The electrocardiogram (ECG) is a fast, simple and reliable method of evaluating the health of the heart, by measuring electrical activity recorded through electrodes placed on the skin. ECGs often need to be analyzed by a cardiologist, taking time which could be spent on improving patient care and outcomes.Because of this, automatic ECG classification systems using machine learning have been proposed, which can learn complex interactions between ECG features and use this to detect abnormalities. However, algorithms built for this purpose often fail to generalize well to unseen data, reporting initially impressive results which drop dramatically when applied to new environments. Additionally, machine learning algorithms suffer a ‘black-box’ issue, in which it is difficult to determine how a decision has been made. This is vital for applications in healthcare, as clinicians need to be able to verify the process of evaluation in order to trust the algorithm.This paper proposes a method for visualizing model decisions across each class in the MIT-BIH arrhythmia dataset, using adapted saliency maps averaged across complete classes to determine what patterns are being learned. We do this by building two algorithms based on state-of-the-art models. This paper highlights how these maps can be used to find problems in the model which could be affecting generalizability and model performance. Comparing saliency maps across complete classes gives an overall impression of confounding variables or other biases in the model, unlike what would be highlighted when comparing saliency maps on an ECG-by-ECG basis

Correlated storage assignment approach in warehouses: A systematic literature review

Purpose: Correlation-based storage assignment approach has been intensively explored during the last three decades to improve the order picking efficiency. The purpose of this study is to present a comprehensive assessment of the literature about the state-of-the-art techniques used to solve correlated storage location assignment problems (CSLAP). Design/methodology/approach: A systematic literature review has been carried out based on content analysis to identify, select, analyze, and critically summarize all the studies available on CSLAP. This study begins with the selection of relevant keywords, and narrowing down the selected papers based on various criteria. Findings: Most correlated storage assignment problems are expressed as NP-hard integer programming models. The studies have revealed that CSLAP is evaluated with many approaches. The solution methods can be mainly categorized into heuristic approach, meta-heuristic approach, and data mining approach. With the advancement of computing power, researchers have taken up the challenge of solving more complex storage assignment problems. Furthermore, applications of the models developed are being tested on actual industry data to comprehend the efficiency of the models. Practical implications: The content of this article can be used as a guide to help practitioners and researchers to become adequately knowledgeable on CSLAP for their future work. Originality/value: Since there has been no recent state-of-the-art evaluation of CSLAP, this paper fills that need by systematizing and unifying recent work and identifying future research scopes

Quantifying Earth system interactions for sustainable food production via expert elicitation

Several safe boundaries of critical Earth system processes have already been crossed due to human perturbations; not accounting for their interactions may further narrow the safe operating space for humanity. Using expert knowledge elicitation, we explored interactions among seven variables representing Earth system processes relevant to food production, identifying many interactions little explored in Earth system literature. We found that green water and land system change affect other Earth system processes strongly, while land, freshwater and ocean components of biosphere integrity are the most impacted by other Earth system processes, most notably blue water and biogeochemical flows. We also mapped a complex network of mechanisms mediating these interactions and created a future research prioritization scheme based on interaction strengths and existing knowledge gaps. Our study improves the understanding of Earth system interactions, with sustainability implications including improved Earth system modelling and more explicit biophysical limits for future food production