Investigating Data Visualization Decision Errors: Do Tools Enable Users to Make Bad Decisions and more confidently?

The current generation of Analytics and Business Intelligence (ABI) tools are easy to use and present relevant information in an elegant form. Their promise to “bring analytics to everyone in the company” would mean that the users creating visualizations and making decisions based on these visualizations may not have the training in data science to understand if they are drawing conclusions that are supported by the data. This study proposes to investigate the impact of cognitive biases and the Kruger-Dunning effect within the use of ABI tools on the resultant decision quality and the confidence the decision maker places in that decision

Predicting user confidence during visual decision making

© 2018 ACM People are not infallible consistent “oracles”: their confidence in decision-making may vary significantly between tasks and over time. We have previously reported the benefits of using an interface and algorithms that explicitly captured and exploited users’ confidence: error rates were reduced by up to 50% for an industrial multi-class learning problem; and the number of interactions required in a design-optimisation context was reduced by 33%. Having access to users’ confidence judgements could significantly benefit intelligent interactive systems in industry, in areas such as intelligent tutoring systems and in health care. There are many reasons for wanting to capture information about confidence implicitly. Some are ergonomic, but others are more “social”—such as wishing to understand (and possibly take account of) users’ cognitive state without interrupting them. We investigate the hypothesis that users’ confidence can be accurately predicted from measurements of their behaviour. Eye-tracking systems were used to capture users’ gaze patterns as they undertook a series of visual decision tasks, after each of which they reported their confidence on a 5-point Likert scale. Subsequently, predictive models were built using “conventional” machine learning approaches for numerical summary features derived from users’ behaviour. We also investigate the extent to which the deep learning paradigm can reduce the need to design features specific to each application by creating “gaze maps”—visual representations of the trajectories and durations of users’ gaze fixations—and then training deep convolutional networks on these images. Treating the prediction of user confidence as a two-class problem (confident/not confident), we attained classification accuracy of 88% for the scenario of new users on known tasks, and 87% for known users on new tasks. Considering the confidence as an ordinal variable, we produced regression models with a mean absolute error of ≈0.7 in both cases. Capturing just a simple subset of non-task-specific numerical features gave slightly worse, but still quite high accuracy (e.g., MAE ≈ 1.0). Results obtained with gaze maps and convolutional networks are competitive, despite not having access to longer-term information about users and tasks, which was vital for the “summary” feature sets. This suggests that the gaze-map-based approach forms a viable, transferable alternative to handcrafting features for each different application. These results provide significant evidence to confirm our hypothesis, and offer a way of substantially improving many interactive artificial intelligence applications via the addition of cheap non-intrusive hardware and computationally cheap prediction algorithms

Diagnosing serious infections in acutely ill children in ambulatory care (ERNIE 2 study protocol, part A): diagnostic accuracy of a clinical decision tree and added value of a point-of-care C-reactive protein test and oxygen saturation

Background: Acute illness is the most common presentation of children to ambulatory care. In contrast, serious infections are rare and often present at an early stage. To avoid complications or death, early recognition and adequate referral are essential. In a recent large study children were included prospectively to construct a symptom-based decision tree with a sensitivity and negative predictive value of nearly 100%. To reduce the number of false positives, point-of-care tests might be useful, providing an immediate result at bedside. The most probable candidate is C-reactive protein, as well as a pulse oximetry. Methods: This is a diagnostic accuracy study of signs, symptoms and point-of-care tests for serious infections. Acutely ill children presenting to a family physician or paediatrician will be included consecutively in Flanders, Belgium. Children testing positive on the decision tree will get a point-of-care C-reactive protein test. Children testing negative will randomly either receive a point-of-care C-reactive protein test or usual care. The outcome of interest is hospital admission more than 24 hours with a serious infection within 10 days. Aiming to include over 6500 children, we will report the diagnostic accuracy of the decision tree (+/- the point-of-care C-reactive protein test or pulse oximetry) in sensitivity, specificity, positive and negative likelihood ratios, and positive and negative predictive values. New diagnostic algorithms will be constructed through classification and regression tree and multiple logistic regression analysis. Discussion: We aim to improve detection of serious infections, and present a practical tool for diagnostic triage of acutely ill children in primary care. We also aim to reduce the number of investigations and admissions in children with non-serious infections

Data and Predictive Analytics Use for Logistics and Supply Chain Management

Purpose The purpose of this paper is to explore the social process of Big Data and predictive analytics (BDPA) use for logistics and supply chain management (LSCM), focusing on interactions among technology, human behavior and organizational context that occur at the technology’s post-adoption phases in retail supply chain (RSC) organizations. Design/methodology/approach The authors follow a grounded theory approach for theory building based on interviews with senior managers of 15 organizations positioned across multiple echelons in the RSC. Findings Findings reveal how user involvement shapes BDPA to fit organizational structures and how changes made to the technology retroactively affect its design and institutional properties. Findings also reveal previously unreported aspects of BDPA use for LSCM. These include the presence of temporal and spatial discontinuities in the technology use across RSC organizations. Practical implications This study unveils that it is impossible to design a BDPA technology ready for immediate use. The emergent process framework shows that institutional and social factors require BDPA use specific to the organization, as the technology comes to reflect the properties of the organization and the wider social environment for which its designers originally intended. BDPA is, thus, not easily transferrable among collaborating RSC organizations and requires managerial attention to the institutional context within which its usage takes place. Originality/value The literature describes why organizations will use BDPA but fails to provide adequate insight into how BDPA use occurs. The authors address the “how” and bring a social perspective into a technology-centric area

Survey of data mining approaches to user modeling for adaptive hypermedia

The ability of an adaptive hypermedia system to create tailored environments depends mainly on the amount and accuracy of information stored in each user model. Some of the difficulties that user modeling faces are the amount of data available to create user models, the adequacy of the data, the noise within that data, and the necessity of capturing the imprecise nature of human behavior. Data mining and machine learning techniques have the ability to handle large amounts of data and to process uncertainty. These characteristics make these techniques suitable for automatic generation of user models that simulate human decision making. This paper surveys different data mining techniques that can be used to efficiently and accurately capture user behavior. The paper also presents guidelines that show which techniques may be used more efficiently according to the task implemented by the applicatio

Designing for Appropriate Reliance: The Roles of AI Uncertainty Presentation, Initial User Decision, and User Demographics in AI-Assisted Decision-Making

Appropriate reliance is critical to achieving synergistic human-AI collaboration. For instance, when users over-rely on AI assistance, their human-AI team performance is bounded by the model's capability. This work studies how the presentation of model uncertainty may steer users' decision-making toward fostering appropriate reliance. Our results demonstrate that showing the calibrated model uncertainty alone is inadequate. Rather, calibrating model uncertainty and presenting it in a frequency format allow users to adjust their reliance accordingly and help reduce the effect of confirmation bias on their decisions. Furthermore, the critical nature of our skin cancer screening task skews participants' judgment, causing their reliance to vary depending on their initial decision. Additionally, step-wise multiple regression analyses revealed how user demographics such as age and familiarity with probability and statistics influence human-AI collaborative decision-making. We discuss the potential for model uncertainty presentation, initial user decision, and user demographics to be incorporated in designing personalized AI aids for appropriate reliance.Comment: Accepted to CSCW202

Do People Make Decisions Under Risk Based on Ignorance? An Empirical Test of the Priority Heuristic against Cumulative Prospect Theory

Brandstätter, Gigerenzer and Hertwig (2006) put forward the priority heuristic (PH) as a fast and frugal heuristic for decisions under risk. According to the PH, individuals do not make trade-offs between gains and probabilities, as proposed by expected utility models such as cumulative prospect theory (CPT), but use information in a non-compensatory manner and ignore information. We conducted three studies to test the PH empirically by analyzing individual choice patterns, decision times and information search parameters in diagnostic decision tasks. Results on all three dependent variables conflict with the predictions of the PH and can be better explained by the CPT. The predictive accuracy of the PH was high for decision tasks in which the predic-tions align with the predictions of the CPT but very low for decision tasks in which this was not the case. The findings indicate that earlier results supporting the PH might have been caused by the selection of decision tasks that were not diagnostic for the PH as compared to CPT.Decision Strategy, Fast and Frugal Heuristics, Bounded Rationality, Decision Latency, Process Tracing, Cumulative Prospect Theory