Maximal benefits and possible detrimental effects of binary decision aids

Binary decision aids, such as alerts, are a simple and widely used form of automation. The formal analysis of a user's task performance with an aid sees the process as the combination of information from two detectors who both receive input about an event and evaluate it. The user's decisions are based on the output of the aid and on the information, the user obtains independently. We present a simple method for computing the maximal benefits a user can derive from a binary aid as a function of the user's and the aid's sensitivities. Combining the user and the aid often adds little to the performance the better detector could achieve alone. Also, if users assign non-optimal weights to the aid, performance may drop dramatically. Thus, the introduction of a valid aid can actually lower detection performance, compared to a more sensitive user working alone. Similarly, adding a user to a system with high sensitivity may lower its performance. System designers need to consider the potential adverse effects of introducing users or aids into systems

Measures of Reliance and Compliance in Aided Visual Scanning

Dieser Beitrag ist mit Zustimmung des Rechteinhabers aufgrund einer (DFG geförderten) Allianz- bzw. Nationallizenz frei zugänglich.This publication is with permission of the rights owner freely accessible due to an Alliance licence and a national licence (funded by the DFG, German Research Foundation) respectively.Objective: We study the dependence or independence of reliance and compliance as two responses to alarms to understand the mechanisms behind these responses. Background: Alarms, alerts, and other binary cues affect user behavior in complex ways. The suggestion has been made that there are two different responses to alerts—compliance (the tendency to perform an action cued by the alert) and reliance (the tendency to refrain from actions as long as no alert is issued). The study tests the degree to which these two responses are indeed independent. Method: An experiment tested the effects of the positive and negative predictive values of the alerts (PPV and NPV) on measures of compliance and reliance based on cutoff settings, response times, and subjective confidence. Results: For cutoff settings and response times, compliance was unaffected by the irrelevant NPV, whereas reliance depended on the irrelevant PPV. For subjective estimates, there were no significant effects of the irrelevant variables. Conclusion: Results suggest that compliance is relatively stable and unaffected by irrelevant information (the NPV), whereas reliance is also affected by the PPV. The results support the notion that reliance and compliance are separate, but related, forms of trust. Application: False alarm rates, which affect PPV, determine both the response to alerts (compliance) and the tendency to limit precautions when no alert is issued (reliance)

Error and Threat Detection: A Review and Evaluation of Current Literature

The present project provides a comprehensive review of the literature related to threat and error detection. Although there are current models for understanding the concepts of error and threat, little is known about how individuals detect errors and threats when they occur. Awareness of error and threat is crucial for advancement of safety in the aviation domain. Four areas were discussed related to error and threat detection. First, the general error and threat detection literature was reviewed. Second, the physiological foundations for error and threat detection were discussed. Third, the paper examined cognitive aspects of error and threat detection. Last, the paper elaborated on the role emotion may play in threat detection. The review concludes with suggestions for error and threat management and courses of action that can be taken within the aviation domain to train individuals in error and threat detection

Response Criterion Placement Modulates the Effects of Graded Alerting Systems on Human Performance and Learning in a Target Detection Task

Human operators can perform better with the use of an automated diagnostic aid than without the use of an aid in a signal detection task. This experiment aimed to determine whether any differences existed among graded aids—automated diagnostic aids that use a scale of confidence levels reflecting a spectrum of probabilistic information or uncertainty when making a judgment—that enabled better human detection performance, and either binary or graded aid produced better learning. Participants performed a visual search framed as a medical decision making task. Stimuli were arrays of random polygons (“cells”) generated by distorting a prototype shape. The target was a shape more strongly distorted than the accompanying distracters. A target was present on half of the trials. Each participant performed the task with the assistance of either a binary aid, one of three graded aids, or no aid. The aids’ sensitivities were the same (d′ = 2); the difference between the aids lay in the placement of their decision criteria, which determines a tradeoff between the aid’s predictive value and the frequency with which it makes a diagnosis. The graded aid with 90% reliability provided a judgment on the greatest number of trials, the graded aid with 94% reliability gave a judgment on fewer trials, and the third graded aid with 96% reliability gave a judgment on the least number of trials. The binary aid with 84% reliability gave a judgment on each trial. All aids improved human detection performance, though the graded aids trended towards improving performance more than the binary aid. The binary and graded aids did not produce significantly better or worse learning than did unaided performance. The binary and graded aids did not significantly help learning, but they certainly did not worsen human detection performance when compared to the unaided condition. These results imply that the decision boundaries of a graded alert might be fixed to encourage appropriate reliance on the aid and improve human detection performance, and indicate employing either a graded or binary automated aid may be beneficial to learning in a detection task

Theoretical, Measured and Subjective Responsibility in Aided Decision Making

When humans interact with intelligent systems, their causal responsibility for outcomes becomes equivocal. We analyze the descriptive abilities of a newly developed responsibility quantification model (ResQu) to predict actual human responsibility and perceptions of responsibility in the interaction with intelligent systems. In two laboratory experiments, participants performed a classification task. They were aided by classification systems with different capabilities. We compared the predicted theoretical responsibility values to the actual measured responsibility participants took on and to their subjective rankings of responsibility. The model predictions were strongly correlated with both measured and subjective responsibility. A bias existed only when participants with poor classification capabilities relied less-than-optimally on a system that had superior classification capabilities and assumed higher-than-optimal responsibility. The study implies that when humans interact with advanced intelligent systems, with capabilities that greatly exceed their own, their comparative causal responsibility will be small, even if formally the human is assigned major roles. Simply putting a human into the loop does not assure that the human will meaningfully contribute to the outcomes. The results demonstrate the descriptive value of the ResQu model to predict behavior and perceptions of responsibility by considering the characteristics of the human, the intelligent system, the environment and some systematic behavioral biases. The ResQu model is a new quantitative method that can be used in system design and can guide policy and legal decisions regarding human responsibility in events involving intelligent systems

Understanding the management of electronic test result notifications in the outpatient setting

<p>Abstract</p> <p>Background</p> <p>Notifying clinicians about abnormal test results through electronic health record (EHR) -based "alert" notifications may not always lead to timely follow-up of patients. We sought to understand barriers, facilitators, and potential interventions for safe and effective management of abnormal test result delivery via electronic alerts.</p> <p>Methods</p> <p>We conducted a qualitative study consisting of six 6-8 member focus groups (N = 44) at two large, geographically dispersed Veterans Affairs facilities. Participants included full-time primary care providers, and personnel representing diagnostic services (radiology, laboratory) and information technology. We asked participants to discuss barriers, facilitators, and suggestions for improving timely management and follow-up of abnormal test result notifications and encouraged them to consider technological issues, as well as broader, human-factor-related aspects of EHR use such as organizational, personnel, and workflow.</p> <p>Results</p> <p>Providers reported receiving a large number of alerts containing information unrelated to abnormal test results, many of which were believed to be unnecessary. Some providers also reported lacking proficiency in use of certain EHR features that would enable them to manage alerts more efficiently. Suggestions for improvement included improving display and tracking processes for critical alerts in the EHR, redesigning clinical workflow, and streamlining policies and procedures related to test result notification.</p> <p>Conclusion</p> <p>Providers perceive several challenges for fail-safe electronic communication and tracking of abnormal test results. A multi-dimensional approach that addresses technology as well as the many non-technological factors we elicited is essential to design interventions to reduce missed test results in EHRs.</p

Driver Attention Assessment Using Physiological Measures from EEG, ECG, and EDA Signals †

In this paper, we consider the evaluation of the mental attention state of individuals driving in a simulated environment. We tested a pool of subjects while driving on a highway and trying to overcome various obstacles placed along the course in both manual and autonomous driving scenarios. Most systems described in the literature use cameras to evaluate features such as blink rate and gaze direction. In this study, we instead analyse the subjects' Electrodermal activity (EDA) Skin Potential Response (SPR), their Electrocardiogram (ECG), and their Electroencephalogram (EEG). From these signals we extract a number of physiological measures, including eye blink rate and beta frequency band power from EEG, heart rate from ECG, and SPR features, then investigate their capability to assess the mental state and engagement level of the test subjects. In particular, and as confirmed by statistical tests, the signals reveal that in the manual scenario the subjects experienced a more challenged mental state and paid higher attention to driving tasks compared to the autonomous scenario. A different experiment in which subjects drove in three different setups, i.e., a manual driving scenario and two autonomous driving scenarios characterized by different vehicle settings, confirmed that manual driving is more mentally demanding than autonomous driving. Therefore, we can conclude that the proposed approach is an appropriate way to monitor driver attention

Do cyclists make better drivers? Associations between cycling experience and change detection in road scenes

Efficient processing of visual information is crucial to safe driving. Previous research has demonstrated that driving experience strongly affects attentional allocation, with large differences between novice and experienced drivers. Expanding on this, we explored the influence of non-driving experiences on attentional allocation by comparing drivers with and without cycling experience. Based on situation awareness field studies, we predicted cyclist-drivers would demonstrate superior performance. Participants were 42 experienced drivers (17 female, 25 male) aged 30–50 years (M = 39.8): 20 drivers and 22 cyclist-drivers. The experiment used a change detection flicker task, in which participants must determine whether two alternating images are identical (change-absent) or differ in a single detail (change-present). The changed object was either a road sign, car, pedestrian, or bicycle. Change target significantly affected both accuracy and response time: all participants were slower and less accurate at detecting changes to road signs, compared with when the change was a moving road user (i.e., car, pedestrian, bicycle). Accuracy did not differ significantly between groups, but cyclist-drivers were significantly faster than drivers at identifying changes, with the effect being largest for bicycle and sign changes. The results suggest that cycling experience is associated with more efficient attentional processing for road scenesVanessa Beanland is supported by an Australian Research Council Discovery Early Career Researcher Award (DE150100083)

Metacognition, Numeracy, and Automation-aided Decision-making

Automated decision aids can improve human decision-making but the benefits are often compromised by inefficient use. The current experiment examined whether metacognition—the ability to assess self-performance—and numeracy—the ability to understand and work with numbers—predict the efficiency of automation use in a signal detection task. Two-hundred twenty-one participants classified random dot images as blue or orange dominant, receiving assistance from an 84% reliable decision aid on some trials. Type 1 and metacognitive signal detection measures were estimated from participants’ confidence ratings, and numeracy was measured using a subjective scale. The inefficiency of automation use was assessed by measuring the deviation from optimal bias following cues from the aid (bias error). Data gave strong evidence that metacognition was not associated with bias error, and anecdotal evidence that numeracy and suboptimality were weakly negatively correlated. These results suggest that operators used a strategy of combining the aid’s judgments with their own that is not metacognitively driven, but may depend on numeracy