Elicitation and representation of expert knowledge for computer aided diagnosis in mammography

To study how professional radiologists describe, interpret and make decisions about micro-calcifications in mammograms. The purpose was to develop a model of the radiologists' decision making for use in CADMIUM II, a computerized aid for mammogram interpretation that combines symbolic reasoning with image processing

Human-machine diversity in the use of computerised advisory systems: a case study

Computer-based advisory systems form with their users composite, human-machine systems. Redundancy and diversity between the human and the machine are often important for the dependability of such systems. We discuss the modelling approach we applied in a case study. The goal is to assess failure probabilities for the analysis of X-ray films for detecting cancer, performed by a person assisted by a computer-based tool. Differently from most approaches to human reliability assessment, we focus on the effects of failure diversity — or correlation — between humans and machines. We illustrate some of the modelling and prediction problems, especially those caused by the presence of the human component. We show two alternative models, with their pros and cons, and illustrate, via numerical examples and analytically, some interesting and non-intuitive answers to questions about reliability assessment and design choices for human-computer systems

How to discriminate between computer-aided and computer-hindered decisions: a case study in mammography

Background. Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals (“readers”) interpreted 180 mammograms, both with and without computer support. Method. We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader’s discriminating ability (Youden’s determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). Results. Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003–0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034–0.257) for the 15 relatively difficult cancers. Conclusions. Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems

Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography

To investigate the effects of incorrect computer output on the reliability of the decisions of human users. This work followed an independent UK clinical trial that evaluated the impact of computer-aided detection(CAD) in breast screening. The aim was to use data from this trial to feed into probabilistic models (similar to those used in "reliability engineering") which would detect and assess possible ways of improving the human-CAD interaction. Some analyses required extra data; therefore, two supplementary studies were conducted. Study 1 was designed to elucidate the effects of computer failure on human performance. Study 2 was conducted to clarify unexpected findings from Study 1

Gaining assurance in a voter-verifiable voting system

The literature on e-voting systems has many examples of discussion of the correctness of the computer and communication algorithms of such systems, as well as discussions of their vulnerabilities. However, a gap in the literature concerns the practical need (before adoption of a specific e-voting system) for a complete case demonstrating that the system as a whole has sufficiently high probability of exhibiting the desired properties when in use in an actual election. This paper discusses the problem of producing such a case, with reference to a specific system: a version of the Prêt à Voter scheme for voter-verifiable e-voting. We show a possible organisation of a case in terms of four main requirements – accuracy, privacy, termination and ‘trustedness’– and show some of the detailed organisation that such a case should have, the diverse kinds of evidence that needs to be gathered and some of the interesting difficulties that arise

Why Are People's Decisions Sometimes Worse with Computer Support?

In many applications of computerised decision support, a recognised source of undesired outcomes is operators' apparent over-reliance on automation. For instance, an operator may fail to react to a potentially dangerous situation because a computer fails to generate an alarm. However, the very use of terms like "over-reliance" betrays possible misunderstandings of these phenomena and their causes, which may lead to ineffective corrective action (e.g. training or procedures that do not counteract all the causes of the apparently "over-reliant" behaviour). We review relevant literature in the area of "automation bias" and describe the diverse mechanisms that may be involved in human errors when using computer support. We discuss these mechanisms, with reference to errors of omission when using "alerting systems", with the help of examples of novel counterintuitive findings we obtained from a case study in a health care application, as well as other examples from the literature

The population of SNe/SNRs in the starburst galaxy Arp 220. A self-consistent analysis of 20 years of VLBI monitoring

The nearby ultra-luminous infrared galaxy (ULIRG) Arp 220 is an excellent laboratory for studies of extreme astrophysical environments. For 20 years, Very Long Baseline Interferometry (VLBI) has been used to monitor a population of compact sources thought to be supernovae (SNe), supernova remnants (SNRs) and possibly active galactic nuclei (AGNs). Using new and archival VLBI data spanning 20 years, we obtain 23 high-resolution radio images of Arp 220 at wavelengths from 18 cm to 2 cm. From model-fitting to the images we obtain estimates of flux densities and sizes of all detected sources. We detect radio continuum emission from 97 compact sources and present flux densities and sizes for all analysed observation epochs. We find evidence for a LD-relation within Arp 220, with larger sources being less luminous. We find a compact source LF $n(L)\propto L^\beta$ with $\beta=-2.19\pm0.15$, similar to SNRs in normal galaxies. Based on simulations we argue that there are many relatively large and weak sources below our detection threshold. The observations can be explained by a mixed population of SNe and SNRs, where the former expand in a dense circumstellar medium (CSM) and the latter interact with the surrounding interstellar medium (ISM). Nine sources are likely luminous, type IIn SNe. This number of luminous SNe correspond to few percent of the total number of SNe in Arp 220 which is consistent with a total SN-rate of 4 yr$^{-1}$ as inferred from the total radio emission given a normal stellar initial mass function (IMF). Based on the fitted luminosity function, we argue that emission from all compact sources, also below our detection threshold, make up at most 20\% of the total radio emission at GHz frequencies.Comment: Accepted for publication in Astronomy and Astrophysic

CAD in mammography: lesion-level versus case-level analysis of the effects of prompts on human decisions

Object: To understand decision processes in CAD-supported breast screening by analysing how prompts affect readers’ judgements of individual mammographic features (lesions). To this end we analysed hitherto unexamined details of reports completed by mammogram readers in an earlier evaluation of a CAD tool. Material and methods: Assessments of lesions were extracted from 5,839 reports for 59 cancer cases. Statistical analyses of these data focused on what features readers considered when recalling a cancer case and how readers reacted to CAD prompts. Results: About 13.5% of recall decisions were found to be caused by responses to features other than those indicating actual cancer. Effects of CAD: lesions were more likely to be examined if prompted; the presence of a prompt on a cancer increased the probability of both detection and recall especially for less accurate readers in subtler cases; lack of prompts made cancer features less likely to be detected; false prompts made non-cancer features more likely to be classified as cancer. Conclusion: The apparent lack of impact reported for CAD in some studies is plausibly due to CAD systematically affecting readers’ identification of individual features, in a beneficial way for certain combinations of readers and features and a damaging way for others. Mammogram readers do not ignore prompts. Methodologically, assessing CAD by numbers of recalled cancer cases may be misleading

Use of computer-aided detection (CAD) tools in screening mammography: a multidisciplinary investigation

We summarise a set of analyses and studies conducted to assess the effects of the use of a computer-aided detection (CAD) tool in breast screening. We have used an interdisciplinary approach that combines: (a) statistical analyses inspired by reliability modelling in engineering; (b) experimental studies of decisions of mammography experts using the tool, interpreted in the light of human factors psychology; and (c) ethnographic observations of the use of the tool both in trial conditions and in everyday screening practice. Our investigations have shown patterns of human behaviour and effects of computer-based advice that would not have been revealed by a standard clinical trial approach. For example, we found that the negligible measured effect of CAD could be explained by a range of effects on experts' decisions, beneficial in some cases and detrimental in others. There is some evidence of the latter effects being due to the experts using the computer tool differently from the intentions of the developers. We integrate insights from the different pieces of evidence and highlight their implications for the design, evaluation and deployment of this sort of computer tool