Human-AI Interaction in the Presence of Ambiguity: From Deliberation-based Labeling to Ambiguity-aware AI

Ambiguity, the quality of being open to more than one interpretation, permeates our lives. It comes in different forms including linguistic and visual ambiguity, arises for various reasons and gives rise to disagreements among human observers that can be hard or impossible to resolve. As artificial intelligence (AI) is increasingly infused into complex domains of human decision making it is crucial that the underlying AI mechanisms also support a notion of ambiguity. Yet, existing AI approaches typically assume that there is a single correct answer for any given input, lacking mechanisms to incorporate diverse human perspectives in various parts of the AI pipeline, including data labeling, model development and user interface design. This dissertation aims to shed light on the question of how humans and AI can be effective partners in the presence of ambiguous problems. To address this question, we begin by studying group deliberation as a tool to detect and analyze ambiguous cases in data labeling. We present three case studies that investigate group deliberation in the context of different labeling tasks, data modalities and types of human labeling expertise. First, we present CrowdDeliberation, an online platform for synchronous group deliberation in novice crowd work, and show how worker deliberation affects resolvability and accuracy in text classification tasks of varying subjectivity. We then translate our findings to the expert domain of medical image classification to demonstrate how imposing additional structure on deliberation arguments can improve the efficiency of the deliberation process without compromising its reliability. Finally, we present CrowdEEG, an online platform for collaborative annotation and deliberation of medical time series data, implementing an asynchronous and highly structured deliberation process. Our findings from an observational study with 36 sleep health professionals help explain how disagreements arise and when they can be resolved through group deliberation. Beyond investigating group deliberation within data labeling, we also demonstrate how the resulting deliberation data can be used to support both human and artificial intelligence. To this end, we first present results from a controlled experiment with ten medical generalists, suggesting that reading deliberation data from medical specialists significantly improves generalists' comprehension and diagnostic accuracy on difficult patient cases. Second, we leverage deliberation data to simulate and investigate AI assistants that not only highlight ambiguous cases, but also explain the underlying sources of ambiguity to end users in human-interpretable terms. We provide evidence suggesting that this form of ambiguity-aware AI can help end users to triage and trust AI-provided data classifications. We conclude by outlining the main contributions of this dissertation and directions for future research

A foundation model for generalizable disease detection from retinal images

Medical artificial intelligence (AI) offers great potential for recognizing signs of health conditions in retinal images and expediting the diagnosis of eye diseases and systemic disorders1. However, the development of AI models requires substantial annotation and models are usually task-specific with limited generalizability to different clinical applications2. Here, we present RETFound, a foundation model for retinal images that learns generalizable representations from unlabelled retinal images and provides a basis for label-efficient model adaptation in several applications. Specifically, RETFound is trained on 1.6 million unlabelled retinal images by means of self-supervised learning and then adapted to disease detection tasks with explicit labels. We show that adapted RETFound consistently outperforms several comparison models in the diagnosis and prognosis of sight-threatening eye diseases, as well as incident prediction of complex systemic disorders such as heart failure and myocardial infarction with fewer labelled data. RETFound provides a generalizable solution to improve model performance and alleviate the annotation workload of experts to enable broad clinical AI applications from retinal imaging

A foundation model for generalizable disease detection from retinal images

Medical artificial intelligence (AI) offers great potential for recognizing signs of health conditions in retinal images and expediting the diagnosis of eye diseases and systemic disorders 1. However, the development of AI models requires substantial annotation and models are usually task-specific with limited generalizability to different clinical applications 2. Here, we present RETFound, a foundation model for retinal images that learns generalizable representations from unlabelled retinal images and provides a basis for label-efficient model adaptation in several applications. Specifically, RETFound is trained on 1.6 million unlabelled retinal images by means of self-supervised learning and then adapted to disease detection tasks with explicit labels. We show that adapted RETFound consistently outperforms several comparison models in the diagnosis and prognosis of sight-threatening eye diseases, as well as incident prediction of complex systemic disorders such as heart failure and myocardial infarction with fewer labelled data. RETFound provides a generalizable solution to improve model performance and alleviate the annotation workload of experts to enable broad clinical AI applications from retinal imaging.</p

Learning-by-Concordance (LbC): introducing undergraduate students to the complexity and uncertainty of clinical practice

Background: A current challenge in medical education is the steep exposure to the complexity and uncertainty of clinical practice in early clerkship. The gap between pre-clinical courses and the reality of clinical decision-making can be overwhelming for undergraduate students. The Learning-by-Concordance (LbC) approach aims to bridge this gap by embedding complexity and uncertainty by relying on real-life situations and exposure to expert reasoning processes to support learning. LbC provides three forms of support: 1) expert responses that students compare with their own, 2) expert explanations and 3) recognized scholars’ key-messages.Method: Three different LbC inspired learning tools were used by 900 undergraduate medical students in three courses: Concordance-of-Reasoning in a 1st-year hematology course; Concordance-of-Perception in a 2nd-year pulmonary physio-pathology course, and; Concordance-of-Professional-Judgment with 3rd-year clerkship students. Thematic analysis was conducted on freely volunteered qualitative comments provided by 404 students.Results:  Absence of a right answer was challenging for 1st year concordance-of-reasoning group; the 2nd year visual concordance group found radiology images initially difficult and unnerving and the 3rd year concordance-of-judgment group recognized the importance of divergent expert opinion.Conclusions: Expert panel answers and explanations constitute an example of “cognitive apprenticeship” that could contribute to the development of appropriate professional reasoning processes

Diabetic Retinopathy Classification and Interpretation using Deep Learning Techniques

La retinopatia diabètica és una malaltia crònica i una de les principals causes de ceguesa i discapacitat visual en els pacients diabètics. L'examen ocular a través d'imatges de la retina és utilitzat pels metges per detectar les lesions relacionades amb aquesta malaltia. En aquesta tesi, explorem diferents mètodes innovadors per a la classificació automàtica del grau de malaltia utilitzant imatges del fons d'ull. Per a aquest propòsit, explorem mètodes basats en l'extracció i classificació automàtica, basades en xarxes neuronals profundes. A més, dissenyem un nou mètode per a la interpretació dels resultats. El model està concebut de manera modular per a que pugui ser utilitzat en d'altres xarxes i dominis de classificació. Demostrem experimentalment que el nostre model d'interpretació és capaç de detectar lesions de retina a la imatge únicament a partir de la informació de classificació. A més, proposem un mètode per comprimir la representació interna de la informació de la xarxa. El mètode es basa en una anàlisi de components independents sobre la informació del vector d'atributs intern de la xarxa generat pel model per a cada imatge. Usant el nostre mètode d'interpretació esmentat anteriorment també és possible visualitzar aquests components en la imatge. Finalment, presentem una aplicació experimental del nostre millor model per classificar imatges de retina d'una població diferent, concretament de l'Hospital de Reus. Els mètodes proposats arriben al nivell de rendiment de l'oftalmòleg i són capaços d'identificar amb gran detall les lesions presents en les imatges, que es dedueixen només de la informació de classificació de la imatge.La retinopatía diabética es una enfermedad crónica y una de las principales causas de ceguera y discapacidad visual en los pacientes diabéticos. El examen ocular a través de imágenes de la retina es utilizado por los médicos para detectar las lesiones relacionadas con esta enfermedad. En esta tesis, exploramos diferentes métodos novedosos para la clasificación automática del grado de enfermedad utilizando imágenes del fondo de la retina. Para este propósito, exploramos métodos basados en la extracción y clasificación automática, basadas en redes neuronales profundas. Además, diseñamos un nuevo método para la interpretación de los resultados. El modelo está concebido de manera modular para que pueda ser utilizado utilizando otras redes y dominios de clasificación. Demostramos experimentalmente que nuestro modelo de interpretación es capaz de detectar lesiones de retina en la imagen únicamente a partir de la información de clasificación. Además, proponemos un método para comprimir la representación interna de la información de la red. El método se basa en un análisis de componentes independientes sobre la información del vector de atributos interno de la red generado por el modelo para cada imagen. Usando nuestro método de interpretación mencionado anteriormente también es posible visualizar dichos componentes en la imagen. Finalmente, presentamos una aplicación experimental de nuestro mejor modelo para clasificar imágenes de retina de una población diferente, concretamente del Hospital de Reus. Los métodos propuestos alcanzan el nivel de rendimiento del oftalmólogo y son capaces de identificar con gran detalle las lesiones presentes en las imágenes, que se deducen solo de la información de clasificación de la imagen.Diabetic Retinopathy is a chronic disease and one of the main causes of blindness and visual impairment for diabetic patients. Eye screening through retinal images is used by physicians to detect the lesions related with this disease. In this thesis, we explore different novel methods for the automatic diabetic retinopathy disease grade classification using retina fundus images. For this purpose, we explore methods based in automatic feature extraction and classification, based on deep neural networks. Furthermore, as results reported by these models are difficult to interpret, we design a new method for results interpretation. The model is designed in a modular manner in order to generalize its possible application to other networks and classification domains. We experimentally demonstrate that our interpretation model is able to detect retina lesions in the image solely from the classification information. Additionally, we propose a method for compressing model feature-space information. The method is based on a independent component analysis over the disentangled feature space information generated by the model for each image and serves also for identifying the mathematically independent elements causing the disease. Using our previously mentioned interpretation method is also possible to visualize such components on the image. Finally, we present an experimental application of our best model for classifying retina images of a different population, concretely from the Hospital de Reus. The methods proposed, achieve ophthalmologist performance level and are able to identify with great detail lesions present on images, inferred only from image classification information

Automating the eye examination using optical coherence tomography

Optical coherence tomography (OCT) devices are becoming ubiquitous in eye clinics worldwide to aid the diagnosis and monitoring of eye disease. Much of this uptake relates to the ability to non-invasively capture micron-resolution images, enabling objective and quantitative data to be obtained from ocular structures. Although safe and reasonably quick to perform, the costs involved with operating OCT devices are not trivial, and the requirement for OCT and other imaging in addition to other clinical measures is placing increasing demand on ophthalmology clinics, contributing to fragmented patient pathways and often extended waiting times. In this thesis, a novel “binocular optical coherence tomography” system that seeks to overcome some of the limitations of current commercial OCT systems, is clinically evaluated. This device incorporates many aspects of the eye examination into a single patient-operated instrument, and aims to improve the efficiency and quality of eye care while reducing the overall labour and equipment costs. A progressive framework of testing is followed that includes human factors and usability testing, followed by early stage diagnostic studies to assess the agreement, repeatability, and reproducibility of individual diagnostic features. Health economics analysis of the retinal therapy clinic is used to model cost effectiveness of current practice and with binocular OCT implementation. The binocular OCT and development of other low-cost OCT systems may improve accessibility, however there remains a relative shortage of experts to interpret the images. Artificial intelligence (AI) is likely to play a role in rapid and automated image classification. This thesis explores the application of AI within retinal therapy clinics to predict the onset of exudative age-related macular degeneration in fellow eyes of patients undergoing treatment in their first eye. Together with automated and simultaneous imaging of both eyes with binocular OCT and the potential for low-cost patient-facing systems, AI is likely to have a role in personalising management plans, especially in a future where preventive treatments are available

Connectivity of the Outer Plexiform Layer of the Mouse Retina

The retina has two synaptic layers: In the outer plexiform layer (OPL), signals from the photoreceptors (PRs) are relayed to the bipolar cells (BCs) with one type of horizontal cell (HC) as interneuron. In the inner plexiform layer (IPL), the retinal ganglion cells (RGCs) receive input from the bipolar cells, modulated by multiple types of amacrine cells. The axons of the retinal ganglion cells form the optic nerve which transmit the visual signal to the higher regions of the brain (Masland 2012). Studies of signal processing in the retina usually focus on the inner plexiform layer. Here, the main computations take place such as direction selectivity, orientation selectivity and object motion detection (Gollisch and Meister 2010). However, to fully understand how these computations arise, it is also important to understand how the input to the ganglion cells is computed and thus to understand the functional differences between BC signals. While these are shaped to some extent in the IPL through amacrine cell feedback (Franke et al. 2017), they are also influenced by computations in the OPL (Drinnenberg et al. 2018). Accordingly, it is essential to understand how the bipolar cell signals are formed and what the exact connectivity in the OPL is. This thesis project aims at a quantitative picture of the mouse outer retina connectome. It takes the approach of systematically analyzing connectivity between the cell types in the OPL based on available high-resolution 3D electron microscopy imaging data (Helmstaedter et al. 2013). We reconstructed photoreceptor axon terminals, horizontal cells and bipolar cells, and quantified their contact statistics. We identified a new structure on HC dendrites which likely defines a second synaptic layer in the OPL below the PRs. Based on the reconstructed morphology, we created a biophysical model of a HC dendrite to gain insights into potential functional mechanisms. Our results reveal several new connectivity patterns in the mouse OPL and suggest that HCs perform two functional roles at two distinct output sites at the same time. The project emphasizes how large-scale EM data can boost research on anatomical connectivity and beyond and highlights the value of the resulting data for detailed biophysical modeling. Moreover, it shows how the known amount of complexity increases with the level of detail with which we can study a subject. Beyond that, this thesis project demonstrates the benefits of data sharing and open science which only enabled our studies

Aerospace medicine and biology: A continuing bibliography with indexes, supplement 128, May 1974

This special bibliography lists 282 reports, articles, and other documents introduced into the NASA scientific and technical information system in April 1974