Learning from crowds in digital pathology using scalable variational Gaussian processes

This work was supported by the Agencia Estatal de Investigacion of the Spanish Ministerio de Ciencia e Innovacion under contract PID2019-105142RB-C22/AEI/10.13039/501100011033, and the United States National Institutes of Health National Cancer Institute Grants U01CA220401 and U24CA19436201. P.M. contribution was mostly before joining Microsoft Research, when he was supported by La Caixa Banking Foundation (ID 100010434, Barcelona, Spain) through La Caixa Fellowship for Doctoral Studies LCF/BQ/ES17/11600011.The volume of labeled data is often the primary determinant of success in developing machine learning algorithms. This has increased interest in methods for leveraging crowds to scale data labeling efforts, and methods to learn from noisy crowd-sourced labels. The need to scale labeling is acute but particularly challenging in medical applications like pathology, due to the expertise required to generate quality labels and the limited availability of qualified experts. In this paper we investigate the application of Scalable Variational Gaussian Processes for Crowdsourcing (SVGPCR) in digital pathology. We compare SVGPCR with other crowdsourcing methods using a large multi-rater dataset where pathologists, pathology residents, and medical students annotated tissue regions breast cancer. Our study shows that SVGPCR is competitive with equivalent methods trained using goldstandard pathologist generated labels, and that SVGPCR meets or exceeds the performance of other crowdsourcing methods based on deep learning. We also show how SVGPCR can effectively learn the class-conditional reliabilities of individual annotators and demonstrate that Gaussian-process classifiers have comparable performance to similar deep learning methods. These results suggest that SVGPCR can meaningfully engage non-experts in pathology labeling tasks, and that the classconditional reliabilities estimated by SVGPCR may assist in matching annotators to tasks where they perform well.Agencia Estatal de Investigacion of the Spanish Ministerio de Ciencia e Innovacion PID2019-105142RB-C22/AEI/10.13039/501100011033United States Department of Health & Human ServicesNational Institutes of Health (NIH) - USANIH National Cancer Institute (NCI) U01CA220401 U24CA19436201La Caixa Banking Foundation (Barcelona, Spain) Barcelona, Spain) through La Caixa Fellowship 100010434 LCF/BQ/ES17/1160001

Interactive Learning for Multimedia at Large

International audienceInteractive learning has been suggested as a key method for addressing analytic multimedia tasks arising in several domains. Until recently, however, methods to maintain interactive performance at the scale of today's media collections have not been addressed. We propose an interactive learning approach that builds on and extends the state of the art in user relevance feedback systems and high-dimensional indexing for multimedia. We report on a detailed experimental study using the ImageNet and YFCC100M collections, containing 14 million and 100 million images respectively. The proposed approach outperforms the relevant state-of-the-art approaches in terms of interactive performance, while improving suggestion relevance in some cases. In particular, even on YFCC100M, our approach requires less than 0.3 s per interaction round to generate suggestions, using a single computing core and less than 7 GB of main memory

Zero‐shot learning by exploiting class‐related and attribute‐related prior knowledge

The existing attribute‐based zero‐shot learning models at different levels ignore some necessary prior knowledge. It is essential to improve classification accuracy of zero‐shot learning that how to mine attribute‐related and class‐related prior knowledge further being incorporated into the attribute prediction models. For the mining of class‐related prior knowledge, measurement of the class–class correlation by using whitened cosine similarity is proposed. Likewise for the mining of attribute‐related prior knowledge, measurements of the attribute–class and attribute–attribute correlation are proposed by using sparse representation coefficient. Therefore, a novel indirect attribute prediction (IAP) model is presented by exploiting class‐related and attribute‐related prior knowledge (IAP_CAPK). Experimental results on animals with attributes and a‐Pascal/a‐Yahoo datasets show that, when compared with IAP and direct attribute prediction, the proposed IAP_CAPK not only yields more accurate attribute prediction and zero‐shot image classification, but also achieves much higher computational efficiency

Crowdsourced emphysema assessment

\u3cp\u3eClassification of emphysema patterns is believed to be useful for improved diagnosis and prognosis of chronic obstructive pulmonary disease. Emphysema patterns can be assessed visually on lung CT scans. Visual assessment is a complex and time-consuming task performed by experts, making it unsuitable for obtaining large amounts of labeled data. We investigate if visual assessment of emphysema can be framed as an image similarity task that does not require expert. Substituting untrained annotators for experts makes it possible to label data sets much faster and at a lower cost. We use crowd annotators to gather similarity triplets and use t-distributed stochastic triplet embedding to learn an embedding. The quality of the embedding is evaluated by predicting expert assessed emphysema patterns. We find that although performance varies due to low quality triplets and randomness in the embedding, we still achieve a median F \u3csub\u3e1\u3c/sub\u3e score of 0.58 for prediction of four patterns. \u3c/p\u3

Texture retrieval in the wild through detection-based attributes

Capturing the essence of a textile image in a robust way is important to retrieve it in a large repository, especially if it has been acquired in the wild (by taking a photo of the textile of interest). In this paper we show that a texel-based representation fits well with this task. In particular, we refer to Texel-Att, a recent texel-based descriptor which has shown to capture fine grained variations of a texture, for retrieval purposes. After a brief explanation of Texel-Att, we will show in our experiments that this descriptor is robust to distortions resulting from acquisitions in the wild by setting up an experiment in which textures from the ElBa (an Element-Based texture dataset) are artificially distorted and then used to retrieve the original image. We compare our approach with existing descriptors using a simple ranking framework based on distance functions. Results show that even under extreme conditions (such as a down-sampling with a factor of 10), we perform better than alternative approaches