2,157 research outputs found
Multimorbidity Content-Based Medical Image Retrieval Using Proxies
Content-based medical image retrieval is an important diagnostic tool that
improves the explainability of computer-aided diagnosis systems and provides
decision making support to healthcare professionals. Medical imaging data, such
as radiology images, are often multimorbidity; a single sample may have more
than one pathology present. As such, image retrieval systems for the medical
domain must be designed for the multi-label scenario. In this paper, we propose
a novel multi-label metric learning method that can be used for both
classification and content-based image retrieval. In this way, our model is
able to support diagnosis by predicting the presence of diseases and provide
evidence for these predictions by returning samples with similar pathological
content to the user. In practice, the retrieved images may also be accompanied
by pathology reports, further assisting in the diagnostic process. Our method
leverages proxy feature vectors, enabling the efficient learning of a robust
feature space in which the distance between feature vectors can be used as a
measure of the similarity of those samples. Unlike existing proxy-based
methods, training samples are able to assign to multiple proxies that span
multiple class labels. This multi-label proxy assignment results in a feature
space that encodes the complex relationships between diseases present in
medical imaging data. Our method outperforms state-of-the-art image retrieval
systems and a set of baseline approaches. We demonstrate the efficacy of our
approach to both classification and content-based image retrieval on two
multimorbidity radiology datasets
Comparing Neural and Attractiveness-based Visual Features for Artwork Recommendation
Advances in image processing and computer vision in the latest years have
brought about the use of visual features in artwork recommendation. Recent
works have shown that visual features obtained from pre-trained deep neural
networks (DNNs) perform very well for recommending digital art. Other recent
works have shown that explicit visual features (EVF) based on attractiveness
can perform well in preference prediction tasks, but no previous work has
compared DNN features versus specific attractiveness-based visual features
(e.g. brightness, texture) in terms of recommendation performance. In this
work, we study and compare the performance of DNN and EVF features for the
purpose of physical artwork recommendation using transactional data from
UGallery, an online store of physical paintings. In addition, we perform an
exploratory analysis to understand if DNN embedded features have some relation
with certain EVF. Our results show that DNN features outperform EVF, that
certain EVF features are more suited for physical artwork recommendation and,
finally, we show evidence that certain neurons in the DNN might be partially
encoding visual features such as brightness, providing an opportunity for
explaining recommendations based on visual neural models.Comment: DLRS 2017 workshop, co-located at RecSys 201
A clinically motivated self-supervised approach for content-based image retrieval of CT liver images
Deep learning-based approaches for content-based image retrieval (CBIR) of computed tomography (CT) liver images is an active field of research, but suffer from some critical limitations. First, they are heavily reliant on labeled data, which can be challenging and costly to acquire. Second, they lack transparency and explainability, which limits the trustworthiness of deep CBIR systems. We address these limitations by: (1) Proposing a self-supervised learning framework that incorporates domain-knowledge into the training procedure, and, (2) by providing the first representation learning explainability analysis in the context of CBIR of CT liver images. Results demonstrate improved performance compared to the standard self-supervised approach across several metrics, as well as improved generalization across datasets. Further, we conduct the first representation learning explainability analysis in the context of CBIR, which reveals new insights into the feature extraction process. Lastly, we perform a case study with cross-examination CBIR that demonstrates the usability of our proposed framework. We believe that our proposed framework could play a vital role in creating trustworthy deep CBIR systems that can successfully take advantage of unlabeled data
Semantic bottleneck for computer vision tasks
This paper introduces a novel method for the representation of images that is
semantic by nature, addressing the question of computation intelligibility in
computer vision tasks. More specifically, our proposition is to introduce what
we call a semantic bottleneck in the processing pipeline, which is a crossing
point in which the representation of the image is entirely expressed with
natural language , while retaining the efficiency of numerical representations.
We show that our approach is able to generate semantic representations that
give state-of-the-art results on semantic content-based image retrieval and
also perform very well on image classification tasks. Intelligibility is
evaluated through user centered experiments for failure detection
Explainability in Music Recommender Systems
The most common way to listen to recorded music nowadays is via streaming
platforms which provide access to tens of millions of tracks. To assist users
in effectively browsing these large catalogs, the integration of Music
Recommender Systems (MRSs) has become essential. Current real-world MRSs are
often quite complex and optimized for recommendation accuracy. They combine
several building blocks based on collaborative filtering and content-based
recommendation. This complexity can hinder the ability to explain
recommendations to end users, which is particularly important for
recommendations perceived as unexpected or inappropriate. While pure
recommendation performance often correlates with user satisfaction,
explainability has a positive impact on other factors such as trust and
forgiveness, which are ultimately essential to maintain user loyalty.
In this article, we discuss how explainability can be addressed in the
context of MRSs. We provide perspectives on how explainability could improve
music recommendation algorithms and enhance user experience. First, we review
common dimensions and goals of recommenders' explainability and in general of
eXplainable Artificial Intelligence (XAI), and elaborate on the extent to which
these apply -- or need to be adapted -- to the specific characteristics of
music consumption and recommendation. Then, we show how explainability
components can be integrated within a MRS and in what form explanations can be
provided. Since the evaluation of explanation quality is decoupled from pure
accuracy-based evaluation criteria, we also discuss requirements and strategies
for evaluating explanations of music recommendations. Finally, we describe the
current challenges for introducing explainability within a large-scale
industrial music recommender system and provide research perspectives.Comment: To appear in AI Magazine, Special Topic on Recommender Systems 202
Scalable Privacy-Compliant Virality Prediction on Twitter
The digital town hall of Twitter becomes a preferred medium of communication
for individuals and organizations across the globe. Some of them reach
audiences of millions, while others struggle to get noticed. Given the impact
of social media, the question remains more relevant than ever: how to model the
dynamics of attention in Twitter. Researchers around the world turn to machine
learning to predict the most influential tweets and authors, navigating the
volume, velocity, and variety of social big data, with many compromises. In
this paper, we revisit content popularity prediction on Twitter. We argue that
strict alignment of data acquisition, storage and analysis algorithms is
necessary to avoid the common trade-offs between scalability, accuracy and
privacy compliance. We propose a new framework for the rapid acquisition of
large-scale datasets, high accuracy supervisory signal and multilanguage
sentiment prediction while respecting every privacy request applicable. We then
apply a novel gradient boosting framework to achieve state-of-the-art results
in virality ranking, already before including tweet's visual or propagation
features. Our Gradient Boosted Regression Tree is the first to offer
explainable, strong ranking performance on benchmark datasets. Since the
analysis focused on features available early, the model is immediately
applicable to incoming tweets in 18 languages.Comment: AffCon@AAAI-19 Best Paper Award; Presented at AAAI-19 W1: Affective
Content Analysi
Towards Explainable Interactive Multi-Modal Video Retrieval with vitrivr
This paper presents the most recent iteration of the vitrivr multimedia retrieval system for its participation in the Video Browser Showdown (VBS) 2021. Building on existing functionality for interactive multi-modal retrieval, we overhaul query formulation and results presentation for queries which specify temporal context, extend our database with index structures for similarity search and present experimental functionality aimed at improving the explainability of results with the objective of better supporting users in the selection of results and the provision of relevance feedback
- …