9,312 research outputs found
Prior-RadGraphFormer: A Prior-Knowledge-Enhanced Transformer for Generating Radiology Graphs from X-Rays
The extraction of structured clinical information from free-text radiology
reports in the form of radiology graphs has been demonstrated to be a valuable
approach for evaluating the clinical correctness of report-generation methods.
However, the direct generation of radiology graphs from chest X-ray (CXR)
images has not been attempted. To address this gap, we propose a novel approach
called Prior-RadGraphFormer that utilizes a transformer model with prior
knowledge in the form of a probabilistic knowledge graph (PKG) to generate
radiology graphs directly from CXR images. The PKG models the statistical
relationship between radiology entities, including anatomical structures and
medical observations. This additional contextual information enhances the
accuracy of entity and relation extraction. The generated radiology graphs can
be applied to various downstream tasks, such as free-text or structured reports
generation and multi-label classification of pathologies. Our approach
represents a promising method for generating radiology graphs directly from CXR
images, and has significant potential for improving medical image analysis and
clinical decision-making.Comment: In GRAIL @ MICCAI 202
A survey on knowledge-enhanced multimodal learning
Multimodal learning has been a field of increasing interest, aiming to
combine various modalities in a single joint representation. Especially in the
area of visiolinguistic (VL) learning multiple models and techniques have been
developed, targeting a variety of tasks that involve images and text. VL models
have reached unprecedented performances by extending the idea of Transformers,
so that both modalities can learn from each other. Massive pre-training
procedures enable VL models to acquire a certain level of real-world
understanding, although many gaps can be identified: the limited comprehension
of commonsense, factual, temporal and other everyday knowledge aspects
questions the extendability of VL tasks. Knowledge graphs and other knowledge
sources can fill those gaps by explicitly providing missing information,
unlocking novel capabilities of VL models. In the same time, knowledge graphs
enhance explainability, fairness and validity of decision making, issues of
outermost importance for such complex implementations. The current survey aims
to unify the fields of VL representation learning and knowledge graphs, and
provides a taxonomy and analysis of knowledge-enhanced VL models
Graph Meets LLMs: Towards Large Graph Models
Large models have emerged as the most recent groundbreaking achievements in
artificial intelligence, and particularly machine learning. However, when it
comes to graphs, large models have not achieved the same level of success as in
other fields, such as natural language processing and computer vision. In order
to promote applying large models for graphs forward, we present a perspective
paper to discuss the challenges and opportunities associated with developing
large graph models. First, we discuss the desired characteristics of large
graph models. Then, we present detailed discussions from three key
perspectives: representation basis, graph data, and graph models. In each
category, we provide a brief overview of recent advances and highlight the
remaining challenges together with our visions. Finally, we discuss valuable
applications of large graph models. We believe this perspective can encourage
further investigations into large graph models, ultimately pushing us one step
closer towards artificial general intelligence (AGI). We are the first to
comprehensively study large graph models, to the best of our knowledge.Comment: Accepted by NeurIPS 2023 New Frontiers in Graph Learning Workshop.
Comments are welcom
- …