41 research outputs found
Two-path 3D CNNs for calibration of system parameters for OCT-based motion compensation
Automatic motion compensation and adjustment of an intraoperative imaging
modality's field of view is a common problem during interventions. Optical
coherence tomography (OCT) is an imaging modality which is used in
interventions due to its high spatial resolution of few micrometers and its
temporal resolution of potentially several hundred volumes per second. However,
performing motion compensation with OCT is problematic due to its small field
of view which might lead to tracked objects being lost quickly. We propose a
novel deep learning-based approach that directly learns input parameters of
motors that move the scan area for motion compensation from optical coherence
tomography volumes. We design a two-path 3D convolutional neural network (CNN)
architecture that takes two volumes with an object to be tracked as its input
and predicts the necessary motor input parameters to compensate the object's
movement. In this way, we learn the calibration between object movement and
system parameters for motion compensation with arbitrary objects. Thus, we
avoid error-prone hand-eye calibration and handcrafted feature tracking from
classical approaches. We achieve an average correlation coefficient of 0.998
between predicted and ground-truth motor parameters which leads to sub-voxel
accuracy. Furthermore, we show that our deep learning model is real-time
capable for use with the system's high volume acquisition frequency.Comment: Accepted at SPIE: Medical Imaging 201
Force Estimation from OCT Volumes using 3D CNNs
\textit{Purpose} Estimating the interaction forces of instruments and tissue
is of interest, particularly to provide haptic feedback during robot assisted
minimally invasive interventions. Different approaches based on external and
integrated force sensors have been proposed. These are hampered by friction,
sensor size, and sterilizability. We investigate a novel approach to estimate
the force vector directly from optical coherence tomography image volumes.
\textit{Methods} We introduce a novel Siamese 3D CNN architecture. The
network takes an undeformed reference volume and a deformed sample volume as an
input and outputs the three components of the force vector. We employ a deep
residual architecture with bottlenecks for increased efficiency. We compare the
Siamese approach to methods using difference volumes and two-dimensional
projections. Data was generated using a robotic setup to obtain ground truth
force vectors for silicon tissue phantoms as well as porcine tissue.
\textit{Results} Our method achieves a mean average error of 7.7 +- 4.3 mN
when estimating the force vector. Our novel Siamese 3D CNN architecture
outperforms single-path methods that achieve a mean average error of 11.59 +-
6.7 mN. Moreover, the use of volume data leads to significantly higher
performance compared to processing only surface information which achieves a
mean average error of 24.38 +- 22.0 mN. Based on the tissue dataset, our
methods shows good generalization in between different subjects.
\textit{Conclusions} We propose a novel image-based force estimation method
using optical coherence tomography. We illustrate that capturing the
deformation of subsurface structures substantially improves force estimation.
Our approach can provide accurate force estimates in surgical setups when using
intraoperative optical coherence tomography.Comment: Published in the International Journal of Computer Assisted Radiology
and Surger
Learning Preference-Based Similarities from Face Images using Siamese Multi-Task CNNs
Online dating has become a common occurrence over the last few decades. A key
challenge for online dating platforms is to determine suitable matches for
their users. A lot of dating services rely on self-reported user traits and
preferences for matching. At the same time, some services largely rely on user
images and thus initial visual preference. Especially for the latter approach,
previous research has attempted to capture users' visual preferences for
automatic match recommendation. These approaches are mostly based on the
assumption that physical attraction is the key factor for relationship
formation and personal preferences, interests, and attitude are largely
neglected. Deep learning approaches have shown that a variety of properties can
be predicted from human faces to some degree, including age, health and even
personality traits. Therefore, we investigate the feasibility of bridging
image-based matching and matching with personal interests, preferences, and
attitude. We approach the problem in a supervised manner by predicting
similarity scores between two users based on images of their faces only. The
ground-truth for the similarity matching scores is determined by a test that
aims to capture users' preferences, interests, and attitude that are relevant
for forming romantic relationships. The images are processed by a Siamese
Multi-Task deep learning architecture. We find a statistically significant
correlation between predicted and target similarity scores. Thus, our results
indicate that learning similarities in terms of interests, preferences, and
attitude from face images appears to be feasible to some degree
Feasibility of a markerless tracking system based on optical coherence tomography
Clinical tracking systems are popular but typically require specific tracking
markers. During the last years, scanning speed of optical coherence tomography
(OCT) has increased to A-scan rates above 1 MHz allowing to acquire volume
scans of moving objects. Thorefore, we propose a markerless tracking system
based on OCT to obtain small volumetric images including information of
sub-surface structures at high spatio-temporal resolution. In contrast to
conventional vision based approaches, this allows identifying natural landmarks
even for smooth and homogeneous surfaces. We describe the optomechanical setup
and process flow to evaluate OCT volumes for translations and accordingly
adjust the position of the field-of-view to follow moving samples. While our
current setup is still preliminary, we demonstrate tracking of motion
transversal to the OCT beam of up to 20 mm/s with errors around 0.2 mm and even
better for some scenarios. Tracking is evaluated on a clearly structured and on
a homogeneous phantom as well as on actual tissue samples. The results show
that OCT is promising for fast and precise tracking of smooth, monochromatic
objects in medical scenarios.Comment: Accepted at SPIE Medical Imaging 201
A Deep Learning Approach for Pose Estimation from Volumetric OCT Data
Tracking the pose of instruments is a central problem in image-guided
surgery. For microscopic scenarios, optical coherence tomography (OCT) is
increasingly used as an imaging modality. OCT is suitable for accurate pose
estimation due to its micrometer range resolution and volumetric field of view.
However, OCT image processing is challenging due to speckle noise and
reflection artifacts in addition to the images' 3D nature. We address pose
estimation from OCT volume data with a new deep learning-based tracking
framework. For this purpose, we design a new 3D convolutional neural network
(CNN) architecture to directly predict the 6D pose of a small marker geometry
from OCT volumes. We use a hexapod robot to automatically acquire labeled data
points which we use to train 3D CNN architectures for multi-output regression.
We use this setup to provide an in-depth analysis on deep learning-based pose
estimation from volumes. Specifically, we demonstrate that exploiting volume
information for pose estimation yields higher accuracy than relying on 2D
representations with depth information. Supporting this observation, we provide
quantitative and qualitative results that 3D CNNs effectively exploit the depth
structure of marker objects. Regarding the deep learning aspect, we present
efficient design principles for 3D CNNs, making use of insights from the 2D
deep learning community. In particular, we present Inception3D as a new
architecture which performs best for our application. We show that our deep
learning approach reaches errors at our ground-truth label's resolution. We
achieve a mean average error of \SI{14.89 \pm 9.3}{\micro\metre} and
\SI{0.096 \pm 0.072}{\degree} for position and orientation learning,
respectively.Comment: https://doi.org/10.1016/j.media.2018.03.00
4D Spatio-Temporal Deep Learning with 4D fMRI Data for Autism Spectrum Disorder Classification
Autism spectrum disorder (ASD) is associated with behavioral and
communication problems. Often, functional magnetic resonance imaging (fMRI) is
used to detect and characterize brain changes related to the disorder.
Recently, machine learning methods have been employed to reveal new patterns by
trying to classify ASD from spatio-temporal fMRI images. Typically, these
methods have either focused on temporal or spatial information processing.
Instead, we propose a 4D spatio-temporal deep learning approach for ASD
classification where we jointly learn from spatial and temporal data. We employ
4D convolutional neural networks and convolutional-recurrent models which
outperform a previous approach with an F1-score of 0.71 compared to an F1-score
of 0.65.Comment: Accepted at MIDL 201
Melanoma detection with electrical impedance spectroscopy and dermoscopy using joint deep learning models
The initial assessment of skin lesions is typically based on dermoscopic
images. As this is a difficult and time-consuming task, machine learning
methods using dermoscopic images have been proposed to assist human experts.
Other approaches have studied electrical impedance spectroscopy (EIS) as a
basis for clinical decision support systems. Both methods represent different
ways of measuring skin lesion properties as dermoscopy relies on visible light
and EIS uses electric currents. Thus, the two methods might carry complementary
features for lesion classification. Therefore, we propose joint deep learning
models considering both EIS and dermoscopy for melanoma detection. For this
purpose, we first study machine learning methods for EIS that incorporate
domain knowledge and previously used heuristics into the design process. As a
result, we propose a recurrent model with state-max-pooling which automatically
learns the relevance of different EIS measurements. Second, we combine this new
model with different convolutional neural networks that process dermoscopic
images. We study ensembling approaches and also propose a cross-attention
module guiding information exchange between the EIS and dermoscopy model. In
general, combinations of EIS and dermoscopy clearly outperform models that only
use either EIS or dermoscopy. We show that our attention-based, combined model
outperforms other models with specificities of 34.4% (CI 31.3-38.4), 34.7% (CI
31.0-38.8) and 53.7% (CI 50.1-57.6) for dermoscopy, EIS and the combined model,
respectively, at a clinically relevant sensitivity of 98%.Comment: Accepted at SPIE Medical Imaging 202
A Deep Learning Approach for Motion Forecasting Using 4D OCT Data
Forecasting motion of a specific target object is a common problem for
surgical interventions, e.g. for localization of a target region, guidance for
surgical interventions, or motion compensation. Optical coherence tomography
(OCT) is an imaging modality with a high spatial and temporal resolution.
Recently, deep learning methods have shown promising performance for OCT-based
motion estimation based on two volumetric images. We extend this approach and
investigate whether using a time series of volumes enables motion forecasting.
We propose 4D spatio-temporal deep learning for end-to-end motion forecasting
and estimation using a stream of OCT volumes. We design and evaluate five
different 3D and 4D deep learning methods using a tissue data set. Our best
performing 4D method achieves motion forecasting with an overall average
correlation coefficient of 97.41%, while also improving motion estimation
performance by a factor of 2.5 compared to a previous 3D approach.Comment: Accepted for publication at MIDL 2020:
https://openreview.net/forum?id=WVd56kgR
4D Spatio-Temporal Convolutional Networks for Object Position Estimation in OCT Volumes
Tracking and localizing objects is a central problem in computer-assisted
surgery. Optical coherence tomography (OCT) can be employed as an optical
tracking system, due to its high spatial and temporal resolution. Recently, 3D
convolutional neural networks (CNNs) have shown promising performance for pose
estimation of a marker object using single volumetric OCT images. While this
approach relied on spatial information only, OCT allows for a temporal stream
of OCT image volumes capturing the motion of an object at high volumes rates.
In this work, we systematically extend 3D CNNs to 4D spatio-temporal CNNs to
evaluate the impact of additional temporal information for marker object
tracking. Across various architectures, our results demonstrate that using a
stream of OCT volumes and employing 4D spatio-temporal convolutions leads to a
30% lower mean absolute error compared to single volume processing with 3D
CNNs.Comment: Accepted at CURAC 202
Towards Automatic Lesion Classification in the Upper Aerodigestive Tract Using OCT and Deep Transfer Learning Methods
Early detection of cancer is crucial for treatment and overall patient
survival. In the upper aerodigestive tract (UADT) the gold standard for
identification of malignant tissue is an invasive biopsy. Recently,
non-invasive imaging techniques such as confocal laser microscopy and optical
coherence tomography (OCT) have been used for tissue assessment. In particular,
in a recent study experts classified lesions in the UADT with respect to their
invasiveness using OCT images only. As the results were promising, automatic
classification of lesions might be feasible which could assist experts in their
decision making. Therefore, we address the problem of automatic lesion
classification from OCT images. This task is very challenging as the available
dataset is extremely small and the data quality is limited. However, as similar
issues are typical in many clinical scenarios we study to what extent deep
learning approaches can still be trained and used for decision support.Comment: Accepted for publication at CARS 201