Skill, or style? Classification of fetal sonography eye-tracking data

We present a method for classifying human skill at fetal ultrasound scanning from eye-tracking and pupillary data of sonographers. Human skill characterization for this clinical task typically creates groupings of clinician skills such as expert and beginner based on the number of years of professional experience; experts typically have more than 10 years and beginners between 0-5 years. In some cases, they also include trainees who are not yet fully-qualified professionals. Prior work has considered eye movements that necessitates separating eye-tracking data into eye movements, such as fixations and saccades. Our method does not use prior assumptions about the relationship between years of experience and does not require the separation of eye-tracking data. Our best performing skill classification model achieves an F1 score of 98% and 70% for expert and trainee classes respectively. We also show that years of experience as a direct measure of skill, is significantly correlated to the expertise of a sonographer

Automating the human action of first-trimester biometry measurement from real-world freehand ultrasound

Objective: Automated medical image analysis solutions should closely mimic complete human actions to be useful in clinical practice. However, more often an automated image analysis solution represents only part of a human task, which restricts its practical utility. In the case of ultrasound-based fetal biometry, an automated solution should ideally recognize key fetal structures in freehand video guidance, select a standard plane from a video stream and perform biometry. A complete automated solution should automate all three subactions. Methods: In this article, we consider how to automate the complete human action of first-trimester biometry measurement from real-world freehand ultrasound. In the proposed hybrid convolutional neural network (CNN) architecture design, a classification regression-based guidance model detects and tracks fetal anatomical structures (using visual cues) in the ultrasound video. Several high-quality standard planes that contain the mid-sagittal view of the fetus are sampled at multiple time stamps (using a custom-designed confident-frame detector) based on the estimated probability values associated with predicted anatomical structures that define the biometry plane. Automated semantic segmentation is performed on the selected frames to extract fetal anatomical landmarks. A crown–rump length (CRL) estimate is calculated as the mean CRL from these multiple frames. Results: Our fully automated method has a high correlation with clinical expert CRL measurement (Pearson's p = 0.92, R-squared [R2] = 0.84) and a low mean absolute error of 0.834 (weeks) for fetal age estimation on a test data set of 42 videos. Conclusion: A novel algorithm for standard plane detection employs a quality detection mechanism defined by clinical standards, ensuring precise biometric measurements

Self-supervised Representation Learning for Ultrasound Video

Recent advances in deep learning have achieved promising performance for medical image analysis, while in most cases ground-truth annotations from human experts are necessary to train the deep model. In practice, such annotations are expensive to collect and can be scarce for medical imaging applications. Therefore, there is significant interest in learning representations from unlabelled raw data. In this paper, we propose a self-supervised learning approach to learn meaningful and transferable representations from medical imaging video without any type of human annotation. We assume that in order to learn such a representation, the model should identify anatomical structures from the unlabelled data. Therefore we force the model to address anatomy-aware tasks with free supervision from the data itself. Specifically, the model is designed to correct the order of a reshuffled video clip and at the same time predict the geometric transformation applied to the video clip. Experiments on fetal ultrasound video show that the proposed approach can effectively learn meaningful and strong representations, which transfer well to downstream tasks like standard plane detection and saliency prediction.Comment: ISBI 202

Show from Tell:Audio-Visual Modelling in Clinical Settings

Auditory and visual signals usually present together and correlate with each other, not only in natural environments but also in clinical settings. However, the audio-visual modelling in the latter case can be more challenging, due to the different sources of audio/video signals and the noise (both signal-level and semantic-level) in auditory signals -- usually speech. In this paper, we consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations that benefit various clinical tasks, without human expert annotation. A simple yet effective multi-modal self-supervised learning framework is proposed for this purpose. The proposed approach is able to localise anatomical regions of interest during ultrasound imaging, with only speech audio as a reference. Experimental evaluations on a large-scale clinical multi-modal ultrasound video dataset show that the proposed self-supervised method learns good transferable anatomical representations that boost the performance of automated downstream clinical tasks, even outperforming fully-supervised solutions

Audio-visual modelling in a clinical setting

Auditory and visual signals are two primary perception modalities that are usually present together and correlate with each other, not only in natural environments but also in clinical settings. However, audio-visual modelling in the latter case can be more challenging, due to the different sources of audio/video signals and the noise (both signal-level and semantic-level) in auditory signals—usually speech audio. In this study, we consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations that benefit various clinical tasks, without relying on dense supervisory annotations from human experts for the model training. A simple yet effective multi-modal self-supervised learning framework is presented for this purpose. The proposed approach is able to help find standard anatomical planes, predict the focusing position of sonographer’s eyes, and localise anatomical regions of interest during ultrasound imaging. Experimental analysis on a large-scale clinical multi-modal ultrasound video dataset show that the proposed novel representation learning method provides good transferable anatomical representations that boost the performance of automated downstream clinical tasks, even outperforming fully-supervised solutions. Being able to learn such medical representations in a self-supervised manner will contribute to several aspects including a better understanding of obstetric imaging, training new sonographers, more effective assistive tools for human experts, and enhancement of the clinical workflow

Discovering Salient Anatomical Landmarks by Predicting Human Gaze

Anatomical landmarks are a crucial prerequisite for many medical imaging tasks. Usually, the set of landmarks for a given task is predefined by experts. The landmark locations for a given image are then annotated manually or via machine learning methods trained on manual annotations. In this paper, in contrast, we present a method to automatically discover and localize anatomical landmarks in medical images. Specifically, we consider landmarks that attract the visual attention of humans, which we term visually salient landmarks. We illustrate the method for fetal neurosonographic images. First, full-length clinical fetal ultrasound scans are recorded with live sonographer gaze-tracking. Next, a convolutional neural network (CNN) is trained to predict the gaze point distribution (saliency map) of the sonographers on scan video frames. The CNN is then used to predict saliency maps of unseen fetal neurosonographic images, and the landmarks are extracted as the local maxima of these saliency maps. Finally, the landmarks are matched across images by clustering the landmark CNN features. We show that the discovered landmarks can be used within affine image registration, with average landmark alignment errors between 4.1% and 10.9% of the fetal head long axis length.Comment: Accepted at IEEE International Symposium on Biomedical Imaging 2020 (ISBI 2020

Show from Tell: Audio-Visual Modelling in Clinical Settings

Auditory and visual signals usually present together and correlate with each other, not only in natural environments but also in clinical settings. However, the audio-visual modelling in the latter case can be more challenging, due to the different sources of audio/video signals and the noise (both signal-level and semantic-level) in auditory signals -- usually speech. In this paper, we consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations that benefit various clinical tasks, without human expert annotation. A simple yet effective multi-modal self-supervised learning framework is proposed for this purpose. The proposed approach is able to localise anatomical regions of interest during ultrasound imaging, with only speech audio as a reference. Experimental evaluations on a large-scale clinical multi-modal ultrasound video dataset show that the proposed self-supervised method learns good transferable anatomical representations that boost the performance of automated downstream clinical tasks, even outperforming fully-supervised solutions

Audio-visual modelling in a clinical setting

Auditory and visual signals are two primary perception modalities that are usually present together and correlate with each other, not only in natural environments but also in clinical settings. However, audio-visual modelling in the latter case can be more challenging, due to the different sources of audio/video signals and the noise (both signal-level and semantic-level) in auditory signals—usually speech audio. In this study, we consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations that benefit various clinical tasks, without relying on dense supervisory annotations from human experts for the model training. A simple yet effective multi-modal self-supervised learning framework is presented for this purpose. The proposed approach is able to help find standard anatomical planes, predict the focusing position of sonographer’s eyes, and localise anatomical regions of interest during ultrasound imaging. Experimental analysis on a large-scale clinical multi-modal ultrasound video dataset show that the proposed novel representation learning method provides good transferable anatomical representations that boost the performance of automated downstream clinical tasks, even outperforming fully-supervised solutions. Being able to learn such medical representations in a self-supervised manner will contribute to several aspects including a better understanding of obstetric imaging, training new sonographers, more effective assistive tools for human experts, and enhancement of the clinical workflow

International gestational age-specific centiles for umbilical artery doppler indices: A longitudinal prospective cohort study of the INTERGROWTH-21st project

Background: Reference values for umbilical artery Doppler indices are used clinically to assess fetal well-being. However, many studies that have produced reference charts have important methodologic limitations, and these result in significant heterogeneity of reported reference ranges.Objectives: To produce international gestational age-specific centiles for umbilical artery Doppler indices based on longitudinal data and the same rigorous methodology used in the original Fetal Growth Longitudinal Study of the INTERGROWTH-21st Project.Study design: In Phase II of the INTERGROWTH-21st Project (the INTERBIO-21st Study), we prospectively continued enrolling pregnant women according to the same protocol from 3 of the original populations in Pelotas (Brazil), Nairobi (Kenya), and Oxford (United Kingdom) that had participated in the Fetal Growth Longitudinal Study. Women with a singleton pregnancy were recruited at weeks\u27 gestation, confirmed by ultrasound measurement of crown-rump length, and then underwent standardized ultrasound every 5±1 weeks until delivery. From 22 weeks of gestation umbilical artery indices (pulsatility index, resistance index, and systolic/diastolic ratio) were measured in a blinded fashion, using identical equipment and a rigorously standardized protocol. Newborn size at birth was assessed using the international INTERGROWTH-21st Standards, and infants had detailed assessment of growth, nutrition, morbidity, and motor development at 1 and 2 years of age. The appropriateness of pooling data from the 3 study sites was assessed using variance component analysis and standardized site differences. Umbilical artery indices were modeled as functions of the gestational age using an exponential, normal distribution with second-degree fractional polynomial smoothing; goodness of fit for the overall models was assessed.Results: Of the women enrolled at the 3 sites, 1629 were eligible for this study; 431 (27%) met the entry criteria for the construction of normative centiles, similar to the proportion seen in the original fetal growth longitudinal study. They contributed a total of 1243 Doppler measures to the analysis; 74% had 3 measures or more. The healthy low-risk status of the population was confirmed by the low rates of preterm birth (4.9%) and preeclampsia (0.7%). There were no neonatal deaths and satisfactory growth, health, and motor development of the infants at 1 and 2 years of age were documented. Only a very small proportion (2.8%-6.5%) of the variance of Doppler indices was due to between-site differences; in addition, standardized site difference estimates were marginally outside this threshold in only 1 of 27 comparisons, and this supported the decision to pool data from the 3 study sites. All 3 Doppler indices decreased with advancing gestational age. The 3rd, 5th 10th, 50th, 90th, 95th, and 97th centiles according to gestational age for each of the 3 indices are provided, as well as equations to allow calculation of any value as a centile and z scores. The mean pulsatility index according to gestational age = 1.02944 + 77.7456*(gestational age)-2 - 0.000004455*gestational age3.Conclusion: We present here international gestational age-specific normative centiles for umbilical artery Doppler indices produced by studying healthy, low-risk pregnant women living in environments with minimal constraints on fetal growth. The centiles complement the existing INTERGROWTH-21st Standards for assessment of fetal well-being