SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound

Identifying and interpreting fetal standard scan planes during 2D ultrasound mid-pregnancy examinations are highly complex tasks which require years of training. Apart from guiding the probe to the correct location, it can be equally difficult for a non-expert to identify relevant structures within the image. Automatic image processing can provide tools to help experienced as well as inexperienced operators with these tasks. In this paper, we propose a novel method based on convolutional neural networks which can automatically detect 13 fetal standard views in freehand 2D ultrasound data as well as provide a localisation of the fetal structures via a bounding box. An important contribution is that the network learns to localise the target anatomy using weak supervision based on image-level labels only. The network architecture is designed to operate in real-time while providing optimal output for the localisation task. We present results for real-time annotation, retrospective frame retrieval from saved videos, and localisation on a very large and challenging dataset consisting of images and video recordings of full clinical anomaly screenings. We found that the proposed method achieved an average F1-score of 0.798 in a realistic classification experiment modelling real-time detection, and obtained a 90.09% accuracy for retrospective frame retrieval. Moreover, an accuracy of 77.8% was achieved on the localisation task.Comment: 12 pages, 8 figures, published in IEEE Transactions in Medical Imagin

Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images

We propose a novel attention gate (AG) model for medical image analysis that automatically learns to focus on target structures of varying shapes and sizes. Models trained with AGs implicitly learn to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task. This enables us to eliminate the necessity of using explicit external tissue/organ localisation modules when using convolutional neural networks (CNNs). AGs can be easily integrated into standard CNN models such as VGG or U-Net architectures with minimal computational overhead while increasing the model sensitivity and prediction accuracy. The proposed AG models are evaluated on a variety of tasks, including medical image classification and segmentation. For classification, we demonstrate the use case of AGs in scan plane detection for fetal ultrasound screening. We show that the proposed attention mechanism can provide efficient object localisation while improving the overall prediction performance by reducing false positives. For segmentation, the proposed architecture is evaluated on two large 3D CT abdominal datasets with manual annotations for multiple organs. Experimental results show that AG models consistently improve the prediction performance of the base architectures across different datasets and training sizes while preserving computational efficiency. Moreover, AGs guide the model activations to be focused around salient regions, which provides better insights into how model predictions are made. The source code for the proposed AG models is publicly available.Comment: Accepted for Medical Image Analysis (Special Issue on Medical Imaging with Deep Learning). arXiv admin note: substantial text overlap with arXiv:1804.03999, arXiv:1804.0533

A Survey on Deep Learning in Medical Image Analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.Comment: Revised survey includes expanded discussion section and reworked introductory section on common deep architectures. Added missed papers from before Feb 1st 201

Plane Localization in 3-D Fetal Neurosonography for Longitudinal Analysis of the Developing Brain.

The parasagittal (PS) plane is a 2-D diagnostic plane used routinely in cranial ultrasonography of the neonatal brain. This paper develops a novel approach to find the PS plane in a 3-D fetal ultrasound scan to allow image-based biomarkers to be tracked from prebirth through the first weeks of postbirth life. We propose an accurate plane-finding solution based on regression forests (RF). The method initially localizes the fetal brain and its midline automatically. The midline on several axial slices is used to detect the midsagittal plane, which is used as a constraint in the proposed RF framework to detect the PS plane. The proposed learning algorithm guides the RF learning method in a novel way by: 1) using informative voxels and voxel informative strength as a weighting within the training stage objective function, and 2) introducing regularization of the RF by proposing a geometrical feature within the training stage. Results on clinical data indicate that the new automated method is more reproducible than manual plane finding obtained by two clinicians

Recent Advances in Artificial Intelligence-Assisted Ultrasound Scanning

Funded by the Spanish Ministry of Economic Affairs and Digital Transformation (Project MIA.2021.M02.0005 TARTAGLIA, from the Recovery, Resilience, and Transformation Plan financed by the European Union through Next Generation EU funds). TARTAGLIA takes place under the R&D Missions in Artificial Intelligence program, which is part of the Spain Digital 2025 Agenda and the Spanish National Artificial Intelligence Strategy.Ultrasound (US) is a flexible imaging modality used globally as a first-line medical exam procedure in many different clinical cases. It benefits from the continued evolution of ultrasonic technologies and a well-established US-based digital health system. Nevertheless, its diagnostic performance still presents challenges due to the inherent characteristics of US imaging, such as manual operation and significant operator dependence. Artificial intelligence (AI) has proven to recognize complicated scan patterns and provide quantitative assessments for imaging data. Therefore, AI technology has the potential to help physicians get more accurate and repeatable outcomes in the US. In this article, we review the recent advances in AI-assisted US scanning. We have identified the main areas where AI is being used to facilitate US scanning, such as standard plane recognition and organ identification, the extraction of standard clinical planes from 3D US volumes, and the scanning guidance of US acquisitions performed by humans or robots. In general, the lack of standardization and reference datasets in this field makes it difficult to perform comparative studies among the different proposed methods. More open-access repositories of large US datasets with detailed information about the acquisition are needed to facilitate the development of this very active research field, which is expected to have a very positive impact on US imaging.Depto. de Estructura de la Materia, Física Térmica y ElectrónicaFac. de Ciencias FísicasTRUEMinistry of Economic Affairs and Digital Transformation from the Recovery, Resilience, and Transformation PlanNext Generation EU fundspu

Automatic image quality assessment and measurement of fetal head in two-dimensional ultrasound image

Owing to the inconsistent image quality existing in routine obstetric ultrasound (US) scans that leads to a large intraobserver and interobserver variability, the aim of this study is to develop a quality-assured, fully automated US fetal head measurement system. A texton-based fetal head segmentation is used as a prerequi- site step to obtain the head region. Textons are calculated using a filter bank designed specific for US fetal head structure. Both shape- and anatomic-based features calculated from the segmented head region are then fed into a random forest classifier to determine the quality of the image (e.g., whether the image is acquired from a correct imaging plane), from which fetal head measurements [biparietal diameter (BPD), occipital–frontal diam- eter (OFD), and head circumference (HC)] are derived. The experimental results show a good performance of our method for US quality assessment and fetal head measurements. The overall precision for automatic image quality assessment is 95.24% with 87.5% sensitivity and 100% specificity, while segmentation performance shows 99.27% (`0.26) of accuracy, 97.07% (`2.3) of sensitivity, 2.23 mm (`0.74) of the maximum symmetric contour distance, and 0.84 mm (`0.28) of the average symmetric contour distance. The statistical analysis results using paired t-test and Bland–Altman plots analysis indicate that the 95% limits of agreement for inter observer variability between the automated measurements and the senior expert measurements are 2.7 mm of BPD, 5.8 mm of OFD, and 10.4 mm of HC, whereas the mean differences are −0.038 ` 1.38 mm, −0.20 ` 2.98 mm, and −0.72 ` 5.36 mm, respectively. These narrow 95% limits of agreements indicate a good level of consistency between the automated and the senior expert’s measurements

Segmentation of fetal 2D images with deep learning: a review

Image segmentation plays a vital role in providing sustainable medical care in this evolving biomedical image processing technology. Nowadays, it is considered one of the most important research directions in the computer vision field. Since the last decade, deep learning-based medical image processing has become a research hotspot due to its exceptional performance. In this paper, we present a review of different deep learning techniques used to segment fetal 2D images. First, we explain the basic ideas of each approach and then thoroughly investigate the methods used for the segmentation of fetal images. Secondly, the results and accuracy of different approaches are also discussed. The dataset details used for assessing the performance of the respective method are also documented. Based on the review studies, the challenges and future work are also pointed out at the end. As a result, it is shown that deep learning techniques are very effective in the segmentation of fetal 2D images.info:eu-repo/semantics/publishedVersio

Real-Time Automatic Fetal Brain Extraction in Fetal MRI by Deep Learning

Brain segmentation is a fundamental first step in neuroimage analysis. In the case of fetal MRI, it is particularly challenging and important due to the arbitrary orientation of the fetus, organs that surround the fetal head, and intermittent fetal motion. Several promising methods have been proposed but are limited in their performance in challenging cases and in real-time segmentation. We aimed to develop a fully automatic segmentation method that independently segments sections of the fetal brain in 2D fetal MRI slices in real-time. To this end, we developed and evaluated a deep fully convolutional neural network based on 2D U-net and autocontext, and compared it to two alternative fast methods based on 1) a voxelwise fully convolutional network and 2) a method based on SIFT features, random forest and conditional random field. We trained the networks with manual brain masks on 250 stacks of training images, and tested on 17 stacks of normal fetal brain images as well as 18 stacks of extremely challenging cases based on extreme motion, noise, and severely abnormal brain shape. Experimental results show that our U-net approach outperformed the other methods and achieved average Dice metrics of 96.52% and 78.83% in the normal and challenging test sets, respectively. With an unprecedented performance and a test run time of about 1 second, our network can be used to segment the fetal brain in real-time while fetal MRI slices are being acquired. This can enable real-time motion tracking, motion detection, and 3D reconstruction of fetal brain MRI.Comment: This work has been submitted to ISBI 201

Characterization of brain development in preterm children using ultrasound images

El període més important per al desenvolupament del cervell humà és la fase fetal. Durant aquest període de quaranta setmanes, es produeixen canvis morfològics importants al cervell humà, incloent un enorme augment de la superfície cerebral després del desenvolupament dels solcs i circumvolucions. En els nadons prematurs, aquests canvis es produeixen en un entorn extrauterí i s’ha demostrat un deteriorament del desenvolupament cerebral en aquesta població a una edat equivalent al terme. Un atles normalitzat de maduració cerebral amb ultrasons cerebrals pot permetre als clínics avaluar aquests canvis setmanalment des del naixement fins a una edat equivalent al terme. Basat en les imatges dels diferents nadons proporcionats per dos investigadors clínics, aquest estudi proposa una aplicació web implementada amb Python i les seves diferents biblioteques, inclòs Dash, i accessible a través de Docker que permet accedir directament a l’aplicació dissenyada i a la seva base de dades. D’aquesta manera, es proporciona una eina que permet fer una primera definició de les diferents ranures manualment per passar-les finalment per un algorisme amb l’objectiu de millorar la precisió i poder exportar tant la imatge com les coordenades que se n’obtenen.El período más importante para el desarrollo del cerebro humano es la fase fetal. Durante este período de cuarenta semanas, se producen importantes cambios morfológicos en el cerebro humano, incluido un gran aumento en la superficie del cerebro a raíz del desarrollo de surcos y circunvoluciones. En los recién nacidos prematuros, estos cambios se producen en un entorno extrauterino y se ha demostrado un deterioro del desarrollo cerebral en esta población a la edad equivalente a término. Un atlas normalizado de maduración cerebral con ecografía cerebral puede permitir a los médicos evaluar estos cambios semanalmente desde el nacimiento hasta la edad equivalente a término. A partir de las imágenes de los diferentes bebés proporcionados por dos investigadores clínicos, este estudio propone una aplicación web implementada con Python y sus diferentes bibliotecas, incluida Dash, y accesible a través de Docker que permite el acceso directo a la aplicación diseñada y su base de datos. De esta forma, se proporciona una herramienta que permite realizar una primera definición de las diferentes ranuras de forma manual para finalmente pasarlas por un algoritmo con el objetivo de mejorar la precisión y poder exportar tanto la imagen como las coordenadas obtenidas de la misma.The most important period for human’s brain development is the fetal phase. During these forty weeks period, important morphological changes take place in the human brain, including a huge increase in the brain surface following the development of sulci and gyri. In preterm newborns these changes occur in an extrauterine environment, and an impaired brain development has been shown in this population at term equivalent age. A normalized atlas of brain maturation with cerebral ultrasound may allow the clinicians to assess these changes weekly from birth to term equivalent age. Based on the images of the different babies provided by two clinical researchers, this study proposes a web application implemented with python and its different libraries, including Dash, and accessible through docker that allows direct access to the designed app and its database. In this way, a tool is provided that allows a first definition of the different grooves to be made manually to finally pass them through an algorithm with the aim of improving precision and being able to export both the image and the coordinates obtained from it

Automatic deep learning-based pipeline for automatic delineation and measurement of fetal brain structures in routine mid-trimester ultrasound images

Introduction: The aim of this study was to develop a pipeline using state-of-the-art deep learning methods to automatically delineate and measure several of the most important brain structures in fetal brain ultrasound (US) images. Methods: The dataset was composed of 5,331 images of the fetal brain acquired during the routine mid-trimester US scan. Our proposed pipeline automatically performs the following three steps: brain plane classification (transventricular, transthalamic, or transcerebellar plane); brain structures delineation (9 different structures); and automatic measurement (from the structure delineations). The methods were trained on a subset of 4,331 images and each step was evaluated on the remaining 1,000 images. Results: Plane classification reached 98.6% average class accuracy. Brain structure delineation obtained an average pixel accuracy higher than 96% and a Jaccard index higher than 70%. Automatic measurements get an absolute error below 3.5% for the four standard head biometries (head circumference, biparietal diameter, occipitofrontal diameter, and cephalic index), 9% for transcerebellar diameter, 12% for cavum septi pellucidi ratio, and 26% for Sylvian fissure operculization degree. Conclusions: The proposed pipeline shows the potential of deep learning methods to delineate fetal head and brain structures and obtain automatic measures of each anatomical standard plane acquired during routine fetal US examination.The research leading to these results has received funding from the Cerebra Foundation for the Brain Injured Child (Carmarthen, Wales,UK) and ASISA foundation.Peer ReviewedPostprint (published version