Alternative data representations for a Deep Learning-based segmentation pipeline applied to fetal Doppler echocardiography

Treballs Finals de Grau d'Enginyeria Biomèdica. Facultat de Medicina i Ciències de la Salut. Universitat de Barcelona. Curs: 2021-2022. Director/s: Bart Bijnens & Guillermo Jiménez Pérez. Tutora: Fàtima CrispiDoppler echocardiography is a crucial image acquisition technique in fetal medicine that generates spectrums of blood velocities. The current pipeline for its segmentation is very reliant on manual quantification steps, resulting labour-intensive and time expensive. Given the rise of Deep Learning in the medical image segmentation field, some initial Deep Learning based models have been trained and tested for its automatic segmentation. A project in the scope of a grant awarded by the Bill and Melinda Gates Foundation's Global Health program, has obtained some initial good results. Their baseline solution proposed uses a W-net with 6 levels and a binary mask as data representation with values of 1 from the reference line to the curve position. However, these results could be improved. The aim of this project is to design Deep Learning based models using alternative data representations in order to find an alternative solution that overperforms the baseline solution. The dataset used contains 7063 fetal Doppler echocardiographic images which are split into training, validation and test sets. The model architectures used are U-net and W-net architectures with different levels, from 5 to 7. The data representations proposed are a binary mask around the curve position using different width values, and a linear regression. 24 models are trained combining all the architectures with the several data representations, using Dice loss for binary mask data representation models and mean square error (MSE) loss for models using linear regression. For the performance evaluation, different metrics are used when models predict unseen data from the test set. The results show that the baseline solution overperforms the alternative solutions tested in this project. It is observed that more complex and deep architectures with a data representation based on binary masks that generate big shapes work better for these images. Further alternative solutions can be studied in order to develop a much powerful segmentation tool

Sensing coherent structures from the wall

Any device or application in which a solid body and a fluid are in relative motion has to deal with the presence of drag forces. Counteract such forces in an effcient and environmentally-friendly way has become very important in recent years, in view of the sustainable-development challenges imposed by global authorities. The interaction between any engineering device and a surrounding uid occurs most of the time in a turbulent regime. Despite the chaotic appearance of any turbulent ow, a close inspection reveals the presence of patterns that maintain their structure over a period of time. These patterns, known as coherent structures, are of utmost importance, since they are a low-dimensional expression of a complex highly-dimensional dynamical system that can be used to target affordable control strategies. Taking into account that control models rely on proper identi cation of the ow state, it is necessary to acquire this information in a way that is su ciently robust for real applications. For example, the experimental tools normally used in wind tunnels (for instance, velocity eld measurement techniques like Particle Image Velocimetry) cannot be used in an airborne ight. It becomes necessary to acquire this information from sensors that can be embedded in the device itself. The present thesis attempts to provide a technique based on deep neural networks that allows to reconstruct turbulent ows and their energy-carrying coherent structures from sensors embedded in the wall. The choice of neural networks, characterized by their ability to approximate non-linear relations, is based on the development of new algorithms that has occurred in recent years due to the availability of new computational resources. The rst part the dissertation aims to develop a model that combines the ability of prediction in nonlinear systems of deep neural networks with the compression capabilities of proper orthogonal decomposition (POD). For that purpose, a direct numerical simulation (DNS) of a turbulent channel ow of Ret = 1000 is used. Using wall-shear-stress information, the proposed architecture extracts information into feature maps through a series of convolutional neural networks, which are later converted in POD modes coefficients by using fully-connected neural networks. Since the rst POD modes can be ascribed to the most energetic structures in any ow, the proposed networks aim to provide an estimation of the fi rst 10 POD modes. This approach allows to reconstruct wall-parallel ow elds containing the most energetic structures in the ow. This approach is compared with a baseline linear method, proving that neural networks are able to trace better the nonlinear effect of turbulent ows in the wall-normal direction. The next step is to improve this model to reconstruct as much ow scales as possible. For such purpose, a DNS of turbulent open-channel ow is used, with Ret = [180-550]. The aforementioned network is transformed into a fully convolutional network, which allows to exploit better the spatial organization of the POD modes and retrieve a signifi cant larger amount of flow energy, surpassing 90%. The results show that the prediction quality decays with increasing wall distance. This is because the imprint of the flow structures at the wall is related in a linear way for all scales that have a size of the order of the distance to the wall. In contrast, the smaller scales do not impose their imprint directly on the wall, and the nonlinear relationship between the two attenuates with distance. For these results, the energy spectrum of the flow has also been evaluated, showing that the accuracy loss of the predictions begins with the smallest scales. However, the predictions are capable of recovering the largest structures of the ow, which are the ones that transport the bulk of energy. Furthermore, this network is compared with a network that does not exploit the compressibility advantages of POD. The results are overall similar, although the network based on POD appear slightly more robust for increasing distance from the wall. While the previous networks have used fully-resolved wall information coming from DNS simulation, n real applications it is often difficult to embed such a ne mesh of sensors in the wall. Among the different types of architecture that comprise neural networks, generative adversarial networks (GANs) have stood out for their ability to recover the small-scale details of low-resolution images. In fact, they have already been proposed as a tool to recover resolution in flow measurements. In this thesis the use of GANs to retrieve high-resolution wall information is presented. GANs performance is evaluated for different information loss ratios, showing that even in extreme cases GANs are capable of recovering the most energetic scales of the flow footprint at the wall. Finally, as the use of GANs has been proven successful for both wall and flow fields, it is proposed to use them to make a direct estimation of the flow from wall quantities. The results offer a signi cant improvement in flow reconstruction when compared to the previous results. This improvement can be ascribed to the complexity of GAN architectures, which allows to optimize better the amount of information fed into the network without suffering vanishing-gradient problems. The reconstruction performances are also tested with low-resolution input data. Although the predictions get worse when moving away from the wall also in this case, the rate is lower than in the previous cases. This allows to have reconstructions that capture the coherent structures away from the wall.Cualquier dispositivo o aplicación en la que un cuerpo sólido y un fluido estén en movimiento relativo tiene que lidiar con la presencia de fuerzas de arrastre. A la vista de los desafíos de desarrollo sostenible que han impuesto las autoridades mundiales, contrarrestar estas fuerzas de una manera e ficiente y amistosa con el medio ambiente se ha convertido en un tema de suma importancia. La interacción entre un dispositivo de carácter industrial y el fluido que lo rodea ocurre la mayor parte del tiempo en el régimen turbulento. A pesar de la apariencia caótica que tienen los flujos turbulentos, una inspección de cerca revela la presencia de patrones que mantienen su estructura durante algún tiempo. Estos patrones, conocidos como estructuras coherentes, son de suma importancia, ya que son una expresión de baja dimensión de un sistema dinámico complejo de alta dimensionalidad que puede usarse para desarrollar estrategias de control asequibles. Teniendo en cuenta que los modelos de control se basan en la identi cación adecuada del estado del flujo, es necesario adquirir esta información de una manera que sea lo su ficientemente robusta para aplicaciones reales. Por ejemplo, las herramientas experimentales que se utilizan normalmente en los túneles de viento, como la velocimetría de imágenes de partículas para la medición del campo de velocidad, no se pueden utilizar durante el vuelo de un avión regular. Es necesario adquirir esta información a partir de sensores que se puedan incrustar en el propio dispositivo. La presente tesis intenta proporcionar una técnica basada en redes neuronales profundas que permita reconstruir flujos turbulentos y sus estructuras coherentes portadoras de energía a partir de sensores embebidos en la pared. La elección de las redes neuronales, caracterizadas por su capacidad para aproximar relaciones no lineales, se basa en el desarrollo de nuevos algoritmos que han aparecido en los últimos años, gracias a los avances de los nuevos recursos computacionales. La primera parte de esta disertación tiene como objetivo desarrollar un modelo que combine la capacidad de predicción en sistemas no lineales de redes neuronales profundas con las capacidades de compresión de la Descomposición Modal Ortogonal (Proper Orthogonal Decomposition, POD). Para ello, se utiliza una simulación numérica directa (Direct Numerical Simulation, DNS) de un flujo de canal turbulento, con Ret = 1000. Midiendo los esfuerzos de cortadura en la pared, la arquitectura propuesta extrae información en mapas de características a través de una serie de redes neuronales convolucionales, que luego se convierten en coe cientes de modos POD mediante el uso de redes neuronales completamente conectadas. Dado que los primeros modos POD se pueden atribuir a las estructuras más energéticas en cualquier flujo, las redes propuestas tienen como objetivo proporcionar una estimación de los primeros 10 modos POD. Este enfoque permite reconstruir campos de ujo paralelos a la pared que contienen sus estructuras más energéticas. Este método propuesto se compara con un método lineal de referencia, lo que demuestra que las redes neuronales pueden rastrear mejor el efecto no lineal de los flujos turbulentos en la dirección normal de la pared. El siguiente paso es mejorar este modelo para reconstruir tantas escalas del flujo como sea posible. Para ello se utiliza un DNS de flujo turbulento en canal abierto, con Ret = [180-550]. La mencionada red se transforma en una red totalmente convolucional, lo que permite aprovechar mejor la organización espacial de los modos POD y recuperar una cantidad signi ficativamente mayor de energía de flujo, superando el 90%. Los resultados muestran que la calidad de la predicción decae a medida que aumenta la distancia a la pared. Esto se debe a que la huella de las estructuras del flujo en la pared esáa relacionada de forma lineal para todas las escalas que tienen un tamaño del orden de la distancia a la pared. Por el contrario, las escalas más pequeñas no imponen su huella directamente en la pared y la relación no lineal entre las dos se atenúa con la distancia. Para estos resultados, también se ha evaluado el espectro de energía del flujo, mostrando que la pérdida de precisión de las predicciones comienza con las escalas más pequeñas. Sin embargo, las predicciones son capaces de recuperar las estructuras más grandes del flujo, que son las que transportan la mayor parte de la energía. Además, esta red se compara con una red que no aprovecha las ventajas de compresibilidad de la POD. Los resultados son en general similares, aunque la red basada en la POD parece ligeramente más robusta cuando aumenta la distancia desde la pared. Si bien las redes anteriores han utilizado información de pared completamente resuelta proveniente de la simulación de DNS, en aplicaciones reales es difícil incrustar una malla tan fi na de sensores en la pared. Entre los distintos tipos de arquitectura que componen las redes neuronales, las redes de generación por confrontación (Generative Adversarial Networks, GAN) se han destacado por su capacidad para recuperar detalles a pequeña escala en imágenes de baja resolución. De hecho, ya se han propuesto como herramienta para recuperar la resolución en medidas de flujo. En esta tesis se presenta una alternativa basada en GANs para recuperar información de pared de alta resolución. Se han evaluado las prestaciones de las GANs para diferentes ratios de pérdida de información, demostrando que incluso en casos extremos las GANs son capaces de recuperar las escalas más energéticas de la huella de flujo en la pared. Finalmente, dado que el uso de las GANs ha demostrado ser exitoso tanto para los campos de pared como para los de flujo, se propone usarlos para hacer una estimación directa del flujo a partir de las cantidades de pared. Los resultados ofrecen una mejora signifi cativa en la reconstrucción del flujo en comparación con los resultados anteriores. Esta mejora se puede atribuir a la complejidad de las arquitecturas GAN, que permite optimizar mejor la cantidad de información introducida en la red sin sufrir problemas de desvanecimiento de gradiente. Las prestaciones de reconstrucción también se han probado con datos de entrada de baja resolución. Aunque en este caso las predicciones también empeoran al alejarse del muro, la tasa de disminución es menor que en los casos anteriores. Esto permite tener reconstrucciones que capturan las estructuras coherentes lejos de la pared.Programa de Doctorado en Mecánica de Fluidos por la Universidad Carlos III de Madrid, la Universidad de Jaén, la Universidad de Zaragoza, la Universidad Nacional de Educación a Distancia, la Universidad Politécnica de Madrid y la Universidad Rovira i VirgiliPresidente: Javier Jiménez Sendín.- Secretario: Manuel García-Villalba Navaridas.- Vocal: Sergio Hoyas Calv

A Construction Kit for Efficient Low Power Neural Network Accelerator Designs

Implementing embedded neural network processing at the edge requires efficient hardware acceleration that couples high computational performance with low power consumption. Driven by the rapid evolution of network architectures and their algorithmic features, accelerator designs are constantly updated and improved. To evaluate and compare hardware design choices, designers can refer to a myriad of accelerator implementations in the literature. Surveys provide an overview of these works but are often limited to system-level and benchmark-specific performance metrics, making it difficult to quantitatively compare the individual effect of each utilized optimization technique. This complicates the evaluation of optimizations for new accelerator designs, slowing-down the research progress. This work provides a survey of neural network accelerator optimization approaches that have been used in recent works and reports their individual effects on edge processing performance. It presents the list of optimizations and their quantitative effects as a construction kit, allowing to assess the design choices for each building block separately. Reported optimizations range from up to 10'000x memory savings to 33x energy reductions, providing chip designers an overview of design choices for implementing efficient low power neural network accelerators

Monitoring Urban Changes in Mariupol/Ukraine in 2022/23

The ability to constantly monitor urban changes is of large socio-economic interest. Previous works have already shown approaches in this field with the use of Deep Neural Networks (DNNs) and transfer learning. However, they fell short in demonstrating temporal scale outside of either the training or transfer domain. This work builds on existing research and proves that transfer learning with the use of historic data is a feasible solution, which still allows the urban change monitoring of later years. We considered a case with limited access to public and free Very High Resolution (VHR) imagery to guide the transfer. To provide a high temporal resolution, the core data of our monitoring method comprised multi-modal Synthetic Aperture Radar (SAR) and optical multispectral observations from Sentinel 1 and Sentinel 2, respectively. We chose a practical application of our methods for monitoring urban-related changes in the city of Mariupol in Ukraine during the beginning of the Russo-Ukrainian War in 2022/23. During this conflict, availability of VHR data was limited and hence an inexpensive direct transfer to the years 2022/23 was rendered impossible. Instead, a transfer was made for the years 2017-2020 that provided sufficient public and free VHR data with an application of the transferred model in the years late 2021 to mid-2023. It was shown that transferring for the years 2017-2020 with this inexpensive historical VHR data enabled monitoring during times of war in 2022/23. An ablation study on the impact of the frequency of observations showed our method as resilient to even a large loss of observations. However, it also indicated that our method, despite the multi-modal input, was more dependent on optical observations than SAR observations. Neither the indirect transfer, nor the malfunction of Sentinel 1B had a significant impact on the monitoring capabilities of our method

Lightweight Object Detection Ensemble Framework for Autonomous Vehicles in Challenging Weather Conditions

The computer vision systems driving autonomous vehicles are judged by their ability to detect objects and obstacles in the vicinity of the vehicle in diverse environments. Enhancing this ability of a self-driving car to distinguish between the elements of its environment under adverse conditions is an important challenge in computer vision. For example, poor weather conditions like fog and rain lead to image corruption which can cause a drastic drop in object detection (OD) performance. The primary navigation of autonomous vehicles depends on the effectiveness of the image processing techniques applied to the data collected from various visual sensors. Therefore, it is essential to develop the capability to detect objects like vehicles and pedestrians under challenging conditions such as like unpleasant weather. Ensembling multiple baseline deep learning models under different voting strategies for object detection and utilizing data augmentation to boost the models' performance is proposed to solve this problem. The data augmentation technique is particularly useful and works with limited training data for OD applications. Furthermore, using the baseline models significantly speeds up the OD process as compared to the custom models due to transfer learning. Therefore, the ensembling approach can be highly effective in resource-constrained devices deployed for autonomous vehicles in uncertain weather conditions. The applied techniques demonstrated an increase in accuracy over the baseline models and were able to identify objects from the images captured in the adverse foggy and rainy weather conditions. The applied techniques demonstrated an increase in accuracy over the baseline models and reached 32.75% mean average precision (mAP) and 52.56% average precision (AP) in detecting cars in the adverse fog and rain weather conditions present in the dataset. The effectiveness of multiple voting strategies for bounding box predictions on the dataset is also demonstrated. These strategies help increase the explainability of object detection in autonomous systems and improve the performance of the ensemble techniques over the baseline models

Neural network-based urban change monitoring with deep-temporal multispectral and SAR remote sensing data

Remote-sensing-driven urban change detection has been studied in many ways for decades for a wide field of applications, such as understanding socio-economic impacts, identifying new settlements, or analyzing trends of urban sprawl. Such kinds of analyses are usually carried out manually by selecting high-quality samples that binds them to small-scale scenarios, either temporarily limited or with low spatial or temporal resolution. We propose a fully automated method that uses a large amount of available remote sensing observations for a selected period without the need to manually select samples. This enables continuous urban monitoring in a fully automated process. Furthermore, we combine multispectral optical and synthetic aperture radar (SAR) data from two eras as two mission pairs with synthetic labeling to train a neural network for detecting urban changes and activities. As pairs, we consider European Remote Sensing (ERS-1/2) and Landsat 5 Thematic Mapper (TM) for 1991-2011 and Sentinel 1 and 2 for 2017-2021. For every era, we use three different urban sites-Limassol, Rotterdam, and Liege-with at least 500 km(2) each, and deep observation time series with hundreds and up to over a thousand of samples. These sites were selected to represent different challenges in training a common neural network due to atmospheric effects, different geographies, and observation coverage. We train one model for each of the two eras using synthetic but noisy labels, which are created automatically by combining state-of-the-art methods, without the availability of existing ground truth data. To combine the benefit of both remote sensing types, the network models are ensembles of optical- and SAR-specialized sub-networks. We study the sensitivity of urban and impervious changes and the contribution of optical and SAR data to the overall solution. Our implementation and trained models are available publicly to enable others to utilize fully automated continuous urban monitoring.Web of Science1315art. no. 300

Integrating Technology Into Wildlife Surveys

Technology is rapidly improving and being incorporated into field biology, with survey methods such as machine learning and uncrewed aircraft systems (UAS) headlining efforts. UAS paired with machine learning algorithms have been used to detect caribou, nesting waterfowl and seabirds, marine mammals, white-tailed deer, and more in over 19 studies within the last decade alone. Simultaneously, UAS and machine learning have also been implemented for infrastructure monitoring at wind energy facilities as wind energy construction and use has skyrocketed globally. As part of both pre-construction and regulatory compliance of newly constructed wind energy facilities, monitoring of impacts to wildlife is assessed through ground surveys following the USFWS Land-based Wind Energy Guidelines. To streamline efforts at wind energy facilities and improve efficiency, safety, and accuracy in data collection, UAS platforms may be leveraged to not only monitor infrastructure, but also impacts to wildlife in the form of both pre- and post-construction surveys. In this study, we train, validate, and test a machine learning approach, a convolutional neural network (CNN), in the detection and classification of bird and bat carcasses. Further, we compare the trained CNN to the currently accepted and widely used method of human ground surveyors in a simulated post-construction monitoring scenario. Last, we establish a baseline comparison of manual image review of waterfowl pair surveys with currently used ground surveyors that could inform both pre-construction efforts at energy facilities, along with long-standing federal and state breeding waterfowl surveys. For the initial training of the CNN, we collected 1,807 images of bird and bat carcasses that were split into 80.0% training and 20.0% validation image sets. Overall detection was extremely high at 98.7%. We further explored the dataset by evaluating the trained CNN’s ability to identify species and the variables that impacted identification. Classification of species was successful in 90.5% of images and was associated with sun angle and wind speed. Next, we performed a proof of concept to determine the utility of the trained CNN against ground surveyors in ground covers and with species that were both used in the initial training of the model and novel. Ground surveyors performed similar to those surveying at wind energy facilities with 63.2% detection, while the trained CNN fell short at 28.9%. Ground surveyor detection was weakly associated with carcass density within a plot and strongly with carcass size. Similarly, detection by the CNN was associated with carcass size, ground cover type, visual obstruction of vegetation, and weakly with carcass density within a plot. Finally, we examined differences in breeding waterfowl counts between ground surveyors and UAS image reviewers and found that manual review of UAS imagery yielded similar to slightly higher counts of waterfowl. Significant training, testing, and repeated validation of novel image data sets should be performed prior to implementing survey methods reliant upon machine learning algorithms. Additionally, further research is needed to determine potential biases of counting live waterfowl in aerial imagery, such as bird movement and double counting. While our initial results show that UAS imagery and machine learning can improve upon current techniques, extensive follow-up is strongly recommended in the form of proof-of-concept studies and additional validation to confirm the utility of the application in new environments with new species that allow models to be generalized. Remotely sensed imagery paired with machine learning algorithms have the potential to expedite and standardize monitoring of wildlife at wind energy facilities and beyond, improving data streams and potentially reducing costs for the benefit of both conservation agencies and the energy industry

GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging

Deep Learning (DL) has greatly highlighted the potential impact of optimized machine learning in both the scientific and clinical communities. The advent of open-source DL libraries from major industrial entities, such as TensorFlow (Google), PyTorch (Facebook), and MXNet (Apache), further contributes to DL promises on the democratization of computational analytics. However, increased technical and specialized background is required to develop DL algorithms, and the variability of implementation details hinders their reproducibility. Towards lowering the barrier and making the mechanism of DL development, training, and inference more stable, reproducible, and scalable, without requiring an extensive technical background, this manuscript proposes the Generally Nuanced Deep Learning Framework (GaNDLF). With built-in support for k-fold cross-validation, data augmentation, multiple modalities and output classes, and multi-GPU training, as well as the ability to work with both radiographic and histologic imaging, GaNDLF aims to provide an end-to-end solution for all DL-related tasks, to tackle problems in medical imaging and provide a robust application framework for deployment in clinical workflows