Artificial intelligence in histopathology image analysis for cancer precision medicine

In recent years, there have been rapid advancements in the field of computational pathology. This has been enabled through the adoption of digital pathology workflows that generate digital images of histopathological slides, the publication of large data sets of these images and improvements in computing infrastructure. Objectives in computational pathology can be subdivided into two categories, first the automation of routine workflows that would otherwise be performed by pathologists and second the addition of novel capabilities. This thesis focuses on the development, application, and evaluation of methods in this second category, specifically the prediction of gene expression from pathology images and the registration of pathology images among each other. In Study I, we developed a computationally efficient cluster-based technique to perform transcriptome-wide predictions of gene expression in prostate cancer from H&E-stained whole-slide-images (WSIs). The suggested method outperforms several baseline methods and is non-inferior to single-gene CNN predictions, while reducing the computational cost with a factor of approximately 300. We included 15,586 transcripts that encode proteins in the analysis and predicted their expression with different modelling approaches from the WSIs. In a cross-validation, 6,618 of these predictions were significantly associated with the RNA-seq expression estimates with FDR-adjusted p-values <0.001. Upon validation of these 6,618 expression predictions in a held-out test set, the association could be confirmed for 5,419 (81.9%). Furthermore, we demonstrated that it is feasible to predict the prognostic cell-cycle progression score with a Spearman correlation to the RNA-seq score of 0.527 [0.357, 0.665]. The objective of Study II is the investigation of attention layers in the context of multiple-instance-learning for regression tasks, exemplified by a simulation study and gene expression prediction. We find that for gene expression prediction, the compared methods are not distinguishable regarding their performance, which indicates that attention mechanisms may not be superior to weakly supervised learning in this context. Study III describes the results of the ACROBAT 2022 WSI registration challenge, which we organised in conjunction with the MICCAI 2022 conference. Participating teams were ranked on the median 90th percentile of distances between registered and annotated target landmarks. Median 90th percentiles for eight teams that were eligible for ranking in the test set consisting of 303 WSI pairs ranged from 60.1 µm to 15,938.0 µm. The best performing method therefore has a score slightly below the median 90th percentile of distances between first and second annotator of 67.0 µm. Study IV describes the data set that we published to facilitate the ACROBAT challenge. The data set is available publicly through the Swedish National Data Service SND and consists of 4,212 WSIs from 1,153 breast cancer patients. Study V is an example of the application of WSI registration for computational pathology. In this study, we investigate the possibility to register invasive cancer annotations from H&E to KI67 WSIs and then subsequently train cancer detection models. To this end, we compare the performance of models optimised with registered annotations to the performance of models that were optimised with annotations generated for the KI67 WSIs. The data set consists of 272 female breast cancer cases, including an internal test set of 54 cases. We find that in this test set, the performance of both models is not distinguishable regarding performance, while there are small differences in model calibration

Knowledge Elicitation in Deep Learning Models

Embora o aprendizado profundo (mais conhecido como deep learning) tenha se tornado uma ferramenta popular na solução de problemas modernos em vários domínios, ele apresenta um desafio significativo - a interpretabilidade. Esta tese percorre um cenário de elicitação de conhecimento em modelos de deep learning, lançando luz sobre a visualização de características, mapas de saliência e técnicas de destilação. Estas técnicas foram aplicadas a duas arquiteturas: redes neurais convolucionais (CNNs) e um modelo de pacote (Google Vision). A nossa investigação forneceu informações valiosas sobre a sua eficácia na elicitação e interpretação do conhecimento codificado. Embora tenham demonstrado potencial, também foram observadas limitações, sugerindo espaço para mais desenvolvimento neste campo. Este trabalho não só realça a necessidade de modelos de deep learning mais transparentes e explicáveis, como também impulsiona o desenvolvimento de técnicas para extrair conhecimento. Trata-se de garantir uma implementação responsável e enfatizar a importância da transparência e compreensão no aprendizado de máquina. Além de avaliar os métodos existentes, esta tese explora também o potencial de combinar múltiplas técnicas para melhorar a interpretabilidade dos modelos de deep learning. Uma mistura de visualização de características, mapas de saliência e técnicas de destilação de modelos foi usada de uma maneira complementar para extrair e interpretar o conhecimento das arquiteturas escolhidas. Os resultados experimentais destacam a utilidade desta abordagem combinada, revelando uma compreensão mais abrangente dos processos de tomada de decisão dos modelos. Além disso, propomos um novo modelo para a elicitação sistemática de conhecimento em deep learning, que integra de forma coesa estes métodos. Este quadro demonstra o valor de uma abordagem holística para a interpretabilidade do modelo, em vez de se basear num único método. Por fim, discutimos as implicações éticas do nosso trabalho. À medida que os modelos de deep learning continuam a permear vários setores, desde a saúde até às finanças, garantir que as suas decisões são explicáveis e justificadas torna-se cada vez mais crucial. A nossa investigação sublinha esta importância, preparando o terreno para a criação de sistemas de inteligência artificial mais transparentes e responsáveis no futuro.Though a buzzword in modern problem-solving across various domains, deep learning presents a significant challenge - interpretability. This thesis journeys through a landscape of knowledge elicitation in deep learning models, shedding light on feature visualization, saliency maps, and model distillation techniques. These techniques were applied to two deep learning architectures: convolutional neural networks (CNNs) and a black box package model (Google Vision). Our investigation provided valuable insights into their effectiveness in eliciting and interpreting the encoded knowledge. While they demonstrated potential, limitations were also observed, suggesting room for further development in this field. This work does not just highlight the need for more transparent, more explainable deep learning models, it gives a gentle nudge to developing innovative techniques to extract knowledge. It is all about ensuring responsible deployment and emphasizing the importance of transparency and comprehension in machine learning. In addition to evaluating existing methods, this thesis also explores the potential for combining multiple techniques to enhance the interpretability of deep learning models. A blend of feature visualization, saliency maps, and model distillation techniques was used in a complementary manner to extract and interpret the knowledge from our chosen architectures. Experimental results highlight the utility of this combined approach, revealing a more comprehensive understanding of the models' decision-making processes. Furthermore, we propose a novel framework for systematic knowledge elicitation in deep learning, which cohesively integrates these methods. This framework showcases the value of a holistic approach toward model interpretability rather than relying on a single method. Lastly, we discuss the ethical implications of our work. As deep learning models continue to permeate various sectors, from healthcare to finance, ensuring their decisions are explainable and justified becomes increasingly crucial. Our research underscores this importance, laying the groundwork for creating more transparent, accountable AI systems in the future

Real-time Biomechanical Modeling for Intraoperative Soft Tissue Registration

Computer assisted surgery systems intraoperatively support the surgeon by providing information on the location of hidden risk and target structures during surgery. However, soft tissue deformations make intraoperative registration (and thus intraoperative navigation) difficult. In this work, a novel, biomechanics based approach for real-time soft tissue registration from sparse intraoperative sensor data such as stereo endoscopic images is presented to overcome this problem

Image computing tools for the investigation of the neurological effects of preterm birth and corticosteroid administration

In this thesis we present a range of computational tools for medical imaging purposes within two main research projects. The first one is a methodological project oriented towards the improvement of the performance of a numerical computation utilised in diffeomorphic image registration. The second research project is a pre-clinical study aimed at the investigation of the effects of antenatal corticosteroids in a preterm rabbit animal model. In the first part we addressed the problem of integrating stationary velocity fields. This mathematical challenge had originated with early studies in fluid dynamics and had been subsequently mathematically formalised in the Lie group theory. Given a tangent velocity field defined in the tridimensional space as in input, the goal is to compute the position of the particles to which the velocity field is applied. This computation, also called numerical Lie exponential, is a fundamental component of several medical image registration algorithm based on diffeomorphisms, i.e. bijective differentiable maps with differentiable inverse. It is as well a widely utilised tool in computational anatomy to quantify the differences between two anatomical shapes measuring the parameters of the transformation that belongs to a metric vector space. The resulting new class of algorithms introduced in this thesis was created combining the known scaling and squaring algorithm with a class of numerical integrators aimed to solve systems of ordinary differential equations called exponential integrators. The introduced scaling and squaring based approximated exponential integrator algorithm have improved the computational time and accuracy respect to the state- of-the-art methods. The second part of the research is a pre-clinical trial carried forward in collab- oration with the Department of Development and Regeneration, Woman and Child Cluster at the KU Leuven University. The clinical research question is related to the understanding of the possible negative effects of administering antenatal cor- ticosteroids for preterm birth. To tackle this problem we designed and started a pre-clinical study using a New Zealand perinatal rabbit model. In this part of the research I was involved in the research team to provide the tools to automatise the data analysis and to eliminate the time consuming and non reproducible manual segmentation step. The main result of this collaboration is the creation of the first multi-modal multi-atlas for the newborn rabbit brain. This is embedded in a segmentation propagation and label fusion algorithm at the core of the proposed open-sourced automatic pipeline, having as input the native scanner format and as output the main MRI readouts, such as volume, fractional anisotropy and mean diffusivity

Data-driven patient-specific breast modeling: a simple, automatized, and robust computational pipeline

Background: Breast-conserving surgery is the most acceptable option for breast cancer removal from an invasive and psychological point of view. During the surgical procedure, the imaging acquisition using Magnetic Image Resonance is performed in the prone configuration, while the surgery is achieved in the supine stance. Thus, a considerable movement of the breast between the two poses drives the tumor to move, complicating the surgeon's task. Therefore, to keep track of the lesion, the surgeon employs ultrasound imaging to mark the tumor with a metallic harpoon or radioactive tags. This procedure, in addition to an invasive characteristic, is a supplemental source of uncertainty. Consequently, developing a numerical method to predict the tumor movement between the imaging and intra-operative configuration is of significant interest. Methods: In this work, a simulation pipeline allowing the prediction of patient-specific breast tumor movement was put forward, including personalized preoperative surgical drawings. Through image segmentation, a subject-specific finite element biomechanical model is obtained. By first computing an undeformed state of the breast (equivalent to a nullified gravity), the estimated intra-operative configuration is then evaluated using our developed registration methods. Finally, the model is calibrated using a surface acquisition in the intra-operative stance to minimize the prediction error. Findings: The capabilities of our breast biomechanical model to reproduce real breast deformations were evaluated. To this extent, the estimated geometry of the supine breast configuration was computed using a corotational elastic material model formulation. The subject-specific mechanical properties of the breast and skin were assessed, to get the best estimates of the prone configuration. The final results are a Mean Absolute Error of 4.00 mm for the mechanical parameters E_breast = 0.32 kPa and E_skin = 22.72 kPa. The optimized mechanical parameters are congruent with the recent state-of-the-art. The simulation (including finding the undeformed and prone configuration) takes less than 20 s. The Covariance Matrix Adaptation Evolution Strategy optimizer converges on average between 15 to 100 iterations depending on the initial parameters for a total time comprised between 5 to 30 min. To our knowledge, our model offers one of the best compromises between accuracy and speed. The model could be effortlessly enriched through our recent work to facilitate the use of complex material models by only describing the strain density energy function of the material. In a second study, we developed a second breast model aiming at mapping a generic model embedding breast-conserving surgical drawing to any patient. We demonstrated the clinical applications of such a model in a real-case scenario, offering a relevant education tool for an inexperienced surgeon

Maintenance Management of Wind Turbines

“Maintenance Management of Wind Turbines” considers the main concepts and the state-of-the-art, as well as advances and case studies on this topic. Maintenance is a critical variable in industry in order to reach competitiveness. It is the most important variable, together with operations, in the wind energy industry. Therefore, the correct management of corrective, predictive and preventive politics in any wind turbine is required. The content also considers original research works that focus on content that is complementary to other sub-disciplines, such as economics, finance, marketing, decision and risk analysis, engineering, etc., in the maintenance management of wind turbines. This book focuses on real case studies. These case studies concern topics such as failure detection and diagnosis, fault trees and subdisciplines (e.g., FMECA, FMEA, etc.) Most of them link these topics with financial, schedule, resources, downtimes, etc., in order to increase productivity, profitability, maintainability, reliability, safety, availability, and reduce costs and downtime, etc., in a wind turbine. Advances in mathematics, models, computational techniques, dynamic analysis, etc., are employed in analytics in maintenance management in this book. Finally, the book considers computational techniques, dynamic analysis, probabilistic methods, and mathematical optimization techniques that are expertly blended to support the analysis of multi-criteria decision-making problems with defined constraints and requirements

An ALE method for simuations of elastic surfaces in flow

Die Dynamik von elastischen Membranen, Kapseln und Schalen hat sich zu einem aktiven Forschungsgebiet in der simulationsgestützten Physik und Biologie entwickelt. Die dünne Oberfläche dieser elastischen Materialien ermöglicht es, sie effizient als Hyperfläche zu approximieren. Solche Oberflächen reagieren auf Dehnungen in Oberflächenrichtung und Verformungen in Normalenrichtung mit einer elastischen Kraft. Zusätzlich können Oberflächenspannungskräfte auftreten. In dieser Arbeit präsentieren wir eine neuartige Arbitrary Lagrangian-Eulerian (ALE) Methode um solche in (Navier-Stokes) Fluiden eingebetteten elastischen Schalen zu simulieren. Dadurch, dass das Gitter an die elastische Oberfläche angepasst ist, kombiniert die vorgeschlagene Methode hohe Genauigkeit mit Effizienz in der Berechnung der Lösungen. Folglich kann man die Simulationen mit einer verhältnismäßig geringen Gitterauflösung durchführen. Der Fokus dieser Arbeit liegt bei achsensymmetrischen Formen und Strömungen, wie sie bei vielen biophysikalischen Anwendungen zu finden sind. Neben einer allgemeinen dreidimensionalen Beschreibung formulieren wir achsensymmetrische Kräfte auf der Oberfläche, für welche wir eine Diskretisierung mit der Finite Differenzen Methode vorschlagen, welche an eine Finite-Elemente Methode für die umgebenden Fluide gekoppelt ist. Weiterhin entwickeln wir eine Strategie zur impliziten Kopplung der Kräfte, um Zeitschrittrestriktionen zu reduzieren. In verschiedenen numerischen Tests werden wir zeigen, dass akkurate Ergebnisse schon in einer Größenordnung von Minuten auf einer Single-Core CPU erreicht werden können. Die Methode wurde in drei aktuellen Anwendungen verwendet, wobei mindestens zwei davon nach unserer Kenntnis im Moment mit keiner anderen numerischen Methode simuliert werden können: Zunächst präsentieren wir Simulationen von biologischen Zellen, die im Zuge eines RT-DC (Real-Time Deformability Cytometry) Experiments durch einen schmalen mikrofluidischen Kanal advektiert und dabei verformt werden. Danach zeigen wir die Ergebnisse erster Simulationen der uniaxialen Kompression biologischer Zellen zwischen zwei parallelen Platten im Zuge eines AFM Experiments. Schließlich präsentieren wir Resultate erster Simulationen von neuartigen mikroschwimmenden Schalen, welche lediglich durch äußere Einflüsse (wie z.B. Ultraschall), zum Schwimmen angeregt werden können.The dynamics of membranes, shells, and capsules in fluid flow has become an active research area in computational physics and computational biology. The small thickness of these elastic materials enables their efficient approximation as a hypersurface, which exhibits an elastic response to in-plane stretching and out-of-plane bending, possibly accompanied by a surface tension force. In this work, we present a novel arbitrary Lagrangian-Eulerian (ALE) method to simulate such elastic surfaces immersed in Navier-Stokes fluids. The method combines high accuracy with computational efficiency, since the grid is matched to the elastic surface and can therefore be resolved with relatively few grid points. The focus of this work is on axisymmetric shapes and flow conditions, which are present in a wide range of biophysical problems. Next to a general three-dimensional description, we formulate axisymmetric elastic surface forces and propose a discretization with surface finite-differences coupled to evolving finite elements. We further develop an implicit coupling strategy to reduce time step restrictions. Several numerical test cases show that accurate results can be achieved at computational times on the order of minutes on a single core CPU. Three state-of-the-art applications are demonstrated, where to our knowledge at least two of them cannot be simulated with any other numerical method so far. First, simulations of biological cells being advected through a microfluidic channel and therefore being deformed during an RT-DC (Real-Time Deformability Cytometry) experiment are presented. Then, the uniaxial compression of the cortex of a biological cell during an AFM experiment is investigated. Finally, we present the results of first simulations of the observed shape oscillations of novel microswimming shells which can be locomoted by exterior influences (e.g. ultrasound waves) only