Software for full-color 3D reconstruction of the biological tissues internal structure

A software for processing sets of full-color images of biological tissue histological sections is developed. We used histological sections obtained by the method of high-precision layer-by-layer grinding of frozen biological tissues. The software allows restoring the image of the tissue for an arbitrary cross-section of the tissue sample. Thus, our method is designed to create a full-color 3D reconstruction of the biological tissue structure. The resolution of 3D reconstruction is determined by the quality of the initial histological sections. The newly developed technology available to us provides a resolution of up to 5 - 10 {\mu}m in three dimensions.Comment: 11 pages, 8 figure

Finding Nano-\"Otzi: Semi-Supervised Volume Visualization for Cryo-Electron Tomography

Cryo-Electron Tomography (cryo-ET) is a new 3D imaging technique with unprecedented potential for resolving submicron structural detail. Existing volume visualization methods, however, cannot cope with its very low signal-to-noise ratio. In order to design more powerful transfer functions, we propose to leverage soft segmentation as an explicit component of visualization for noisy volumes. Our technical realization is based on semi-supervised learning where we combine the advantages of two segmentation algorithms. A first weak segmentation algorithm provides good results for propagating sparse user provided labels to other voxels in the same volume. This weak segmentation algorithm is used to generate dense pseudo labels. A second powerful deep-learning based segmentation algorithm can learn from these pseudo labels to generalize the segmentation to other unseen volumes, a task that the weak segmentation algorithm fails at completely. The proposed volume visualization uses the deep-learning based segmentation as a component for segmentation-aware transfer function design. Appropriate ramp parameters can be suggested automatically through histogram analysis. Finally, our visualization uses gradient-free ambient occlusion shading to further suppress visual presence of noise, and to give structural detail desired prominence. The cryo-ET data studied throughout our technical experiments is based on the highest-quality tilted series of intact SARS-CoV-2 virions. Our technique shows the high impact in target sciences for visual data analysis of very noisy volumes that cannot be visualized with existing techniques

Capturing Nucleation at 4D Atomic Resolution

Nucleation plays a critical role in many physical and biological phenomena ranging from crystallization, melting and evaporation to the formation of clouds and the initiation of neurodegenerative diseases. However, nucleation is a challenging process to study in experiments especially in the early stage when several atoms/molecules start to form a new phase from its parent phase. Here, we advance atomic electron tomography to study early stage nucleation at 4D atomic resolution. Using FePt nanoparticles as a model system, we reveal that early stage nuclei are irregularly shaped, each has a core of one to few atoms with the maximum order parameter, and the order parameter gradient points from the core to the boundary of the nucleus. We capture the structure and dynamics of the same nuclei undergoing growth, fluctuation, dissolution, merging and/or division, which are regulated by the order parameter distribution and its gradient. These experimental observations differ from classical nucleation theory (CNT) and to explain them we propose the order parameter gradient (OPG) model. We show the OPG model generalizes CNT and energetically favours diffuse interfaces for small nuclei and sharp interfaces for large nuclei. We further corroborate this model using molecular dynamics simulations of heterogeneous and homogeneous nucleation in liquid-solid phase transitions of Pt. We anticipate that the OPG model is applicable to different nucleation processes and our experimental method opens the door to study the structure and dynamics of materials with 4D atomic resolution.Comment: 42 pages, 5 figures, 12 supplementary figures and one supplementary tabl

Machine Learning on Neutron and X-Ray Scattering

Neutron and X-ray scattering represent two state-of-the-art materials characterization techniques that measure materials' structural and dynamical properties with high precision. These techniques play critical roles in understanding a wide variety of materials systems, from catalysis to polymers, nanomaterials to macromolecules, and energy materials to quantum materials. In recent years, neutron and X-ray scattering have received a significant boost due to the development and increased application of machine learning to materials problems. This article reviews the recent progress in applying machine learning techniques to augment various neutron and X-ray scattering techniques. We highlight the integration of machine learning methods into the typical workflow of scattering experiments. We focus on scattering problems that faced challenge with traditional methods but addressable using machine learning, such as leveraging the knowledge of simple materials to model more complicated systems, learning with limited data or incomplete labels, identifying meaningful spectra and materials' representations for learning tasks, mitigating spectral noise, and many others. We present an outlook on a few emerging roles machine learning may play in broad types of scattering and spectroscopic problems in the foreseeable future.Comment: 56 pages, 12 figures. Feedback most welcom

Algorithmic and infrastructural software development for cryo electron tomography

Many Cryo Electron Microscopy (cryoEM) software packages have accumulated significant technical debts over the years, resulting in overcomplicated codebases that are costly to maintain and that slow down development. In this thesis, we advocate for the development of open-source cryoEM core libraries as a solution to this debt and with the ultimate goal of improving the developer and user experience. First, a brief summary of cryoEM is presented, with an emphasis on projection algorithms and tomography. Second, the requirements of modern and future cryoEM image processing are discussed. Third, a new experimental cryoEM core library written in modern C++ is introduced. This library prioritises performance and code reusability, and is designed around a few core functions which offers an efficient model to manipulate multidimensional arrays at an index-wise and element-wise level. C++ template metaprogramming allowed us to develop modular and transparent compute backends, that provide great CPU and GPU performance, unified in an easy to use interface. Fourth, new projection algorithms will be described, notably a grid-driven approach to accurately insert and sample central slices in 3-dimensional (3d) Fourier space. A Fourier-based fused backward-forward projection, further improving the computational efficiency and accuracy of reprojections, will also be presented. Fifth, and as part of our efforts to test and showcase the library, we have started to implement a tilt series alignment package that gathers existing and new techniques into an automated pipeline. The current program first estimates the per-tilt translations and specimen stage rotation using a coarse alignment based on cosine stretching. It then fits the Thon rings of each tilt image as part of a global optimization to estimate the specimen inclination. Finally, we are using our Fourier-based fused reprojection to efficiently refine the per-tilt translations, and are starting to explore ways that would allow us to refine the per-tilt stage rotations

Lose The Views: Limited Angle CT Reconstruction via Implicit Sinogram Completion

Computed Tomography (CT) reconstruction is a fundamental component to a wide variety of applications ranging from security, to healthcare. The classical techniques require measuring projections, called sinograms, from a full 180$^\circ$ view of the object. This is impractical in a limited angle scenario, when the viewing angle is less than 180$^\circ$, which can occur due to different factors including restrictions on scanning time, limited flexibility of scanner rotation, etc. The sinograms obtained as a result, cause existing techniques to produce highly artifact-laden reconstructions. In this paper, we propose to address this problem through implicit sinogram completion, on a challenging real world dataset containing scans of common checked-in luggage. We propose a system, consisting of 1D and 2D convolutional neural networks, that operates on a limited angle sinogram to directly produce the best estimate of a reconstruction. Next, we use the x-ray transform on this reconstruction to obtain a "completed" sinogram, as if it came from a full 180$^\circ$ measurement. We feed this to standard analytical and iterative reconstruction techniques to obtain the final reconstruction. We show with extensive experimentation that this combined strategy outperforms many competitive baselines. We also propose a measure of confidence for the reconstruction that enables a practitioner to gauge the reliability of a prediction made by our network. We show that this measure is a strong indicator of quality as measured by the PSNR, while not requiring ground truth at test time. Finally, using a segmentation experiment, we show that our reconstruction preserves the 3D structure of objects effectively.Comment: Spotlight presentation at CVPR 201

Automated Segmentation of Large Image Datasets using Artificial Intelligence for Microstructure Characterisation, Damage Analysis and High-Throughput Modelling Input

Many properties of commonly used materials are driven by their microstructure, which can be influenced by the composition and manufacturing processes. To optimise future materials, understanding the microstructure is critically important. Here, we present two novel approaches based on artificial intelligence that allow the segmentation of the phases of a microstructure for which simple numerical approaches, such as thresholding, are not applicable: One is based on the nnU-Net neural network, and the other on generative adversarial networks (GAN). Using large panoramic scanning electron microscopy images of dual-phase steels as a case study, we demonstrate how both methods effectively segment intricate microstructural details, including martensite, ferrite, and damage sites, for subsequent analysis. Either method shows substantial generalizability across a range of image sizes and conditions, including heat-treated microstructures with different phase configurations. The nnU-Net excels in mapping large image areas. Conversely, the GAN-based method performs reliably on smaller images, providing greater step-by-step control and flexibility over the segmentation process. This study highlights the benefits of segmented microstructural data for various purposes, such as calculating phase fractions, modelling material behaviour through finite element simulation, and conducting geometrical analyses of damage sites and the local properties of their surrounding microstructure.Comment: 37 pages, 24 figure