Joint denoising and distortion correction of atomic scale scanning transmission electron microscopy images

Nowadays, modern electron microscopes deliver images at atomic scale. The precise atomic structure encodes information about material properties. Thus, an important ingredient in the image analysis is to locate the centers of the atoms shown in micrographs as precisely as possible. Here, we consider scanning transmission electron microscopy (STEM), which acquires data in a rastering pattern, pixel by pixel. Due to this rastering combined with the magnification to atomic scale, movements of the specimen even at the nanometer scale lead to random image distortions that make precise atom localization difficult. Given a series of STEM images, we derive a Bayesian method that jointly estimates the distortion in each image and reconstructs the underlying atomic grid of the material by fitting the atom bumps with suitable bump functions. The resulting highly non-convex minimization problems are solved numerically with a trust region approach. Well-posedness of the reconstruction method and the model behavior for faster and faster rastering are investigated using variational techniques. The performance of the method is finally evaluated on both synthetic and real experimental data

Combining Variational Autoencoders and Physical Bias for Improved Microscopy Data Analysis

Electron and scanning probe microscopy produce vast amounts of data in the form of images or hyperspectral data, such as EELS or 4D STEM, that contain information on a wide range of structural, physical, and chemical properties of materials. To extract valuable insights from these data, it is crucial to identify physically separate regions in the data, such as phases, ferroic variants, and boundaries between them. In order to derive an easily interpretable feature analysis, combining with well-defined boundaries in a principled and unsupervised manner, here we present a physics augmented machine learning method which combines the capability of Variational Autoencoders to disentangle factors of variability within the data and the physics driven loss function that seeks to minimize the total length of the discontinuities in images corresponding to latent representations. Our method is applied to various materials, including NiO-LSMO, BiFeO3, and graphene. The results demonstrate the effectiveness of our approach in extracting meaningful information from large volumes of imaging data. The fully notebook containing implementation of the code and analysis workflow is available at https://github.com/arpanbiswas52/PaperNotebooksComment: 20 pages, 7 figures in main text, 4 figures in Supp Ma

Review : Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy

Joint non-rigid image registration and reconstruction for quantitative atomic resolution scanning transmission electron microscopy

Aberration corrected scanning transmission electron microscopes (STEM) enable to determine local strain fields, composition and bonding states at atomic resolution. The precision to locate atomic columns is often obstructed by scan artifacts limiting the quantitative interpretation of STEM datasets. Here, a novel bias-corrected non-rigid registration approach is presented that compensates for fast and slow scan artifacts in STEM image series. The bias-correction is responsible for the correction of the slow scan artifacts and based on a explicit coupling of the deformations of the individual images in a series via a minimization of the average deformation. This allows to reduce fast scan noise in an image series and slow scan distortions simultaneously. The novel approach is tested on synthetic and experimental images and its implication on atomic resolution strain and elemental mapping is discussed

Iterative Phase Retrieval Algorithms for Scanning Transmission Electron Microscopy

Scanning transmission electron microscopy (STEM) has been extensively used for imaging complex materials down to atomic resolution. The most commonly employed STEM imaging modality of annular dark field produces easily-interpretable contrast, but is dose-inefficient and produces little to no contrast for light elements and weakly-scattering samples. An alternative is to use phase contrast STEM imaging, enabled by high speed detectors able to record full images of a diffracted STEM probe over a grid of scan positions. Phase contrast imaging in STEM is highly dose-efficient, able to measure the structure of beam-sensitive materials and even biological samples. Here, we comprehensively describe the theoretical background, algorithmic implementation details, and perform both simulated and experimental tests for three iterative phase retrieval STEM methods: focused-probe differential phase contrast, defocused-probe parallax imaging, and a generalized ptychographic gradient descent method implemented in two and three dimensions. We discuss the strengths and weaknesses of each of these approaches using a consistent framework to allow for easier comparison. This presentation of STEM phase retrieval methods will make these methods more approachable, reproducible and more readily adoptable for many classes of samples.Comment: 25 pages, 11 figures, 1 tabl

액상에 존재하는 개별 나노입자에 대한 3차원 원자구조 분석 방법론

학위논문(박사) -- 서울대학교대학원 : 공과대학 화학생물공학부, 2023. 2. 박정원.Precise three-dimensional (3D) atomic structure determination of individual nanocrystals is a prerequisite for understanding and predicting their physical properties, because the 3D atomic arrangements of materials determine the free energy landscape. We developed a Brownian one-particle reconstruction based on imaging of ensembles of colloidal nanocrystals using graphene liquid cell electron microscopy. Nanocrystals from the same synthesis batch display what are often presumed to be small but possibly important differences in size, lattice distortions, and defects, which can only be understood by structural characterization with high spatial 3D resolution. The structures of individual colloidal platinum nanocrystals are solved by developing atomic-resolution 3D liquid-cell electron microscopy to reveal critical intrinsic heterogeneity of ligand-protected platinum nanocrystals in solution, including structural degeneracies, lattice parameter deviations, internal defects, and strain. These differences in structure lead to substantial contributions to free energies, consequential enough that they must be considered in any discussion of fundamental nanocrystal properties or applications. We introduce computational methods required for successful atomic-resolution 3D reconstruction: (i) tracking of the individual particles throughout the time series, (ii) subtraction of the interfering background of the graphene liquid cell, (iii) identification and rejection of low-quality images, and (iv) tailored strategies for 2D/3D alignment and averaging that differ from those used in biological cryo–electron microscopy. Characterization of lattice symmetry is important because the symmetry is strongly correlated with physical properties of nanomaterials. We introduce direct and quantitative analysis of lattice symmetry by using 3D atomic coordinates obtained by liquid-phase TEM. We investigate symmetry of entire unit-cells composing individual platinum nanoparticles, revealing unique structural characteristics of sub-3 nm Pt nanoparticles. We here introduce a 3D atomic structure determination method for multi-element nanoparticle systems. The method, which is based on low-pass filtration and initial 3D model generation customized for different types of multi-element systems, enables reconstruction of high-resolution 3D Coulomb density maps for ordered and disordered multi-element systems and classification of the heteroatom type. Using high-resolution image datasets obtained from TEM simulations of PbSe, CdSe, and FePt nanoparticles that are structurally relaxed with first-principles calculations in the graphene liquid cell, we show that the types and positions of the constituent atoms are precisely determined with root mean square displacement (RMSD) values less than 24 pm. Our study suggests that it is possible to investigate the 3D atomic structures of synthesized multi-element nanoparticles in liquid phase.재료의 3D 원자 배열이 자유 에너지 환경을 결정한다는 점을 고려했을 때, 개별 나노결정의 정확한 3차원(3D) 원자 구조 분석은 물리적 특성을 이해하고 예측하기 위해 필수 불가결하다. 본 연구자는 그래핀 액체 세포 투과 전자 현미경을 사용하여 콜로이드 나노입자의 앙상블 이미징을 기반으로 하는 "브라운 단일 입자 재구성"을 개발했다. 동일한 합성 배치의 나노입자는 크기, 격자 왜곡 및 결함 등에서 종종 작지만 중요한 것으로 추정되는 것으로 간주되는 구조적 차이점이 있으며, 이는 3D 고해상도 구조 분석에 의해서만 이해할 수 있다. 구조적 퇴화, 격자 매개변수 편차, 내부 결함 및 변형을 포함한 개별 콜로이드 백금 나노입자의 구조적 특성은 원자 분해능 3D 액체 세포 전자 현미경을 개발하여 풀어낼 수 있다. 이러한 구조의 차이는 자유 에너지에 상당한 기여를 하므로 결과적으로 기본적인 나노입자 특성 또는 응용에 대한 논의에서 고려되어야 한다. 본 논문에서는 성공적인 원자 해상도 3D 재구성에 필요한 계산 방법론을 소개한다. 그 방법론에는 다음과 같은 알고리즘이 포함된다. (1) 시계열 이미지에서 개별 나노입 자를 추적하는 알고리즘, (2) 그래핀 액체 셀의 배경 노이즈를 제거하는 알고리즘, (3) 저해상도 이미지를 검출 및 제거하는 알고리즘, (4) 극저온 전자현미경을 이용한 바이오 입자의 재구성에 쓰이는 알고리즘과는 다른 나노입자만을 위해서 고안된 2차원/3차원 정렬 알고리즘. 격자 대칭성은 나노 물질의 물리적 특성과 강한 상관관계가 있기 때문에, 격자 대칭성 분석은 중요하다. 본 논문에서는 액상 투과 전자 현미경을 통해서 얻은 3차원 원자 좌표를 이용하여 격자 대칭을 직접적, 정량적으로 분석할 수 있는 방법론을 소개하고자 한다. 개별 백금 나노입자를 구성하는 전체 unit cell의 대칭성을 조사함으로써, 3 나노미터 이하의 백금 나노입자가 갖는 독특한 구조적 특징을 밝혀내였다. 본 논문에서는 다원소 나노입자 시스템을 위한 3차원 원자 구조 분석법을 소개하고자 한다. 제시된 low-pass filtering과 initial 3D modeling 방법은 다양한 유형의 다원소 시스템에 맞춰져 있으며, 이를 통해 ordered multi-element system과 disordered multi-element system에서 원자의 위치를 파악하고 원소의 종류를 구분할 수 있다. First-principles calculation을 통해 얻은 PbSe, CdSe, FePt 나노입자 구조로부터 그래핀 액체 셀 안에서의 TEM 시뮬레이션 이미지를 얻고, 이를 활용하여 구성 원자의 유형과 위치를 24 피코미터 미만의 오차로 정확도 높게 판별할 수 있음을 확인하였다. 우리의 연구는 액상에서 합성된 다원소 나노입자의 3차원 원자 구조를 조사하는 것이 가능함을 시사한다.Chapter 1. Introdution 1 1.1. Atomic structure property relationships in nanoparticles 1 1.2. Toward atomic structure characterization 2 1.3. Direct observation of 3D atomic structures of individual nanoparticles: Electron tomography and Brownian one-particle reconstruction 3 1.4. Purpose of Research 4 Chapter 2. 3D atomic structures of individual ligand-protected Pt nanoparticles in solution 7 2.1. Introduction 7 2.2. 3D reconstruction from electron microscopy images of Pt nanoparticles in liquid 8 2.2.1. Synthesis of Pt nanoparticles 8 2.2.2. Preparation of graphene liquid cells 9 2.2.3. Acquisition of TEM images 9 2.2.4. 3D reconstruction 10 2.2.5. Atomic position assignment 11 2.2.6. Validation 11 2.2.7. Atomic structure analysis 13 2.3. Atomic structural characteristics of Pt nanoparticles in liquid 16 2.2.1. Effect of surface ligands on the 3D atomic structures of Pt nanoparticles 16 2.3.2. Structural heterogeneity of Pt nanoparticles 18 2.3.3. Strain analysis of individual Pt nanoparticles from the 3D atomic maps 19 2.4. Conclusion 21 Chapter 3. SINGLE: Computational methods for atomic-resolution 3D reconstruction 57 3.1. Introduction 57 3.2. Results 58 3.2.1. Overview of 3D SINGLE 58 3.2.2. The SINGLE workflow 58 3.3. Conclusion 66 Chapter 4. 3-Dimensional scanning of unit cell symmetries in individual nanoparticles by using Brownian one-particle reconstruction 75 4.1. Introduction 75 4.2. Results 77 4.2.1. Quantitative symmetry analysis from 3D atomic coordinates 77 4.2.2. Direction of symmetry breakage 79 4.2.3. Structural heterogeneity 80 4.2.4. Relationship between symmetry and surface interactions 80 4.3. Conclusion 84 Chapter 5. Method for 3D atomic structure determination of multi-element nanoparticles with graphene liquid-cell TEM 102 5.1. Introduction 102 5.2. Results 104 5.2.1. Overview of multi-element nanoparticle 3D reconstruction 104 5.2.2. Principles for multi-element nanoparticle reconstruction 105 5.2.3. Demonstration using simulated TEM images 106 5.3. Conclusion 111 Bibliography 136 국 문 초 록 144박

Using blind image filtering for images from TEM microscopes

Předložená práce se zabývá problematikou slepé filtrace obrazů z transmisního elektronového mikroskopu. V úvodu práce je uveden popis transmisního elektronového mikroskopu. Navazující část popisuje mechanismy interakce elektronů se zkoumaným vzorkem a z toho vyplývající zobrazovací techniky elektronové mikroskopie. Poslední kapitola teoretické části práce zahrnuje popis vybraných metod slepé filtrace obrazu zejména s využitím dekompozice obrazu na charakteristické složky. Taktéž je zde uveden výčet metod pro zhodnocení úspěšnosti filtrace. V praktické části jsou popsány aplikované metody slepé filtrace obrazů a výsledky filtrování. Jednotlivé metody jsou mezi sebou porovnány. Získané výsledky a využitelnost aplikovaných metod jsou zhodnoceny v diskuzi.This work deals with the blind filtration of the images from the transmission electron microscope. At the beginning of this work there is a basic description of the transmission electron microscope. Following part describes the mechanisms of electron interactions with the observed specimen. Description of basic electron microscopy imaging techniques is included. The last chapter of the theoretical part includes the description of several chosen blind image filtration techniques, especially those using the decomposition of the image into characteristic components. It also contains a summary of methods for evaluation the filtration effectiveness. The practical part focuses on a description of applied blind filtering methods and brings the results of the filtration. Individual methods are compared. In conclusion, the obtained results and usability of the applied methods are discussed.

Single atom imaging with time-resolved electron microscopy

Developments in scanning transmission electron microscopy (STEM) have opened up new possibilities for time-resolved imaging at the atomic scale. However, rapid imaging of single atom dynamics brings with it a new set of challenges, particularly regarding noise and the interaction between the electron beam and the specimen. This thesis develops a set of analytical tools for capturing atomic motion and analyzing the dynamic behaviour of materials at the atomic scale. Machine learning is increasingly playing an important role in the analysis of electron microscopy data. In this light, new unsupervised learning tools are developed here for noise removal under low-dose imaging conditions and for identifying the motion of surface atoms. The scope for real-time processing and analysis is also explored, which is of rising importance as electron microscopy datasets grow in size and complexity. These advances in image processing and analysis are combined with computational modelling to uncover new chemical and physical insights into the motion of atoms adsorbed onto surfaces. Of particular interest are systems for heterogeneous catalysis, where the catalytic activity can depend intimately on the atomic environment. The study of Cu atoms on a graphene oxide support reveals that the atoms undergo anomalous diffusion as a result of spatial and energetic disorder present in the substrate. The investigation is extended to examine the structure and stability of small Cu clusters on graphene oxide, with atomistic modelling used to understand the significant role played by the substrate. Finally, the analytical methods are used to study the surface reconstruction of silicon alongside the electron beam-induced motion of adatoms on the surface. Taken together, these studies demonstrate the materials insights that can be obtained with time-resolved STEM imaging, and highlight the importance of combining state-ofthe- art imaging with computational analysis and atomistic modelling to quantitatively characterize the behaviour of materials with atomic resolution.The research leading to these results has received funding from the European Research Council under the European Union's Seventh Framework Programme (FP7/2007–2013)/ERC grant agreement 291522–3DIMAGE, as well as from the European Union Seventh Framework Programme under Grant Agreement 312483-ESTEEM2 (Integrated Infrastructure Initiative -I3)