Alignment of cryo-EM movies of individual particles by optimization of image translations

Direct detector device (DDD) cameras have revolutionized single particle electron cryomicroscopy (cryo-EM). In addition to an improved camera detective quantum efficiency, acquisition of DDD movies allows for correction of movement of the specimen, due both to instabilities in the microscope specimen stage and electron beam-induced movement. Unlike specimen stage drift, beam-induced movement is not always homogeneous within an image. Local correlation in the trajectories of nearby particles suggests that beam-induced motion is due to deformation of the ice layer. Algorithms have already been described that can correct movement for large regions of frames and for > 1 MDa protein particles. Another algorithm allows individual < 1 MDa protein particle trajectories to be estimated, but requires rolling averages to be calculated from frames and fits linear trajectories for particles. Here we describe an algorithm that allows for individual < 1 MDa particle images to be aligned without frame averaging or linear trajectories. The algorithm maximizes the overall correlation of the shifted frames with the sum of the shifted frames. The optimum in this single objective function is found efficiently by making use of analytically calculated derivatives of the function. To smooth estimates of particle trajectories, rapid changes in particle positions between frames are penalized in the objective function and weighted averaging of nearby trajectories ensures local correlation in trajectories. This individual particle motion correction, in combination with weighting of Fourier components to account for increasing radiation damage in later frames, can be used to improve 3-D maps from single particle cryo-EM.Comment: 11 pages, 4 figure

Study of Computational Image Matching Techniques: Improving Our View of Biomedical Image Data

Image matching techniques are proven to be necessary in various fields of science and engineering, with many new methods and applications introduced over the years. In this PhD thesis, several computational image matching methods are introduced and investigated for improving the analysis of various biomedical image data. These improvements include the use of matching techniques for enhancing visualization of cross-sectional imaging modalities such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), denoising of retinal Optical Coherence Tomography (OCT), and high quality 3D reconstruction of surfaces from Scanning Electron Microscope (SEM) images. This work greatly improves the process of data interpretation of image data with far reaching consequences for basic sciences research. The thesis starts with a general notion of the problem of image matching followed by an overview of the topics covered in the thesis. This is followed by introduction and investigation of several applications of image matching/registration in biomdecial image processing: a) registration-based slice interpolation, b) fast mesh-based deformable image registration and c) use of simultaneous rigid registration and Robust Principal Component Analysis (RPCA) for speckle noise reduction of retinal OCT images. Moving towards a different notion of image matching/correspondence, the problem of view synthesis and 3D reconstruction, with a focus on 3D reconstruction of microscopic samples from 2D images captured by SEM, is considered next. Starting from sparse feature-based matching techniques, an extensive analysis is provided for using several well-known feature detector/descriptor techniques, namely ORB, BRIEF, SURF and SIFT, for the problem of multi-view 3D reconstruction. This chapter contains qualitative and quantitative comparisons in order to reveal the shortcomings of the sparse feature-based techniques. This is followed by introduction of a novel framework using sparse-dense matching/correspondence for high quality 3D reconstruction of SEM images. As will be shown, the proposed framework results in better reconstructions when compared with state-of-the-art sparse-feature based techniques. Even though the proposed framework produces satisfactory results, there is room for improvements. These improvements become more necessary when dealing with higher complexity microscopic samples imaged by SEM as well as in cases with large displacements between corresponding points in micrographs. Therefore, based on the proposed framework, a new approach is proposed for high quality 3D reconstruction of microscopic samples. While in case of having simpler microscopic samples the performance of the two proposed techniques are comparable, the new technique results in more truthful reconstruction of highly complex samples. The thesis is concluded with an overview of the thesis and also pointers regarding future directions of the research using both multi-view and photometric techniques for 3D reconstruction of SEM images

3D SEM Surface Reconstruction from Multi-View Images

The scanning electron microscope (SEM), a promising imaging equipment has been used to determine the surface properties such as compositions or geometries of specimens by achieving increased magnification, contrast, and resolution. SEM micro-graphs, however, remain two-dimensional (2D). The knowledge and information about their three-dimensional (3D) surface structures are critical in many real-world applications. Having 3D surfaces from SEM images provides true anatomic shapes of micro-scale samples which allow for quantitative measurements and informative visualization of the systems being investigated. A novel multi-view approach for reconstruction of SEM images is demonstrated in this research project. This thesis focuses on the 3D SEM surface reconstruction from multi-view images. We investigate an approach to reconstruction of 3D surfaces from stereo SEM image pairs and then discuss how 3D point clouds may be registered to generate more complete 3D shapes from multi-views of the microscopic specimen. Then we introduce a method that uses an algorithm called KAZE, which reconstructs 3D surfaces from multiple views of objects. Then Numerous results are presented to show the effectiveness of the presented approaches

Non-contact strain determination of cell traction effects

Irreversible tissue damage leading to organ failure is a common health problem in today's world. Regenerating these damaged tissues with the help of scaffolds is the solution offered by tissue engineering. In cases where the extra-cellular matrix (ECM) is to be replaced by an artificial substrate (scaffold) or matrix, cellular traction forces (CTF) are exerted by the cells on the scaffold surface. An ideal scaffold should exhibit mechanical characteristics similar to those of the ECM it is intended to replace. In other words, the capacity of a scaffold to withstand deformation should be comparable to that of a natural ECM. And with knowledge of those forces and deformations the properties of the scaffolds may be inferred. Digital Image Correlation (DIC), a non-contact image analysis technique enables us to measure point to point deformation of the scaffold by comparing a sequence of images captured during the process of scaffold deformation. This review discusses the methodology involved and implementation of DIC to measure displacements and strain.Irreversible tissue damage leading to organ failure is a common health problem in today's world. Regenerating these damaged tissues with the help of scaffolds is the solution offered by tissue engineering. In cases where the extra-cellular matrix (ECM) is to be replaced by an artificial substrate (scaffold) or matrix, cellular traction forces (CTF) are exerted by the cells on the scaffold surface. An ideal scaffold should exhibit mechanical characteristics similar to those of the ECM it is intended to replace. In other words, the capacity of a scaffold to withstand deformation should be comparable to that of a natural ECM. And with knowledge of those forces and deformations the properties of the scaffolds may be inferred. Digital Image Correlation (DIC), a non-contact image analysis technique enables us to measure point to point deformation of the scaffold by comparing a sequence of images captured during the process of scaffold deformation. This review discusses the methodology involved and implementation of DIC to measure displacements and strain

3D SEM Surface Reconstruction: An Optimized, Adaptive, and Intelligent Approach

Structural analysis of microscopic objects is a longstanding topic in several scientific disciplines, including biological, mechanical, and material sciences. The scanning electron microscope (SEM), as a promising imaging equipment has been around to determine the surface properties (e.g., compositions or geometries) of specimens by achieving increased magnification, contrast, and resolution greater than one nanometer. Whereas SEM micrographs still remain two-dimensional (2D), many research and educational questions truly require knowledge and information about their three-dimensional (3D) surface structures. Having 3D surfaces from SEM images would provide true anatomic shapes of micro samples which would allow for quantitative measurements and informative visualization of the systems being investigated. In this research project, we novel design and develop an optimized, adaptive, and intelligent multi-view approach named 3DSEM++ for 3D surface reconstruction of SEM images, making a 3D SEM dataset publicly and freely available to the research community. The work is expected to stimulate more interest and draw attention from the computer vision and multimedia communities to the fast-growing SEM application area

Micro-object pose estimation with sim-to-real transfer learning using small dataset

International audience<span style="color: rgb(34, 34, 34); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Oxygen-Sans, Ubuntu, Cantarell, &quot;Helvetica Neue&quot;, sans-serif; font-size: 18px;"&gtThree-dimensional (3D) pose estimation of micro/nano-objects isessential for the implementation of automatic manipulation inmicro/nano-robotic systems. However, out-of-plane pose estimationof a micro/nano-object is challenging, since the images aretypically obtained in 2D using a scanning electron microscope (SEM)or an optical microscope (OM). Traditional deep learning basedmethods require the collection of a large amount of labeled datafor model training to estimate the 3D pose of an object from amonocular image. Here we present a sim-to-real learning-to-matchapproach for 3D pose estimation of micro/nano-objects. Instead ofcollecting large training datasets, simulated data is generated toenlarge the limited experimental data obtained in practice, whilethe domain gap between the generated and experimental data isminimized via image translation based on a generative adversarialnetwork (GAN) model. A learning-to-match approach is used to mapthe generated data and the experimental data to a low-dimensionalspace with the same data distribution for different pose labels,which ensures effective feature embedding. Combining the labeleddata obtained from experiments and simulations, a new trainingdataset is constructed for robust pose estimation. The proposedmethod is validated with images from both SEM and OM, facilitatingthe development of closed-loop control of micro/nano-objects withcomplex shapes in micro/nano-robotic systems.</span&g