SIFT Flow: Dense Correspondence across Scenes and its Applications

While image alignment has been studied in different areas of computer vision for decades, aligning images depicting different scenes remains a challenging problem. Analogous to optical flow where an image is aligned to its temporally adjacent frame, we propose SIFT flow, a method to align an image to its nearest neighbors in a large image corpus containing a variety of scenes. The SIFT flow algorithm consists of matching densely sampled, pixel-wise SIFT features between two images, while preserving spatial discontinuities. The SIFT features allow robust matching across different scene/object appearances, whereas the discontinuity-preserving spatial model allows matching of objects located at different parts of the scene. Experiments show that the proposed approach robustly aligns complex scene pairs containing significant spatial differences. Based on SIFT flow, we propose an alignment-based large database framework for image analysis and synthesis, where image information is transferred from the nearest neighbors to a query image according to the dense scene correspondence. This framework is demonstrated through concrete applications, such as motion field prediction from a single image, motion synthesis via object transfer, satellite image registration and face recognition

Evaluation of process-structure-property relationships of carbon nanotube forests using simulation and deep learning

This work is aimed to explore process-structure-property relationships of carbon nanotube (CNT) forests. CNTs have superior mechanical, electrical and thermal properties that make them suitable for many applications. Yet, due to lack of manufacturing control, there is a huge performance gap between promising properties of individual CNTs and CNT forest properties that hinders their adoption into potential industrial applications. In this research, computational modelling, in-situ electron microscopy for CNT synthesis, and data-driven and high-throughput deep convolutional neural networks are employed to not only accelerate implementing CNTs in various applications but also to establish a framework to make validated predictive models that can be easily extended to achieve application-tailored synthesis of any materials. A time-resolved and physics-based finite-element simulation tool is modelled in MATLAB to investigate synthesis of CNT forests, specially to study the CNT-CNT interactions and generated mechanical forces and their role in ensemble structure and properties. A companion numerical model with similar construct is then employed to examine forest mechanical properties in compression. In addition, in-situ experiments are carried out inside Environmental Scanning Electron Microscope (ESEM) to nucleate and synthesize CNTs. Findings may primarily be used to expand the forest growth and self-assembly knowledge and to validate the assumptions of simulation package. Also, SEM images can be used as feed database to construct a deep learning model to grow CNTs by design. The chemical vapor deposition parameter space of CNT synthesis is so vast that it is not possible to investigate all conceivable combinations in terms of time and costs. Hence, simulated CNT forest morphology images are used to train machine learning and learning algorithms that are able to predict CNT synthesis conditions based on desired properties. Exceptionally high prediction accuracies of R2 > 0.94 is achieved for buckling load and stiffness, as well as accuracies of > 0.91 for the classification task. This high classification accuracy promotes discovering the CNT forest synthesis-structure relationships so that their promising performance can be adopted in real world applications. We foresee this work as a meaningful step towards creating an unsupervised simulation using machine learning techniques that can seek out the desired CNT forest synthesis parameters to achieve desired property sets for diverse applications.Includes bibliographical reference

CT diagnosis of early stroke : the initial approach to the new CAD tool based on multiscale estimation of ischemia

Background: Computer aided diagnosis (CAD) becomes one of the most important diagnostic tools for urgent states in cerebral stroke and other life-threatening conditions where time plays a crucial role. Routine CT is still diagnostically insufficient in hyperacute stage of stroke that is in the therapeutic window for thrombolytic therapy. Authors present computer assistant of early ischemic stroke diagnosis that supports the radiologic interpretations. A new semantic-visualization system of ischemic symptoms applied to noncontrast, routine CT examination was based on multiscale image processing and diagnostic content estimation. Material/Methods: Evaluation of 95 sets of examinations in patients admitted to a hospital with symptoms suggesting stroke was undertaken by four radiologists from two medical centers unaware of the final clinical findings. All of the consecutive cases were considered as having no CT direct signs of hyperacute ischemia. At the first test stage only the CTs performed at the admission were evaluated independently by radiologists. Next, the same early scans were evaluated again with additional use of multiscale computer-assistant of stroke (MulCAS). Computerized suggestion with increased sensitivity to the subtle image manifestations of cerebral ischemia was constructed as additional view representing estimated diagnostic content with enhanced stroke symptoms synchronized to routine CT data preview. Follow-up CT examinations and clinical features confirmed or excluded the diagnosis of stroke constituting 'gold standard' to verify stroke detection performance. Results: Higher AUC (area under curve) values were found for MulCAS -aided radiological diagnosis for all readers and the differences were statistically significant for random readers-random cases parametric and non-parametric DBM MRMC analysis. Sensitivity and specificity of acute stroke detection for the readers was increased by 30% and 4%, respectively. Conclusions: Routine CT completed with proposed method of computer assisted diagnosis provided noticeable better diagnosis efficiency of acute stroke according to the rates and opinions of all test readers. Further research includes fully automatic detection of hypodense regions to complete assisted indications and formulate the suggestions of stroke cases more objectively. Planned prospective studies will let evaluate more accurately the impact of this CAD tool on diagnosis and further treatment in patients suffered from stroke. It is necessary to determine whether this method is possible to be applied widely

Human Performance Modeling and Rendering via Neural Animated Mesh

We have recently seen tremendous progress in the neural advances for photo-real human modeling and rendering. However, it's still challenging to integrate them into an existing mesh-based pipeline for downstream applications. In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos. Our core intuition is to bridge the traditional animated mesh workflow with a new class of highly efficient neural techniques. We first introduce a neural surface reconstructor for high-quality surface generation in minutes. It marries the implicit volumetric rendering of the truncated signed distance field (TSDF) with multi-resolution hash encoding. We further propose a hybrid neural tracker to generate animated meshes, which combines explicit non-rigid tracking with implicit dynamic deformation in a self-supervised framework. The former provides the coarse warping back into the canonical space, while the latter implicit one further predicts the displacements using the 4D hash encoding as in our reconstructor. Then, we discuss the rendering schemes using the obtained animated meshes, ranging from dynamic texturing to lumigraph rendering under various bandwidth settings. To strike an intricate balance between quality and bandwidth, we propose a hierarchical solution by first rendering 6 virtual views covering the performer and then conducting occlusion-aware neural texture blending. We demonstrate the efficacy of our approach in a variety of mesh-based applications and photo-realistic free-view experiences on various platforms, i.e., inserting virtual human performances into real environments through mobile AR or immersively watching talent shows with VR headsets.Comment: 18 pages, 17 figure