Non-line-of-sight 3D imaging with a single-pixel camera

Real time, high resolution 3D reconstruction of scenes hidden from the direct field of view is a challenging field of research with applications in real-life situations related e.g. to surveillance, self-driving cars and rescue missions. Most current techniques recover the 3D structure of a non-lineof-sight (NLOS) static scene by detecting the return signal from the hidden object on a scattering observation area. Here, we demonstrate the full colour retrieval of the 3D shape of a hidden scene by coupling back-projection imaging algorithms with the high-resolution time-of-flight information provided by a single-pixel camera. By using a high effciency Single-Photon Avalanche Diode (SPAD) detector, this technique provides the advantage of imaging with no mechanical scanning parts, with acquisition times down to sub-seconds.Comment: 6 pages, 4 figure

Development of a real-time high-resolution 3D ultrasound imaging system

In this work a real-time high-resolution 3D ultrasound imaging system is developed, allowing the 3D acquisition and imaging of high-resolution ultrasound data for biomedical applications. The system uses ultrasound transducers in the ranges of 30 to 100 MHz, and allows full access of the radiofrequency (RF) ultrasound data for the 3D image reconstruction. This work includes the development of two graphical user interfaces in C++ to interact with a high-resolution ultrasound system and to image the high-resolution ultrasound data. In addition, a methodology is described and implemented in the system for 3D ultrasound reconstruction and visualization. The development of these GUIs allows easy 3D high-resolution ultrasound acquisition and imaging to any user with basic knowledge in ultrasound and with only minimal and faster training in the system and the GUIs. This capability opens the system to any researcher or person interested in performing experimentations with high-resolution ultrasound.;The high-resolution 3D ultrasound imaging system is tested to assess atherosclerosis using different mouse models. To assess atherosclerosis, a series of in vitro studies are performed to 3D scan vessels of mouse aortas and carotids vessels with atherosclerosis, as well as mouse hearts with atherosclerosis. The apolipoprotein E-knockout (APOE) and the apolipoprotein E-A1 adenosine receptor double knockout (DKO) mice model with their wild type control (C57) are used. Three-dimensional reconstructions were rendered showing good matches with the samples morphology. In addition, 3D sections of the data are reconstructed showing atherosclerotic plaque in the samples. The 3D ultrasound reconstruction allows for us to analyze a sample from outside and inside by rotating around any angle and cropping non-relevant data, allowing us to observe shape and appearance of the 3D structures. Finally, after reconstructing and analyzing the 3D ultrasound images, a 3D quantitative assessment of atherosclerotic plaque is performed. After analyzing the samples, the plaque lesions of DKO mouse model exhibits smaller areas than those of the APOE mouse model. Additionally, the C57 mouse model is clean of any atherosclerosis. These findings are in agreement with a previous study of our group for these mouse models

A New Real Time Shape Acquisition with a Laser Scanner: First Test Results

Abstract The first results of a new method for real-time shape acquisition with a laser scanner are presented. The new method is essentially based on the use of a laser beam and a web-cam. A digital filter parameters identification was studied for the laser line detection in the image. After this, a model for the reconstruction in real-time of the laser line in the space was developed. The firsttest rig was just conceived to validate the method; hence, no high resolution cameras were adopted. Nevertheless, the tests have showed encouraging results. Tests were made on both plane and non-plane surfaces. First of all, it was confirmed that it is possible to calibrate the intrinsic parameters of the video system, the position of the image plane and the laser plane in a given frame, all in the same time. Moreover the surface shapes were recognized and recorded with an appreciable accuracy. The tests also showed that the proposed method can be used for robotic applications, such as robotic kinematic calibration and 3D surfaces recognition and recording. For this last purpose, the test rig is fitted on a robot arm that permits to the scanner device to ‘observe’ the 3D object from different and known positions

3D shape instantiation for intra-operative navigation from a single 2D projection

Unlike traditional open surgery where surgeons can see the operation area clearly, in robot-assisted Minimally Invasive Surgery (MIS), a surgeon’s view of the region of interest is usually limited. Currently, 2D images from fluoroscopy, Magnetic Resonance Imaging (MRI), endoscopy or ultrasound are used for intra-operative guidance as real-time 3D volumetric acquisition is not always possible due to the acquisition speed or exposure constraints. 3D reconstruction, however, is key to navigation in complex in vivo geometries and can help resolve this issue. Novel 3D shape instantiation schemes are developed in this thesis, which can reconstruct the high-resolution 3D shape of a target from limited 2D views, especially a single 2D projection or slice. To achieve a complete and automatic 3D shape instantiation pipeline, segmentation schemes based on deep learning are also investigated. These include normalization schemes for training U-Nets and network architecture design of Atrous Convolutional Neural Networks (ACNNs). For U-Net normalization, four popular normalization methods are reviewed, then Instance-Layer Normalization (ILN) is proposed. It uses a sigmoid function to linearly weight the feature map after instance normalization and layer normalization, and cascades group normalization after the weighted feature map. Detailed validation results potentially demonstrate the practical advantages of the proposed ILN for effective and robust segmentation of different anatomies. For network architecture design in training Deep Convolutional Neural Networks (DCNNs), the newly proposed ACNN is compared to traditional U-Net where max-pooling and deconvolutional layers are essential. Only convolutional layers are used in the proposed ACNN with different atrous rates and it has been shown that the method is able to provide a fully-covered receptive field with a minimum number of atrous convolutional layers. ACNN enhances the robustness and generalizability of the analysis scheme by cascading multiple atrous blocks. Validation results have shown the proposed method achieves comparable results to the U-Net in terms of medical image segmentation, whilst reducing the trainable parameters, thus improving the convergence and real-time instantiation speed. For 3D shape instantiation of soft and deforming organs during MIS, Sparse Principle Component Analysis (SPCA) has been used to analyse a 3D Statistical Shape Model (SSM) and to determine the most informative scan plane. Synchronized 2D images are then scanned at the most informative scan plane and are expressed in a 2D SSM. Kernel Partial Least Square Regression (KPLSR) has been applied to learn the relationship between the 2D and 3D SSM. It has been shown that the KPLSR-learned model developed in this thesis is able to predict the intra-operative 3D target shape from a single 2D projection or slice, thus permitting real-time 3D navigation. Validation results have shown the intrinsic accuracy achieved and the potential clinical value of the technique. The proposed 3D shape instantiation scheme is further applied to intra-operative stent graft deployment for the robot-assisted treatment of aortic aneurysms. Mathematical modelling is first used to simulate the stent graft characteristics. This is then followed by the Robust Perspective-n-Point (RPnP) method to instantiate the 3D pose of fiducial markers of the graft. Here, Equally-weighted Focal U-Net is proposed with a cross-entropy and an additional focal loss function. Detailed validation has been performed on patient-specific stent grafts with an accuracy between 1-3mm. Finally, the relative merits and potential pitfalls of all the methods developed in this thesis are discussed, followed by potential future research directions and additional challenges that need to be tackled.Open Acces

Real-Time analysis and visualization for single-molecule based super-resolution microscopy

Accurate multidimensional localization of isolated fluorescent emitters is a time consuming process in single-molecule based super-resolution microscopy. We demonstrate a functional method for real-time reconstruction with automatic feedback control, without compromising the localization accuracy. Compatible with high frame rates of EM-CCD cameras, it relies on a wavelet segmentation algorithm, together with a mix of CPU/GPU implementation. A combination with Gaussian fitting allows direct access to 3D localization. Automatic feedback control ensures optimal molecule density throughout the acquisition process. With this method, we significantly improve the efficiency and feasibility of localization-based super-resolution microscopy

Computational fluid dynamics modeling and in situ physics-based monitoring of aerosol jet printing toward functional assurance of additively-manufactured, flexible and hybrid electronics

Aerosol jet printing (AJP)—a direct-write, additive manufacturing technique—has emerged as the process of choice particularly for the fabrication of flexible and hybrid electronics. AJP has paved the way for high-resolution device fabrication with high placement accuracy, edge definition, and adhesion. In addition, AJP accommodates a broad range of ink viscosity, and allows for printing on non-planer surfaces. Despite the unique advantages and host of strategic applications, AJP is a highly unstable and complex process, prone to gradual drifts in machine behavior and deposited material. Hence, real-time monitoring and control of AJP process is a burgeoning need. In pursuit of this goal, the objectives of the work are, as follows: (i) In situ image acquisition from the traces/lines of printed electronic devices right after deposition. To realize this objective, the AJP experimental setup was instrumented with a high-resolution charge-coupled device (CCD) camera, mounted on a variable-magnification lens (in addition to the standard imaging system, already installed on the AJ printer). (ii) In situ image processing and quantification of the trace morphology. In this regard, several customized image processing algorithms were devised to quantify/extract various aspects of the trace morphology from online images. In addition, based on the concept of shape-from-shading (SfS), several other algorithms were introduced, allowing for not only reconstruction of the 3D profile of the AJ-printed electronic traces, but also quantification of 3D morphology traits, such as thickness, cross-sectional area, and surface roughness, among others. (iii) Development of a supervised multiple-input, single-output (MISO) machine learning model—based on sparse representation for classification (SRC)—with the aim to estimate the device functional properties (e.g., resistance) in near real-time with an accuracy of ≥ 90%. (iv) Forwarding a computational fluid dynamics (CFD) model to explain the underlying aerodynamic phenomena behind aerosol transport and deposition in AJP process, observed experimentally. Overall, this doctoral dissertation paves the way for: (i) implementation of physics-based real-time monitoring and control of AJP process toward conformal material deposition and device fabrication; and (ii) optimal design of direct-write components, such as nozzles, deposition heads, virtual impactors, atomizers, etc

An approach for real world data modelling with the 3D terrestrial laser scanner for built environment

Capturing and modelling 3D information of the built environment is a big challenge. A number of techniques and technologies are now in use. These include EDM, GPS, and photogrammetric application, remote sensing and traditional building surveying applications. However, use of these technologies cannot be practical and efficient in regard to time, cost and accuracy. Furthermore, a multi disciplinary knowledge base, created from the studies and research about the regeneration aspects is fundamental: historical, architectural, archeologically, environmental, social, economic, etc. In order to have an adequate diagnosis of regeneration, it is necessary to describe buildings and surroundings by means of documentation and plans. However, at this point in time the foregoing is considerably far removed from the real situation, since more often than not it is extremely difficult to obtain full documentation and cartography, of an acceptable quality, since the material, constructive pathologies and systems are often insufficient or deficient (flat that simply reflects levels, isolated photographs,..). Sometimes the information in reality exists, but this fact is not known, or it is not easily accessible, leading to the unnecessary duplication of efforts and resources. In this paper, we discussed 3D laser scanning technology, which can acquire high density point data in an accurate, fast way. Besides, the scanner can digitize all the 3D information concerned with a real world object such as buildings, trees and terrain down to millimetre detail Therefore, it can provide benefits for refurbishment process in regeneration in the Built Environment and it can be the potential solution to overcome the challenges above. The paper introduce an approach for scanning buildings, processing the point cloud raw data, and a modelling approach for CAD extraction and building objects classification by a pattern matching approach in IFC (Industry Foundation Classes) format. The approach presented in this paper from an undertaken research can lead to parametric design and Building Information Modelling (BIM) for existing structures. Two case studies are introduced to demonstrate the use of laser scanner technology in the Built Environment. These case studies are the Jactin House Building in East Manchester and the Peel building in the campus of University Salford. Through these case studies, while use of laser scanners are explained, the integration of it with various technologies and systems are also explored for professionals in Built Environmen

3D scanning of cultural heritage with consumer depth cameras

Three dimensional reconstruction of cultural heritage objects is an expensive and time-consuming process. Recent consumer real-time depth acquisition devices, like Microsoft Kinect, allow very fast and simple acquisition of 3D views. However 3D scanning with such devices is a challenging task due to the limited accuracy and reliability of the acquired data. This paper introduces a 3D reconstruction pipeline suited to use consumer depth cameras as hand-held scanners for cultural heritage objects. Several new contributions have been made to achieve this result. They include an ad-hoc filtering scheme that exploits the model of the error on the acquired data and a novel algorithm for the extraction of salient points exploiting both depth and color data. Then the salient points are used within a modified version of the ICP algorithm that exploits both geometry and color distances to precisely align the views even when geometry information is not sufficient to constrain the registration. The proposed method, although applicable to generic scenes, has been tuned to the acquisition of sculptures and in this connection its performance is rather interesting as the experimental results indicate