Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions

Machine Learning Approaches to Human Body Shape Analysis

Soft biometrics, biomedical sciences, and many other fields of study pay particular attention to the study of the geometric description of the human body, and its variations. Although multiple contributions, the interest is particularly high given the non-rigid nature of the human body, capable of assuming different poses, and numerous shapes due to variable body composition. Unfortunately, a well-known costly requirement in data-driven machine learning, and particularly in the human-based analysis, is the availability of data, in the form of geometric information (body measurements) with related vision information (natural images, 3D mesh, etc.). We introduce a computer graphics framework able to generate thousands of synthetic human body meshes, representing a population of individuals with stratified information: gender, Body Fat Percentage (BFP), anthropometric measurements, and pose. This contribution permits an extensive analysis of different bodies in different poses, avoiding the demanding, and expensive acquisition process. We design a virtual environment able to take advantage of the generated bodies, to infer the body surface area (BSA) from a single view. The framework permits to simulate the acquisition process of newly introduced RGB-D devices disentangling different noise components (sensor noise, optical distortion, body part occlusions). Common geometric descriptors in soft biometric, as well as in biomedical sciences, are based on body measurements. Unfortunately, as we prove, these descriptors are not pose invariant, constraining the usability in controlled scenarios. We introduce a differential geometry approach assuming body pose variations as isometric transformations of the body surface, and body composition changes covariant to the body surface area. This setting permits the use of the Laplace-Beltrami operator on the 2D body manifold, describing the body with a compact, efficient, and pose invariant representation. We design a neural network architecture able to infer important body semantics from spectral descriptors, closing the gap between abstract spectral features, and traditional measurement-based indices. Studying the manifold of body shapes, we propose an innovative generative adversarial model able to learn the body shapes. The method permits to generate new bodies with unseen geometries as a walk on the latent space, constituting a significant advantage over traditional generative methods

Robust Nonrigid Registration by Convex Optimization

We present an approach to nonrigid registration of 3D surfaces. We cast isometric embedding as MRF opti-mization and apply efficient global optimization algorithms based on linear programming relaxations. The Markov ran-dom field perspective suggests a natural connection with robust statistics and motivates robust forms of the intrinsic distortion functional. Our approach outperforms a large body of prior work by a significant margin, increasing reg-istration precision on real data by a factor of 3. 1

Data-Driven Shape Analysis and Processing

Data-driven methods play an increasingly important role in discovering geometric, structural, and semantic relationships between 3D shapes in collections, and applying this analysis to support intelligent modeling, editing, and visualization of geometric data. In contrast to traditional approaches, a key feature of data-driven approaches is that they aggregate information from a collection of shapes to improve the analysis and processing of individual shapes. In addition, they are able to learn models that reason about properties and relationships of shapes without relying on hard-coded rules or explicitly programmed instructions. We provide an overview of the main concepts and components of these techniques, and discuss their application to shape classification, segmentation, matching, reconstruction, modeling and exploration, as well as scene analysis and synthesis, through reviewing the literature and relating the existing works with both qualitative and numerical comparisons. We conclude our report with ideas that can inspire future research in data-driven shape analysis and processing.Comment: 10 pages, 19 figure

Construction Scene Point Cloud Acquisition, Object Finding and Clutter Removal in Real Time

Within industrial construction, piping can constitute up to 50% of the cost of a typical project. It has been shown that across the activities involved in pipe fabrication, pipe fitting has the highest impact on the critical path. The pipe fitter is responsible for interpreting the isometric drawing and then performing the tack welds on piping components so that the assembly complies with the design. Three main problems in doing this task are identified as: (1) reading and interpreting the isometric drawing is challenging and error prone for spatially complicated assemblies, (2) in assemblies with tight allowable tolerance, a number of iterations will take place to fit the pipes with compliance to the design. These iterations (rework) will remain unrecorded in the production process, and (3) no continuous measurement tool exists to let the fitter check his/her work in progress against the design information and acceptance specifications. Addressing these problems could substantially improve pipe fitters’ productivity. The objective of this research is to develop a software package integrating a threefold solution to simplify complex tasks involved in pipe fabrication: (1) making design information easier to understand, with the use of a tablet, 3D imaging device and an application software, (2) providing visual feedback on the correctness of fabrication between the design intent and the as-built state, and (3) providing frequent feedback on fabrication using a step-by-step assembly and control framework. The step-by-step framework will reduce the number of required iterations for the pipe fitter. A number of challenges were encountered in order to provide a framework to make real time, visual and frequent feedback. For frequent and visual feedback, a real time 3D data acquisition tool with an acceptable level of accuracy should be adopted. This is due to the speed of fabrication in an industrial facility. The second challenge is to find the object of interest in real time, once a point cloud is acquired, and finally, once the object is found, to optimally remove points that are considered as clutter to improve the visual feedback for the pipe fitters. To address the requirement for a reliable and real time acquisition tool, Chapter 3 explores the capabilities and limitations of low cost range cameras. A commercially available 3D imaging tool was utilized to measure its performance for real time point cloud acquisition. The device was used to inspect two pipe spools altered in size. The acquired point clouds were super-imposed on the BIM (Building Information Model) model of the pipe spools to measure the accuracy of the device. Chapter 4 adapts and examines a real time and automatic object finding algorithm to measure its performance with respect to construction challenges. Then, a K-Nearest Neighbor (KNN) algorithm was employed to classify points as being clutter or corresponding to the object of interest. Chapter 5 investigates the effect of the threshold value “K” in the K-Nearest Neighbor algorithm and optimizing its value for an improved visual feedback. As a result of the work described in this thesis, along with the work of two other master students and a co-op student, a software package was designed and developed. The software package takes advantage of the investigated real time point cloud acquisition device. While the object finding algorithm proved to be effective, a 3-point matching algorithm was used, as it was more intuitive for the users and took less time. The KNN algorithm was utilized to remove clutter points to provide more accurate visual feedback more accurate to the workers