Sliding to predict: vision-based beating heart motion estimation by modeling temporal interactions

Purpose: Technical advancements have been part of modern medical solutions as they promote better surgical alternatives that serve to the benefit of patients. Particularly with cardiovascular surgeries, robotic surgical systems enable surgeons to perform delicate procedures on a beating heart, avoiding the complications of cardiac arrest. This advantage comes with the price of having to deal with a dynamic target which presents technical challenges for the surgical system. In this work, we propose a solution for cardiac motion estimation. Methods: Our estimation approach uses a variational framework that guarantees preservation of the complex anatomy of the heart. An advantage of our approach is that it takes into account different disturbances, such as specular reflections and occlusion events. This is achieved by performing a preprocessing step that eliminates the specular highlights and a predicting step, based on a conditional restricted Boltzmann machine, that recovers missing information caused by partial occlusions. Results: We carried out exhaustive experimentations on two datasets, one from a phantom and the other from an in vivo procedure. The results show that our visual approach reaches an average minima in the order of magnitude of 10-7 while preserving the heart’s anatomical structure and providing stable values for the Jacobian determinant ranging from 0.917 to 1.015. We also show that our specular elimination approach reaches an accuracy of 99% compared to a ground truth. In terms of prediction, our approach compared favorably against two well-known predictors, NARX and EKF, giving the lowest average RMSE of 0.071. Conclusion: Our approach avoids the risks of using mechanical stabilizers and can also be effective for acquiring the motion of organs other than the heart, such as the lung or other deformable objects.Peer ReviewedPostprint (published version

Segmentation of Specular Highlights from Object Surfaces

A major hindrance to image segmentation tasks are the presence of specular highlights on object surfaces. Specular highlights appear on object surfaces where the specular component of reflection from illuminating light sources is so dominant that most detail of the object surface is obscured by a bright region of reflected light. Specular highlights are very common artifacts of most lighting environments and are not part of the intrinsic visible detail of an object surface. As a result, in addition to obscuring visible detail, specular highlight regions of an image can easily deceive image understanding algorithms into interpreting these regions as separate objects or regions on an object with high albedo. Recently, a couple of approaches to identifying specular highlight regions in images of object surfaces have produced some good results using color analysis. Unfortunately these methods work only for dielectric materials (e.g. plastic, rubber etc.) and require that the color of the object be different from the color of the light source. In this paper a technique is presented exploiting the polarization properties of reflected light to identify specular highlight regions. This technique works for both dielectric and metal surfaces regardless of the color of the illuminating light source, or the color detail on the object surface. In addition to separating out diffuse and specular components of reflection, the technique presented here also as a bonus can identify whet her certain image regions correspond to a dielectric or metal object surface. Extensive experimentation will be presented for a variety of dielectric and metal surfaces, both polished and rough. Experimentation with coated surfaces using the technique presented here have not yet been studied

Lightweight Face Relighting

In this paper we present a method to relight human faces in real time, using consumer-grade graphics cards even with limited 3D capabilities. We show how to render faces using a combination of a simple, hardware-accelerated parametric model simulating skin shading and a detail texture map, and provide robust procedures to estimate all the necessary parameters for a given face. Our model strikes a balance between the difficulty of realistic face rendering (given the very specific reflectance properties of skin) and the goal of real-time rendering with limited hardware capabilities. This is accomplished by automatically generating an optimal set of parameters for a simple rendering model. We offer a discussion of the issues in face rendering to discern the pros and cons of various rendering models and to generalize our approach to most of the current hardware constraints. We provide results demonstrating the usability of our approach and the improvements we introduce both in the performance and in the visual quality of the resulting faces

On-site surface reflectometry

The rapid development of Augmented Reality (AR) and Virtual Reality (VR) applications over the past years has created the need to quickly and accurately scan the real world to populate immersive, realistic virtual environments for the end user to enjoy. While geometry processing has already gone a long way towards that goal, with self-contained solutions commercially available for on-site acquisition of large scale 3D models, capturing the appearance of the materials that compose those models remains an open problem in general uncontrolled environments. The appearance of a material is indeed a complex function of its geometry, intrinsic physical properties and furthermore depends on the illumination conditions in which it is observed, thus traditionally limiting the scope of reflectometry to highly controlled lighting conditions in a laboratory setup. With the rapid development of digital photography, especially on mobile devices, a new trend in the appearance modelling community has emerged, that investigates novel acquisition methods and algorithms to relax the hard constraints imposed by laboratory-like setups, for easy use by digital artists. While arguably not as accurate, we demonstrate the ability of such self-contained methods to enable quick and easy solutions for on-site reflectometry, able to produce compelling, photo-realistic imagery. In particular, this dissertation investigates novel methods for on-site acquisition of surface reflectance based on off-the-shelf, commodity hardware. We successfully demonstrate how a mobile device can be utilised to capture high quality reflectance maps of spatially-varying planar surfaces in general indoor lighting conditions. We further present a novel methodology for the acquisition of highly detailed reflectance maps of permanent on-site, outdoor surfaces by exploiting polarisation from reflection under natural illumination. We demonstrate the versatility of the presented approaches by scanning various surfaces from the real world and show good qualitative and quantitative agreement with existing methods for appearance acquisition employing controlled or semi-controlled illumination setups.Open Acces

Low-energy ion beamline scattering apparatus for surface science investigations

We report on the design, construction, and performance of a high current (monolayers/s), mass-filtered ion beamline system for surface scattering studies using inert and reactive species at collision energies below 1500 eV. The system combines a high-density inductively coupled plasma ion source, high-voltage floating beam transport line with magnet mass-filter and neutral stripping, decelerator, and broad based detection capabilities (ions and neutrals in both mass and energy) for products leaving the target surface. The entire system was designed from the ground up to be a robust platform to study ion-surface interactions from a more global perspective, i.e., high fluxes (>100 µA/cm2) of a single ion species at low, tunable energy (50–1400±5 eV full width half maximum) can be delivered to a grounded target under ultrahigh vacuum conditions. The high current at low energy problem is solved using an accel-decel transport scheme where ions are created at the desired collision energy in the plasma source, extracted and accelerated to high transport energy (20 keV to fight space charge repulsion), and then decelerated back down to their original creation potential right before impacting the grounded target. Scattered species and those originating from the surface are directly analyzed in energy and mass using a triply pumped, hybrid detector composed of an electron impact ionizer, hemispherical electrostatic sector, and rf/dc quadrupole in series. With such a system, the collision kinematics, charge exchange, and chemistry occurring on the target surface can be separated by fully analyzing the scattered product flux. Key design aspects of the plasma source, beamline, and detection system are emphasized here to highlight how to work around physical limitations associated with high beam flux at low energy, pumping requirements, beam focusing, and scattered product analysis. Operational details of the beamline are discussed from the perspective of available beam current, mass resolution, projectile energy spread, and energy tunability. As well, performance of the overall system is demonstrated through three proof-of-concept examples: (1) elastic binary collisions at low energy, (2) core-level charge exchange reactions involving 20Ne+ with Mg/Al/Si/P targets, and (3) reactive scattering of CF2+/CF3+ off Si. These studies clearly demonstrate why low, tunable incident energy, as well as mass and energy filtering of products leaving the target surface is advantageous and often essential for studies of inelastic energy losses, hard-collision charge exchange, and chemical reactions that occur during ion-surface scattering

Examples of current radar technology and applications, chapter 5, part B

Basic principles and tradeoff considerations for SLAR are summarized. There are two fundamental types of SLAR sensors available to the remote sensing user: real aperture and synthetic aperture. The primary difference between the two types is that a synthetic aperture system is capable of significant improvements in target resolution but requires equally significant added complexity and cost. The advantages of real aperture SLAR include long range coverage, all-weather operation, in-flight processing and image viewing, and lower cost. The fundamental limitation of the real aperture approach is target resolution. Synthetic aperture processing is the most practical approach for remote sensing problems that require resolution higher than 30 to 40 m

Polarisation photometric stereo

© 2017 This paper concerns a novel approach to fuse two-source photometric stereo (PS) data with polarisation information for complete surface normal recovery for smooth or slightly rough surfaces. PS is a well-established method but is limited in application by its need for three or more well-spaced and known illumination sources and Lambertian reflectance. Polarisation methods are less studied but have shown promise for smooth surfaces under highly controlled capture conditions. However, such methods suffer from inherent ambiguities and the depolarising effects of surface roughness. The method presented in this paper goes some way to overcome these limitations by fusing the most reliable information from PS and polarisation. PS is used with only two sources to deduce a constrained mapping of the surface normal at each point onto a 2D plane. Phase information from polarisation is used to deduce a mapping onto a different plane. The paper then shows how the full surface normal can be obtained from the two mappings. The method is tested on a range of real-world images to demonstrate the advantages over standalone applications of PS or polarisation methods