State-of-The-Art and Applications of 3D Imaging Sensors in Industry, Cultural Heritage, Medicine, and Criminal Investigation

3D imaging sensors for the acquisition of three dimensional (3D) shapes have created, in recent years, a considerable degree of interest for a number of applications. The miniaturization and integration of the optical and electronic components used to build them have played a crucial role in the achievement of compactness, robustness and flexibility of the sensors. Today, several 3D sensors are available on the market, even in combination with other sensors in a “sensor fusion” approach. An importance equal to that of physical miniaturization has the portability of the measurements, via suitable interfaces, into software environments designed for their elaboration, e.g., CAD-CAM systems, virtual renders, and rapid prototyping tools. In this paper, following an overview of the state-of-art of 3D imaging sensors, a number of significant examples of their use are presented, with particular reference to industry, heritage, medicine, and criminal investigation applications

Physically Interacting With Four Dimensions

Thesis (Ph.D.) - Indiana University, Computer Sciences, 2009People have long been fascinated with understanding the fourth dimension. While making pictures of 4D objects by projecting them to 3D can help reveal basic geometric features, 3D graphics images by themselves are of limited value. For example, just as 2D shadows of 3D curves may have lines crossing one another in the shadow, 3D graphics projections of smooth 4D topological surfaces can be interrupted where one surface intersects another. The research presented here creates physically realistic models for simple interactions with objects and materials in a virtual 4D world. We provide methods for the construction, multimodal exploration, and interactive manipulation of a wide variety of 4D objects. One basic achievement of this research is to exploit the free motion of a computer-based haptic probe to support a continuous motion that follows the \emph{local continuity\/} of a 4D surface, allowing collision-free exploration in the 3D projection. In 3D, this interactive probe follows the full local continuity of the surface as though we were in fact \emph{physically touching\/} the actual static 4D object. Our next contribution is to support dynamic 4D objects that can move, deform, and collide with other objects as well as with themselves. By combining graphics, haptics, and collision-sensing physical modeling, we can thus enhance our 4D visualization experience. Since we cannot actually place interaction devices in 4D, we develop fluid methods for interacting with a 4D object in its 3D shadow image using adapted reduced-dimension 3D tools for manipulating objects embedded in 4D. By physically modeling the correct properties of 4D surfaces, their bending forces, and their collisions in the 3D interactive or haptic controller interface, we can support full-featured physical exploration of 4D mathematical objects in a manner that is otherwise far beyond the real-world experience accessible to human beings

Interactive illumination and navigation control in an image-based environment.

Fu Chi-wing.Thesis (M.Phil.)--Chinese University of Hong Kong, 1999.Includes bibliographical references (leaves 141-149).Abstract --- p.iAcknowledgments --- p.iiiChapter 1 --- Introduction --- p.1Chapter 1.1 --- Introduction to Image-based Rendering --- p.1Chapter 1.2 --- Scene Complexity Independent Property --- p.2Chapter 1.3 --- Application of this Research Work --- p.3Chapter 1.4 --- Organization of this Thesis --- p.4Chapter 2 --- Illumination Control --- p.7Chapter 2.1 --- Introduction --- p.7Chapter 2.2 --- Apparent BRDF of Pixel --- p.8Chapter 2.3 --- Sampling Illumination Information --- p.11Chapter 2.4 --- Re-rendering --- p.13Chapter 2.4.1 --- Light Direction --- p.15Chapter 2.4.2 --- Light Intensity --- p.15Chapter 2.4.3 --- Multiple Light Sources --- p.15Chapter 2.4.4 --- Type of Light Sources --- p.18Chapter 2.5 --- Data Compression --- p.22Chapter 2.5.1 --- Intra-pixel coherence --- p.22Chapter 2.5.2 --- Inter-pixel coherence --- p.22Chapter 2.6 --- Implementation and Result --- p.22Chapter 2.6.1 --- An Interactive Viewer --- p.22Chapter 2.6.2 --- Lazy Re-rendering --- p.24Chapter 2.7 --- Conclusion --- p.24Chapter 3 --- Navigation Control - Triangle-based Warping Rule --- p.29Chapter 3.1 --- Introduction to Navigation Control --- p.29Chapter 3.2 --- Related Works --- p.30Chapter 3.3 --- Epipolar Geometry (Perspective Projection Manifold) --- p.31Chapter 3.4 --- Drawing Order for Pixel-Sized Entities --- p.35Chapter 3.5 --- Triangle-based Image Warping --- p.36Chapter 3.5.1 --- Image-based Triangulation --- p.36Chapter 3.5.2 --- Image-based Visibility Sorting --- p.40Chapter 3.5.3 --- Topological Sorting --- p.44Chapter 3.6 --- Results --- p.46Chapter 3.7 --- Conclusion --- p.48Chapter 4 --- Panoramic Projection Manifold --- p.52Chapter 4.1 --- Epipolar Geometry (Spherical Projection Manifold) --- p.53Chapter 4.2 --- Image Triangulation --- p.56Chapter 4.2.1 --- Optical Flow --- p.56Chapter 4.2.2 --- Image Gradient and Potential Function --- p.57Chapter 4.2.3 --- Triangulation --- p.58Chapter 4.3 --- Image-based Visibility Sorting --- p.58Chapter 4.3.1 --- Mapping Criteria --- p.58Chapter 4.3.2 --- Ordering of Two Triangles --- p.59Chapter 4.3.3 --- Graph Construction and Topological Sort --- p.63Chapter 4.4 --- Results --- p.63Chapter 4.5 --- Conclusion --- p.65Chapter 5 --- Panoramic-based Navigation using Real Photos --- p.69Chapter 5.1 --- Introduction --- p.69Chapter 5.2 --- System Overview --- p.71Chapter 5.3 --- Correspondence Matching --- p.72Chapter 5.3.1 --- Basic Model of Epipolar Geometry --- p.72Chapter 5.3.2 --- Epipolar Geometry between two Neighbor Panoramic Nodes --- p.73Chapter 5.3.3 --- Line and Patch Correspondence Matching --- p.74Chapter 5.4 --- Triangle-based Warping --- p.75Chapter 5.4.1 --- Why Warp Triangle --- p.75Chapter 5.4.2 --- Patch and Layer Construction --- p.76Chapter 5.4.3 --- Triangulation and Mesh Subdivision --- p.76Chapter 5.4.4 --- Layered Triangle-based Warping --- p.77Chapter 5.5 --- Implementation --- p.78Chapter 5.5.1 --- Image Sampler and Panoramic Stitcher --- p.78Chapter 5.5.2 --- Interactive Correspondence Matcher and Triangulation --- p.79Chapter 5.5.3 --- Basic Panoramic Viewer --- p.79Chapter 5.5.4 --- Formulating Drag Vector and vn --- p.80Chapter 5.5.5 --- Controlling Walkthrough Parameter --- p.82Chapter 5.5.6 --- Interactive Web-based Panoramic Viewer --- p.83Chapter 5.6 --- Results --- p.84Chapter 5.7 --- Conclusion and Possible Enhancements --- p.84Chapter 6 --- Compositing Warped Images for Object-based Viewing --- p.89Chapter 6.1 --- Modeling Object-based Viewing --- p.89Chapter 6.2 --- Triangulation and Convex Hull Criteria --- p.92Chapter 6.3 --- Warping Multiple Views --- p.94Chapter 6.3.1 --- Method I --- p.95Chapter 6.3.2 --- Method II --- p.95Chapter 6.3.3 --- Method III --- p.95Chapter 6.4 --- Results --- p.97Chapter 6.5 --- Conclusion --- p.100Chapter 7 --- Complete Rendering Pipeline --- p.107Chapter 7.1 --- Reviews on Illumination and Navigation --- p.107Chapter 7.1.1 --- Illumination Rendering Pipeline --- p.107Chapter 7.1.2 --- Navigation Rendering Pipeline --- p.108Chapter 7.2 --- Analysis of the Two Rendering Pipelines --- p.109Chapter 7.2.1 --- Combination on the Architectural Level --- p.109Chapter 7.2.2 --- Ensuring Physical Correctness --- p.111Chapter 7.3 --- Generalizing Apparent BRDF --- p.112Chapter 7.3.1 --- Difficulties to Encode BRDF with Spherical Harmonics --- p.112Chapter 7.3.2 --- Generalize Apparent BRDF --- p.112Chapter 7.3.3 --- Related works for Encoding the generalized apparent BRDF --- p.113Chapter 7.4 --- Conclusion --- p.116Chapter 8 --- Conclusion --- p.117Chapter A --- Spherical Harmonics --- p.120Chapter B --- It is Rare for Cycles to Exist in the Drawing Order Graph --- p.123Chapter B.1 --- Theorem 3 --- p.123Chapter B.2 --- Inside and Outside-directed Triangles in a Triangular Cycle --- p.125Chapter B.2.1 --- Assumption --- p.126Chapter B.2.2 --- Inside-directed and Outside-directed triangles --- p.126Chapter B.3 --- Four Possible Cases to Form a Cycle --- p.127Chapter B.3.1 --- Case(l) Triangular Fan --- p.128Chapter B.3.2 --- Case(2) Two Outside-directed Triangles --- p.129Chapter B.3.3 --- Case(3) Three Outside-directed Triangles --- p.130Chapter B.3.4 --- Case(4) More than Three Outside-directed Triangles --- p.131Chapter B.4 --- Experiment --- p.132Chapter C --- Deriving the Epipolar Line Formula on Cylindrical Projection Manifold --- p.133Chapter C.1 --- Notations --- p.133Chapter C.2 --- General Formula --- p.134Chapter C.3 --- Simplify the General Formula to a Sine Curve --- p.137Chapter C.4 --- Show that the Epipolar Line is a Sine Curve Segment --- p.139Chapter D --- Publications Related to this Research Work --- p.141Bibliography --- p.14

Utilisation de l'Apparence pour le Rendu et l'édition efficaces de scènes capturées

Computer graphics strives to render synthetic images identical to real photographs. Multiple rendering algorithms have been developed for the better part of the last half-century. Traditional algorithms use 3D assets manually generated by artists to render a scene. While the initial scenes were quite simple, the field has developed complex representations of geometry, material and lighting: the three basic components of a 3D scene. Generating such complex assets is hard and requires significant time and skills by professional 3D artists. In addition to asset generation, the rendering algorithms themselves involve complex simulation techniques to solve for global light transport in a scene which costs more time.As the ease of capturing photographs improved, Image-based Rendering (IBR) emerged as an alternative to traditional rendering. Using captured images as input became much faster than generating traditional scene assets. Initial IBR algorithms focused on creating a scene model using the input images to interpolate or warp them and enable free-viewpoint navigation of captured scenes. With time the scene models became more complex and using a geometric proxy computed from the input images became an integral part of IBR. Today using a mesh reconstructed using Structure-from-Motion (SfM) and Multi-view Stereo (MVS) techniques is widely used in IBR even though they introduce significant artifacts due to noisy reconstruction.In this thesis we first propose a novel image-based rendering algorithm, which focuses on rendering a captured scene with good quality at interactive frame rates}. We study different artifacts from previous IBR algorithms and propose an algorithm which builds upon previous work to remove such artifacts. The algorithm utilizes surface appearance in order to treat view-dependent regions differently than diffuse regions. Our Hybrid-IBR algorithm performs favorably against classical and modern IBR approaches for a wide variety of scenes in terms of quality and/or speed.While IBR provides solutions to render a scene, editing them is hard. Editing scenes require estimating a scene's geometry, material appearance and illumination. As our second contribution \textbf{we explicitly estimate \emph{scene-scale} material parameters from a set of captured photographs to enable scene editing}. While commercial photogrammetry solutions recover diffuse texture to aid 3D artists in generating material assets manually, we aim to \emph{automatically} create material texture atlases from captured images of a scene. We take advantage of the visual cues provided by the multi-view observations. Feeding it to a Convolutional Neural Network (CNN) we obtain material maps for each view. Using the predicted maps we create multi-view consistent material texture atlases by aggregating the information in texture space. Using our automatically generated material texture atlases we demonstrate relighting and object insertion in real scenes.Learning-based tasks require large amounts of data with variety to learn the task efficiently. Using synthetic datasets to train is the norm but using traditional rendering to render large datasets is time consuming providing limited variability. We propose \textbf{a new neural rendering-based approach that learns a neural scene representation with variability and use it to generate large amounts of data at a significantly faster rate on the fly}. We demonstrate the advantage of using neural rendering as compared to traditional rendering in terms of speed of generating dataset as well as learning auxiliary tasks given the same computational budget.L’informatique graphique a pour but de rendre des images de synthèse semblables à des photographies. Plusieurs algorithmes de rendu ont été développés au cours du dernier demi-siècle, principalement pour restituer des scènes à base d'éléments 3D créés par des artistes. Alors que les scènes initiales étaient assez simples, des représentations plus complexes de la géométrie, des matériaux et de l'éclairage ont été développés. Créer des scènes aussi complexes nécessite beaucoup de travail et de compétences de la part d'artistes 3D professionnels. Au même temps, les algorithmes de rendu impliquent des techniques de simulation complexes coûteuses en temps, pour résoudre le transport global de la lumière dans une scène.Avec la popularité grandissante de la photo numérique, le rendu basé image (IBR) a émergé comme une alternative au rendu traditionnel. Avec cette approche, l'utilisation de photos comme données d'entrée est devenue beaucoup plus rapide que la génération de scènes classiques. Les algorithmes IBR se sont d’abord concentrés sur la restitution de scènes pour en permettre une exploration libre. Au fil du temps, les modèles de scène sont devenus plus complexes et l'utilisation d'un proxy géométrique inféré à partir d’images est devenue la norme. Aujourd'hui, l'utilisation d'un maillage reconstruit à l'aide des techniques Structure-from-Motion (SfM) et Multi-view Stereo (MVS) est courante en IBR, bien que cette utilisation introduit des artefacts importants. Nous proposons d'abord un nouvel algorithme de rendu basé image, qui se concentre sur le rendu de qualité et en temps interactif d'une scène capturée}. Nous étudions différentes faiblesses des travaux précédents et proposons un algorithme qui s'appuie sur ces travaux pour obtenir de meilleurs résultats. Notre algorithme se base sur l'apparence de la surface pour traiter les régions dont l'apparence dépend de l'angle de vue différemment des régions diffuses. Hybrid-IBR obtient des résultats favorables par rapport aux approches concurrentes pour une grande variété de scènes en termes de qualité et/ou de vitesse.Bien que l'IBR soit une bonne solution de rendu, l'édition de celle-ci est difficile sans une décomposition en différents éléments : la géométrie, l'apparence des matériaux et l'éclairage de la scène. Pour notre deuxième contribution, \textbf{nous estimons explicitement les paramètres de matériaux à \emph{l'échelle de la scène} à partir d'un ensemble de photographies, pour permettre l'édition de la scène}. Alors que les solutions de photogrammétrie commerciales calculent la texture diffuse pour assister la création manuelle de matériaux, nous visons à créer \emph{automatiquement} des atlas de texture de matériaux à partir d'un ensemble d'images d'une scène. Nous nous appuyons sur les informations fournis par ces images et les transmettons à un réseau neuronal convolutif pour obtenir des cartes de matériaux pour chaque vue. En utilisant toutes ces prédictions, nous créons des atlas de texture de matériau cohérents pour toutes les vues en agrégeant les informations dans l'espace texture. Nous démontrons l'utilisation de notre atlas de texture de matériaux généré automatiquement pour rendre des scènes réelles avec un changement d’illumination et avec des objets virtuels insérés.L'apprentissage profond nécessite de grandes quantités de données variées. L'utilisation de données synthétiques est courante, mais l'utilisation du rendu traditionnel pour créer ces données prend du temps et offre une variabilité limitée. Nous proposons \textbf{une nouvelle approche basée sur le rendu neuronal qui apprend une représentation de scène neuronale avec paramètres variables, et l'utilise pour générer au vol de grandes quantités de données à un rythme beaucoup plus rapide}. Nous démontrons l'avantage d'utiliser le rendu neuronal par rapport au rendu traditionnel en termes de budget de temps, ainsi que pour l'apprentissage de tâches auxiliaires avec le même budget de calcul

Synchronized Illumination Modulation for Digital Video Compositing

Informationsaustausch ist eines der Grundbedürfnisse der Menschen. Während früher dazu Wandmalereien,Handschrift, Buchdruck und Malerei eingesetzt wurden, begann man später, Bildfolgen zu erstellen, die als sogenanntes ”Daumenkino” den Eindruck einer Animation vermitteln. Diese wurden schnell durch den Einsatz rotierender Bildscheiben, auf denen mit Hilfe von Schlitzblenden, Spiegeln oder Optiken eine Animation sichtbar wurde, automatisiert – mit sogenannten Phenakistiskopen,Zoetropen oder Praxinoskopen. Mit der Erfindung der Fotografie begannen in der zweiten Hälfte des 19. Jahrhunderts die ersten Wissenschaftler wie Eadweard Muybridge, Etienne-Jules Marey und Ottomar Anschütz, Serienbildaufnahmen zu erstellen und diese dann in schneller Abfolge, als Film, abzuspielen. Mit dem Beginn der Filmproduktion wurden auch die ersten Versuche unternommen, mit Hilfe dieser neuen Technik spezielle visuelle Effekte zu generieren, um damit die Immersion der Bewegtbildproduktionen weiter zu erhöhen. Während diese Effekte in der analogen Phase der Filmproduktion bis in die achtziger Jahre des 20.Jahrhunderts recht beschränkt und sehr aufwendig mit einem enormen manuellen Arbeitsaufwand erzeugt werden mussten, gewannen sie mit der sich rapide beschleunigenden Entwicklung der Halbleitertechnologie und der daraus resultierenden vereinfachten digitalen Bearbeitung immer mehr an Bedeutung. Die enormen Möglichkeiten, die mit der verlustlosen Nachbearbeitung in Kombination mit fotorealistischen, dreidimensionalen Renderings entstanden, führten dazu, dass nahezu alle heute produzierten Filme eine Vielfalt an digitalen Videokompositionseffekten beinhalten. ...Besides home entertainment and business presentations, video projectors are powerful tools for modulating images spatially as well as temporally. The re-evolving need for stereoscopic displays increases the demand for low-latency projectors and recent advances in LED technology also offer high modulation frequencies. Combining such high-frequency illumination modules with synchronized, fast cameras, makes it possible to develop specialized high-speed illumination systems for visual effects production. In this thesis we present different systems for using spatially as well as temporally modulated illumination in combination with a synchronized camera to simplify the requirements of standard digital video composition techniques for film and television productions and to offer new possibilities for visual effects generation. After an overview of the basic terminology and a summary of related methods, we discuss and give examples of how modulated light can be applied to a scene recording context to enable a variety of effects which cannot be realized using standard methods, such as virtual studio technology or chroma keying. We propose using high-frequency, synchronized illumination which, in addition to providing illumination, is modulated in terms of intensity and wavelength to encode technical information for visual effects generation. This is carried out in such a way that the technical components do not influence the final composite and are also not visible to observers on the film set. Using this approach we present a real-time flash keying system for the generation of perspectively correct augmented composites by projecting imperceptible markers for optical camera tracking. Furthermore, we present a system which enables the generation of various digital video compositing effects outside of completely controlled studio environments, such as virtual studios. A third temporal keying system is presented that aims to overcome the constraints of traditional chroma keying in terms of color spill and color dependency. ..

Digital Techniques for Documenting and Preserving Cultural Heritage

In this unique collection the authors present a wide range of interdisciplinary methods to study, document, and conserve material cultural heritage. The methods used serve as exemplars of best practice with a wide variety of cultural heritage objects having been recorded, examined, and visualised. The objects range in date, scale, materials, and state of preservation and so pose different research questions and challenges for digitization, conservation, and ontological representation of knowledge. Heritage science and specialist digital technologies are presented in a way approachable to non-scientists, while a separate technical section provides details of methods and techniques, alongside examples of notable applications of spatial and spectral documentation of material cultural heritage, with selected literature and identification of future research. This book is an outcome of interdisciplinary research and debates conducted by the participants of the COST Action TD1201, Colour and Space in Cultural Heritage, 2012–16 and is an Open Access publication available under a CC BY-NC-ND licence.https://scholarworks.wmich.edu/mip_arc_cdh/1000/thumbnail.jp

FULL 3D RECONSTRUCTION OF DYNAMIC NON-RIGID SCENES: ACQUISITION AND ENHANCEMENT

Recent advances in commodity depth or 3D sensing technologies have enabled us to move closer to the goal of accurately sensing and modeling the 3D representations of complex dynamic scenes. Indeed, in domains such as virtual reality, security, surveillance and e-health, there is now a greater demand for aff ordable and flexible vision systems which are capable of acquiring high quality 3D reconstructions. Available commodity RGB-D cameras, though easily accessible, have limited fi eld-of-view, and acquire noisy and low-resolution measurements which restricts their direct usage in building such vision systems. This thesis targets these limitations and builds approaches around commodity 3D sensing technologies to acquire noise-free and feature preserving full 3D reconstructions of dynamic scenes containing, static or moving, rigid or non-rigid objects. A mono-view system based on a single RGB-D camera is incapable of acquiring full 360 degrees 3D reconstruction of a dynamic scene instantaneously. For this purpose, a multi-view system composed of several RGB-D cameras covering the whole scene is used. In the first part of this thesis, the domain of correctly aligning the information acquired from RGB-D cameras in a multi-view system to provide full and textured 3D reconstructions of dynamic scenes, instantaneously, is explored. This is achieved by solving the extrinsic calibration problem. This thesis proposes an extrinsic calibration framework which uses the 2D photometric and 3D geometric information, acquired with RGB-D cameras, according to their relative (in)accuracies, a ffected by the presence of noise, in a single weighted bi-objective optimization. An iterative scheme is also proposed, which estimates the parameters of noise model aff ecting both 2D and 3D measurements, and solves the extrinsic calibration problem simultaneously. Results show improvement in calibration accuracy as compared to state-of-art methods. In the second part of this thesis, the domain of enhancement of noisy and low-resolution 3D data acquired with commodity RGB-D cameras in both mono-view and multi-view systems is explored. This thesis extends the state-of-art in mono-view template-free recursive 3D data enhancement which targets dynamic scenes containing rigid-objects, and thus requires tracking only the global motions of those objects for view-dependent surface representation and fi ltering. This thesis proposes to target dynamic scenes containing non-rigid objects which introduces the complex requirements of tracking relatively large local motions and maintaining data organization for view-dependent surface representation. The proposed method is shown to be e ffective in handling non-rigid objects of changing topologies. Building upon the previous work, this thesis overcomes the requirement of data organization by proposing an approach based on view-independent surface representation. View-independence decreases the complexity of the proposed algorithm and allows it the flexibility to process and enhance noisy data, acquired with multiple cameras in a multi-view system, simultaneously. Moreover, qualitative and quantitative experimental analysis shows this method to be more accurate in removing noise to produce enhanced 3D reconstructions of non-rigid objects. Although, extending this method to a multi-view system would allow for obtaining instantaneous enhanced full 360 degrees 3D reconstructions of non-rigid objects, it still lacks the ability to explicitly handle low-resolution data. Therefore, this thesis proposes a novel recursive dynamic multi-frame 3D super-resolution algorithm together with a novel 3D bilateral total variation regularization to filter out the noise, recover details and enhance the resolution of data acquired from commodity cameras in a multi-view system. Results show that this method is able to build accurate, smooth and feature preserving full 360 degrees 3D reconstructions of the dynamic scenes containing non-rigid objects