LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference

We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately, thus generating better novel views. Without the depth information, regressing appropriate plane locations is a challenging problem. To solve this issue, we pre-partition the disparity space into bins and design a disparity sampler to regress local offsets for multiple planes in each bin. However, only using such a sampler makes the network not convergent; we further propose two optimizing strategies that combine with different disparity distributions of datasets and propose an occlusion-aware reprojection loss as a simple yet effective geometric supervision technique. We also introduce a self-attention mechanism to improve occlusion inference and present a Block-Sampling Self-Attention (BS-SA) module to address the problem of applying self-attention to large feature maps. We demonstrate the effectiveness of our approach and generate state-of-the-art results on different datasets. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 83.1%-84.7%. We also evaluate the performance on real-world images and demonstrate the benefits.Comment: Accepted by ICCV 202

Machine Learning-Based View Synthesis in Fourier Lightfield Microscopy

Current interest in Fourier lightfield microscopy is increasing, due to its ability to acquire 3D images of thick dynamic samples. This technique is based on simultaneously capturing, in a single shot, and with a monocular setup, a number of orthographic perspective views of 3D microscopic samples. An essential feature of Fourier lightfield microscopy is that the number of acquired views is low, due to the trade-off relationship existing between the number of views and their corresponding lateral resolution. Therefore, it is important to have a tool for the generation of a high number of synthesized view images, without compromising their lateral resolution. In this context we investigate here the use of a neural radiance field view synthesis method, originally developed for its use with macroscopic scenes acquired with a moving (or an array of static) digital camera(s), for its application to the images acquired with a Fourier lightfield microscope. The results obtained and presented in this paper are analyzed in terms of lateral resolution and of continuous and realistic parallax. We show that, in terms of these requirements, the proposed technique works efficiently in the case of the epi-illumination microscopy mode

Enhanced facial expression using oxygenation absorption of facial skin

Facial skin appearance is affected by physical and physiological state of the skin. The facial expression especially the skin appearances are in constant mutability and dynamically changed as human behave, talk and stress. The color of skin is considered to be one of the key indicators for these symptoms. The skin color resolution is highly determined by the scattering and absorption of light within the skin layers. The concentration of chromophores in melanin and hemoglobin oxygenation in the blood plays a pivotal role. An improvement work on prior model to create a realistic textured three-dimensional (3D) facial model for animation is proposed. This thesis considers both surface and subsurface scattering capable of simulating the interaction of light with the human skin. Furthermore, six parameters are used in this research which are the amount of oxygenation, de-oxygenation, hemoglobin, melanin, oil and blend factor for different types of melanin in the skin to generate a perfect match to specific skin types. The proposed model is associated with Blend Shape Interpolation and Facial Action Coding System to create five basic facial emotional expressions namely anger, happy, neutral, sad and fear. Meanwhile, the correlation between blood oxygenation in changing facial skin color for basic natural emotional expressions are measured using the Pulse Oximetry and 3D skin analyzer. The data from different subjects with male and female under different number of partially extreme facial expressions are fed in the model for simulation. The multi-pole method for layered materials is used to calculate the spectral diffusion profiles of two-layered skin which are further utilized to simulate the subsurface scattering of light within the skin. While the subsurface scattering is further combined with the Torrance-Sparrow Bidirectional Reflectance Distribution Function (BRDF) model to simulate the interaction of light with an oily layer at the skin surface. The result is validated by an evaluation procedure for measuring the accountability of a facial model via expressions and skin color of proposed model to the real human. The facial expressions evaluation is verified by calculating Euclidean distance between the facial markers of the real human and the avatar. The second assessment validates the skin color of facial expressions for the proposed avatar via the extraction of Histogram Color Features and Color Coherence Vector of each image with the real human and the previous work. The experimental result shows around 5.12 percent improvement compared to previous work. In achieving the realistic facial expression for virtual human based on facial skin color, texture and oxygenation of hemoglobin, the result demonstrates that the proposed model is beneficial to the development of virtual reality and game environment of computer aided graphics animation systems

From point cloud to textured model the Zamani laser scanning pipeline in heritage documentation

The paper describes the stages of the laser scanning pipeline from data acquisition to the final 3D computer model based on experiences gained during the ongoing creation of data for the African Cultural Heritage Sites and Landscapes database. The various processes are briefly discussed and challenges are highlighted which need to be addressed to develop the full potential of laser scanning. Experiences with fieldwork, scan registration, hole-filling, data cleaning, modelling and texturing are reported. The potential strengths and weaknesses of the emerging tool of “Structure from Motion” are briefly explored for their potential use in combination with laser scanning

From Point Cloud to Textured Model, the Zamani Laser Scanning Pipeline in Heritage Documentation

The paper describes the stages of the laser scanning pipeline from data acquisition to the final 3D computer model based on experiences gained during the ongoing creation of data for the African Cultural Heritage Sites and Landscapes database. The various processes are briefly discussed and challenges are highlighted which need to be addressed to develop the full potential of laser scanning. Experiences with fieldwork, scan registration, hole-filling, data cleaning, modelling and texturing are reported. The potential strengths and weaknesses of the emerging tool of “Structure from Motion” are briefly explored for their potential use in combination with laser scanning

Réalité Augmentée et Environnement Collaboratif : Un Tour d'Horizon

National audienceLa Réalité Augmentée (RA) est généralement définie comme une branche dérivée de la Réalité Virtuelle. D'une façon plus générale, le concept de réalité augmentée regroupe une approche multidisciplinaire visant un mélange entre réel et virtuel. La forte potentialité induite par cette connexion promet un cadre adéquat pour l'interaction 3D ou les applications collaboratives. On présente dans cet article un tour d'horizon des principaux travaux menés à ce jour dans le cadre de l'image et de la RA et plus particulièrement le cadre collaboratif

Multi-View Dynamic Shape Refinement Using Local Temporal Integration

International audienceWe consider 4D shape reconstructions in multi-view environments and investigate how to exploit temporal redundancy for precision refinement. In addition to being beneficial to many dynamic multi-view scenarios this also enables larger scenes where such increased precision can compensate for the reduced spatial resolution per image frame. With precision and scalability in mind, we propose a symmetric (non-causal) local time-window geometric integration scheme over temporal sequences, where shape reconstructions are refined framewise by warping local and reliable geometric regions of neighboring frames to them. This is in contrast to recent comparable approaches targeting a different context with more compact scenes and real-time applications. These usually use a single dense volumetric update space or geometric template, which they causally track and update globally frame by frame, with limitations in scalability for larger scenes and in topology and precision with a template based strategy. Our templateless and local approach is a first step towards temporal shape super-resolution. We show that it improves reconstruction accuracy by considering multiple frames. To this purpose, and in addition to real data examples, we introduce a multi-camera synthetic dataset that provides ground-truth data for mid-scale dynamic scenes

A Fast Geometric Multigrid Method for Curved Surfaces

We introduce a geometric multigrid method for solving linear systems arising from variational problems on surfaces in geometry processing, Gravo MG. Our scheme uses point clouds as a reduced representation of the levels of the multigrid hierarchy to achieve a fast hierarchy construction and to extend the applicability of the method from triangle meshes to other surface representations like point clouds, nonmanifold meshes, and polygonal meshes. To build the prolongation operators, we associate each point of the hierarchy to a triangle constructed from points in the next coarser level. We obtain well-shaped candidate triangles by computing graph Voronoi diagrams centered around the coarse points and determining neighboring Voronoi cells. Our selection of triangles ensures that the connections of each point to points at adjacent coarser and finer levels are balanced in the tangential directions. As a result, we obtain sparse prolongation matrices with three entries per row and fast convergence of the solver.Comment: Ruben Wiersma and Ahmad Nasikun contributed equally. To be published in SIGGRAPH 2023. 16 pages total (8 main, 5 supplement), 14 figure