Mihut AM

The research project covers the area of 3D artificial object compositing into photographs especially in the absence of user-annotated scene geometry. General matting and compositing is a broad area that spans across numerous subdomains of study such as disparity mapping from stereo, dense point cloud reconstruction, image analysis for material inference, and inverse rendering. Taking this into consideration, our work focuses on accurately compositing objects which play an important role in the initial scene relighting (i.e. compositing in between existing geometry while casting accurate soft shadows)

Newcastle University E-Prints

Towards a General Solution for Compositing 3D Artefacts in 2D Photographs  Ana M. MihuțNewcastle UniversityThe research project covers the area of 3D ar�ficial object composi�ng into photographs especially in the absence of user-annotated scene geometry. Ma�ng and composi�ng is a broad area that spans across numerous subdomains of study such as disparity mapping from stereo, dense point  cloud reconstruc�on, image analysis for material inference, and inverse rendering. Taking this into considera�on, our work focuses on accurately composi�ng objects which play an important role in the ini�al scene religh�ng (i.e. composi�ng in between exis�ng geometry while cas�ng accurate so� shadows). Problem Description & MotivationRecent Related WorkMethodologyEvaluation & ResultsReferencesNovel approaches in this area focus on inser�ng synthe�c objects while concomitantly minimizing the requirements for user annota�on.  However, these techniques do not ac-count for the material proper�es of the scene in a manner where composi�ng in be-tween exis�ng geometry could yield a realis�c result. As a consequence, most approach-es use a general caus�c shadow effect which does not accurately represent light source and occluder interac�on, especially where more than one kind of synthe�c object is added to the original scene.Based on factors that define the scene and arefacts’ proper�es , the related work can be split up into the following categories: Scene acquisi�on & representa�on: • Karsch et al. [2014, 2015] propose an automa�c approach based on a trained             classifier. The technique works for non-specular 3D meshes and for indoor scenes. • Zhang et al. [2016] rely on a hybrid technique using both RGB-D hard ware and user      annota�on in order to refurbish indoor rooms.  Illumina�on modelling & material representa�on: • Yeung et al. [2011] developed a technique of copy and pas�ng of translucent               objects in photographs. • Barron and Malik [2015] created a model for scene property decomposi�on from           shading.ana.mihut@newcastle.ac.ukD.O.I: 10.1145/3193025.3193060 h�ps://research.ncl.ac.uk/game/Karsch K.,Sunkavalli K.,Hadap S.,Carr N.,Jin H., Fonte R.,Si�g M.,Forsyth D.: “Automatic Scene   Inference for 3D Object Compositing”. ACM Transac�onsonGraphics33, 3(2014), 1–15. arXiv   1006.4903, doi:10.1145/2602146. Karsch K.:”Inverse Rendering Techniques for Physically Grounded Image Editing”. (2015), 158Zhang E.,Cohen M.F.,Curless B.: “Emptying, refurnishing, and relighting indoor spaces”. ACM    Transac�ons on Graphics (Proceedings of SIGGRAPH Asia 2016) 35, 6(2016)Yeung S.-K., Tang C.-K., Brown M.S., Kang S.B.: “Matting and compositing of transparent     and refractive objects”. ACM Transac�ons on Graphics 30, 1(2011),1–13. doi:10.1145/     1899404.1899406. Barron J.T.,Malik J.: “Shape, illumination, and reflectance from shading”. IEEE Transac�ons    on Pa�ern Analysis and Machine Intelligence 37, 8 (2015), 1670–1687. doi:10.1109/        TPAMI.2014. 2377712. 2The scene acquisi�on & reconstruc�on of depth can be summarized as: •  Our first contribu�on is represented by the depth acquisi�on step, namely, the cor-       rec�on kernel that renders a suitable depth map for dense point reconstruc�on. • The second contribu�on regards the religh�ng step, specifically the so� shadow pro-   jec�on. Our approach represents accurate so� shadows through a sta�s�cal shadow     mapping algorithm that is consistent with the illumina�on from the original 2D scene.The reillumina�on substep for accurate so� shadows is expressed through: We consider four types of experiments: • Based on standard stereo images - rely on stereo image pairs origina�ng from      well-established datasets. Ideal for real-life situa�ons of clu�ered environments. • Based on synthe�c scenes - in order to have control of GT depth and illumina�on.    Ideal for benchmarking our scene acquisi�on and shadow model. •  Applied to AR technology  - Ideal for illustra�ng the performance of our approach    for real-�me applica�ons where composi�ng is required. •  Stage-by-stage comparison - we can obtain intermediate results per stage and     compare with similar techniques, where possible.First term of equation sums all pizel-wise matching costs over the whole imageSecond term adds a penalty for all pixels with neighbours that have a different disparity} }LHS is the left image from the stereo pair. Centre image is the confidence map, along which localized filtering will take place. RHS image is the final depth after filtering.}Exponential shadow mapping. The blue lines illustrate a delimitation of a correct shadow filter, where initial assumption for distance between occluder depth and light source hold true.<=The first equation illustrates a shadow comparison function with the depth value at z sam-pled from the shadow map, where H(x) = 0 if x < 0, and 1 otherwise. Exponential shadow mapping solves the probability of incoming depth zS being in front of values already stored. The result is then used as a light calculation scaling factor for the final shadow fragment.Classic exponential shadow mapping loses contact shadow as the value of the test equation increases to-wards 1.We modify this function in order to invert the amount of shadow calculated in region [0, 1) - allows the value to increase with distance which in turn affords a softer shadow umbra at a smaller filtering kernel size<=LHS: classic exp shadow mapping with lack of contact shadow. Centre: exp in our framework. RHS: our method with contact shadowShadow projection interacting with original scene geometry. Each row contains close-up from both final result (LHS), and in-termediate snapshot (RHS).Performance overview of pipeline stages. Mesh size is represented by poly count. Execution time spans across all stages: stereo match filter, dense reconstruc-tion, light setup (before & after), result reprojection.First image - the original scene from a stereo pair. Second image is the result of our composit-ing method. Third image is our reconstructed depth. Last image is an intermediate snapshot.The present work was funded by EPSRC and School of Compu�ng, Newcastle Universityaretefact 'x

Towards a General Solution for Compositing 3D Artefacts in 2D Photographs

Abstract

Similar works

Full text

Available Versions

Newcastle University E-Prints