Clifford, J

Hilton, A

Ila, V

Kim, H

Pabst, S

Polok, L

Waine, T

University of Surrey

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). SIGGRAPH 2015 Posters, August 09 – 13, 2015, Los Angeles, CA. ACM 978-1-4503-3632-1/15/08. http://dx.doi.org/10.1145/2787626.2792617  Jigsaw - Multi-Modal Big Data Management in Digital Film ProductionSimon Pabst∗Double Negative Ltd.Hansung Kim†University of SurreyLukáš Polok‡Brno University of TechnologyViorela Ila‡Brno University of TechnologyTed Waine∗Double Negative Ltd.Adrian Hilton†University of SurreyJeff Clifford∗Double Negative Ltd.Figure 1: Jigsaw is a big data management software developed by Double Negative. It allows artists to efficiently organize and process thevast amount of heterogeneous data captured on a movie set for digital post production.1 IntroductionModern digital film production uses large quantities of data cap-tured on-set, such as videos, digital photographs, LIDAR scans,spherical photography and many other sources to create the fi-nal film frames. The processing and management of this massiveamount of heterogeneous data consumes enormous resources. Wepropose an integrated pipeline for 2D/3D data registration aimedat film production, based around the prototype application Jigsaw.It allows users to efficiently manage and process various data typesfrom digital photographs to 3D point clouds. A key step in the useof multi-modal 2D/3D data for content production is the registra-tion into a common coordinate frame (match moving). 3D geomet-ric information is reconstructed from 2D data and registered to thereference 3D models using 3D feature matching [Kim and Hilton2014]. We present several highly efficient and robust approachesto this problem. Additionally, we have developed and integrated afast algorithm for incremental marginal covariance calculation [Ilaet al. 2015]. This allows us to estimate and visualize the 3D re-construction error directly on-set, where insufficient coverage orother problems can be addressed right away. We describe the fasthybrid multi-core and GPU accelerated techniques that let us runthese algorithms on a laptop. Jigsaw has been used and evaluatedin several major digital film productions and significantly reducedthe time and work required to manage and process on-set data.2 Our ApproachFig. 2 shows the overall process for the integrated multi-modal dataregistration. We assume that the 3D point cloud obtained fromLIDAR scans is the target reference to register other modalities.∗{sicp, ted, jbc}@dneg.com†{h.kim, a.hilton}@surrey.ac.uk‡{ipolok, ila}@fit.vutbr.czFigure 2: 2D/3D workflow for registration3D data from active range sensors is directly registered to LIDARthrough 3D feature detection and matching. 2D footage is regis-tered via 3D reconstruction such as stereo matching or Structurefrom Motion (SfM) techniques. For example, high dynamic rangespherical imaging is often used to capture high resolution environ-ment maps and lighting conditions on-set. Geometry from sphericalimages can be recovered by vertical stereo matching. The output isa 3D scene containing all reconstructed data in a common coordi-nate frame.Contributions We present Jigsaw and its integrated pipeline forexploring, organizing and processing data captured on-set. We usehighly efficient and robust algorithms exploiting multi-core CPUsand GPUs for the key steps of the workflow. Our optimized GPUsolvers for sparse matrices with variable block sizes for exampleallow us to run a fast Bundle Adjustment on an on-set laptop toget a first look at the captured data. Previously many problems incaptured data were only detected in post-production, often weeksafter principal photography when it was too late to correct. Themulti-modal 2D/3D film production dataset used in this work hasbeen released publicly for research at: http://cvssp.org/impart 1ReferencesILA, V., POLOK, L., SOLONY, M., ZEMCIK, P., AND SMRZ, P.2015. Fast Covariance Recovery in Incremental Nonlinear LeastSquares Solvers. In Proc. ICRA 2015.KIM, H., AND HILTON, A. 2014. Hybrid 3D Feature Descriptionand Matching for Multi-modal Data Registration. In Proc. ICIP2014.1This research was supported by the European Commission FP7 projectIMPART (grant agreement No 316564)

English

Jigsaw - Multi-Modal Big Data Management in Digital Film Production

Jigsaw - Multi-Modal Big Data Management in Digital Film Production

Abstract

Similar works

Full text

Available Versions

University of Surrey